Sidekiq is well-known for memory bloat (see this), in the past I was using Monit to restart Sidekiq when it uses too much memory. As we transitioning to Docker, I want to figure out a cleaner solution to it without using Monit. I tried god first but turned out that it does not work in the container because the netlink does not work in Docker container. So instead, I went old school and wrote a bash script:
#!/usr/bin/env bash num_workers=${SIDEKIQ_WORKERS:=4} max_memory=${SIDEKIQ_MAX_MEMORY:=3000000} # 3GB sidekiq_config=${SIDEKIQ_CONFIG:=config/sidekiq.yml} function finish { echo "Shutting down Sidekiq workers..." for i in `seq 1 $num_workers`; do bundle exec sidekiqctl stop tmp/pids/sidekiq-$i.pid 15 & done wait exit 0 } trap finish SIGHUP SIGINT SIGTERM echo "Starting Sidekiq workers..." for i in `seq 1 $num_workers`; do bundle exec sidekiq -C $sidekiq_config -L log/sidekiq.log -i $i -P tmp/pids/sidekiq-$i.pid -d done while true; do sleep 60 & wait for i in `seq 1 $num_workers`; do pid=$(cat tmp/pids/sidekiq-${i}.pid) rss=$(cat /proc/${pid}/status | grep VmRSS | awk '{print $2}') now=$(date +"%T") echo "[$now] Sidekiq:$i memory usage: $rss KB" if [ $rss ] && [ $rss -gt $max_memory ]; then bundle exec sidekiqctl stop tmp/pids/sidekiq-$i.pid 30 sleep 1 echo "Starting Sidekiq:$i..." bundle exec sidekiq -C $sidekiq_config -L log/sidekiq.log -i $i -P tmp/pids/sidekiq-$i.pid -d fi done done
There is one very tricky thing, in your Docker file, make sure you do the following or your script will never trap the signal:
... ADD ./docker/sidekiq/run_sidekiq.sh ./run_sidekiq.sh RUN chmod o+x run_sidekiq.sh CMD ["./run_sidekiq.sh"]
Here are what I achieve with the scripts above:
- 1. Sidekiq workers will be restarted if they use too much memory.
- When EC2 instance gets terminated, Sidekiq workers will shut down gracefully.
- I sleep better.