Sidekiq is well-known for memory bloat (see this), in the past I was using Monit to restart Sidekiq when it uses too much memory. As we transitioning to Docker, I want to figure out a cleaner solution to it without using Monit. I tried god first but turned out that it does not work in the container because the netlink does not work in Docker container. So instead, I went old school and wrote a bash script:
#!/usr/bin/env bash
num_workers=${SIDEKIQ_WORKERS:=4}
max_memory=${SIDEKIQ_MAX_MEMORY:=3000000} # 3GB
sidekiq_config=${SIDEKIQ_CONFIG:=config/sidekiq.yml}
function finish {
echo "Shutting down Sidekiq workers..."
for i in `seq 1 $num_workers`; do
bundle exec sidekiqctl stop tmp/pids/sidekiq-$i.pid 15 &
done
wait
exit 0
}
trap finish SIGHUP SIGINT SIGTERM
echo "Starting Sidekiq workers..."
for i in `seq 1 $num_workers`; do
bundle exec sidekiq -C $sidekiq_config -L log/sidekiq.log -i $i -P tmp/pids/sidekiq-$i.pid -d
done
while true; do
sleep 60 &
wait
for i in `seq 1 $num_workers`; do
pid=$(cat tmp/pids/sidekiq-${i}.pid)
rss=$(cat /proc/${pid}/status | grep VmRSS | awk '{print $2}')
now=$(date +"%T")
echo "[$now] Sidekiq:$i memory usage: $rss KB"
if [ $rss ] && [ $rss -gt $max_memory ]; then
bundle exec sidekiqctl stop tmp/pids/sidekiq-$i.pid 30
sleep 1
echo "Starting Sidekiq:$i..."
bundle exec sidekiq -C $sidekiq_config -L log/sidekiq.log -i $i -P tmp/pids/sidekiq-$i.pid -d
fi
done
done
There is one very tricky thing, in your Docker file, make sure you do the following or your script will never trap the signal:
... ADD ./docker/sidekiq/run_sidekiq.sh ./run_sidekiq.sh RUN chmod o+x run_sidekiq.sh CMD ["./run_sidekiq.sh"]
Here are what I achieve with the scripts above:
- 1. Sidekiq workers will be restarted if they use too much memory.
- When EC2 instance gets terminated, Sidekiq workers will shut down gracefully.
- I sleep better.
Code, Bitches and Money 