Docker and Sidekiq

Sidekiq is well-known for memory bloat (see this), in the past I was using Monit to restart Sidekiq when it uses too much memory. As we transitioning to Docker, I want to figure out a cleaner solution to it without using Monit. I tried god first but turned out that it does not work in the container because the netlink does not work in Docker container. So instead, I went old school and wrote a bash script:

#!/usr/bin/env bash

num_workers=${SIDEKIQ_WORKERS:=4}
max_memory=${SIDEKIQ_MAX_MEMORY:=3000000} # 3GB
sidekiq_config=${SIDEKIQ_CONFIG:=config/sidekiq.yml}

function finish {
		echo "Shutting down Sidekiq workers..."
		for i in `seq 1 $num_workers`; do
	    bundle exec sidekiqctl stop tmp/pids/sidekiq-$i.pid 15 &
		done
    wait
    exit 0
}
trap finish SIGHUP SIGINT SIGTERM

echo "Starting Sidekiq workers..."
for i in `seq 1 $num_workers`; do
	bundle exec sidekiq -C $sidekiq_config -L log/sidekiq.log -i $i -P tmp/pids/sidekiq-$i.pid -d
done

while true; do
	sleep 60 &
	wait

	for i in `seq 1 $num_workers`; do
		pid=$(cat tmp/pids/sidekiq-${i}.pid)
		rss=$(cat /proc/${pid}/status | grep VmRSS | awk '{print $2}')
		now=$(date +"%T")
		echo "[$now] Sidekiq:$i memory usage: $rss KB"
		if [ $rss ] && [ $rss -gt $max_memory ]; then
			bundle exec sidekiqctl stop tmp/pids/sidekiq-$i.pid 30
			sleep 1
			echo "Starting Sidekiq:$i..."
			bundle exec sidekiq -C $sidekiq_config -L log/sidekiq.log -i $i -P tmp/pids/sidekiq-$i.pid -d
		fi
	done
done

There is one very tricky thing, in your Docker file, make sure you do the following or your script will never trap the signal:

...
ADD ./docker/sidekiq/run_sidekiq.sh ./run_sidekiq.sh

RUN chmod o+x run_sidekiq.sh

CMD ["./run_sidekiq.sh"]

Here are what I achieve with the scripts above:

1. 1. Sidekiq workers will be restarted if they use too much memory.
2. When EC2 instance gets terminated, Sidekiq workers will shut down gracefully.
3. I sleep better.