Skip to Content

Daemon-ize your processes on the cheap, part two: Supervisor

The process supervisor so good I didn't bother evaluating other options!

Posted on 9 mins read

This post should be treated as an historical artifact. It probably contain broken external links and it may no longer reflect my views or opinions.

In part one, I went into a little bit of detail about how I’ve come to view process super­visor tools as a really useful layer of middleware – they abstract out virtually all of the of the logic of forking, nohuping, stdout & stderr redi­rection, and boot-time initial­ization that comes from writing your own init/rc scripts and they provide tremendous func­tion­ality beyond the standard init script.

This time around I want to focus on Super­visor, which is one of the more popular and well docu­mented process super­visors in use in the Ops space. At a high level, I’m going over how I set up a proof-of-concept Graphite/StatsD server, and the Graphiti front-end that I’m using to visu­alize the metrics I’m collecting. I like this example because it’s a complete hodge-podge of tech­nologies: Graphite & Carbon are written in Python, Etsy’s StatsD1 is written in Node, and Graphiti is a Sinatra appli­cation. Those tech­nologies are all over the “how do I start this at boot-time?” map, because that’s the world we live in these days.

Here’s how I set up a Graphite stack using Supervisor

Super­visor uses an ini-style config format. This makes it very, very easy to under­stand, and very, very easy to tease apart, which is one of the big things that made Super­visor appealing to me. Super­visor also under­stands the idea of defining an include path for config file snippets, so your config­u­ration can be broken up into logical units at the filesystem level – this should be imme­di­ately appealing to anyone using a config­u­ration management system (Puppet, Chef, CFEngine, some awful-but-functional home-brewed shell scripts you’ve begged the IT director or CIO to let you retire).

Super­visor uses an ini-style config format. This makes it very, very easy to under­stand, and very, very easy to tease apart, which is one of the big things that made Super­visor appealing to me. Super­visor also under­stands the idea of defining an include path for config file snippets, so your config­u­ration can be broken up into logical units at the filesystem level – this should be imme­di­ately appealing to anyone using a config­u­ration management system ([Puppet][link0], Chef, CFEngine, some awful-but-functional home-brewed shell scripts you’ve begged the IT director or CIO to let you retire).

The how-to and why-for’s of how I installed Super­visor are less important than how it’s configured2, but what’s notable for this example is that I installed it using vendor packages so it’s not name-spaced into a location like /opt or /usr/local. This is strictly a matter of taste, as long as Super­visor knows where to find its dependencies.

Here’s my global Super­visor config file. It’s pretty standard, with virtually nothing changed from the vendor-provided config file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
[unix_http_server]
file = /var/run/supervisor.sock ; (the path to the socket file)
chmod = 0700 ; socket file mode (default 0700)

[supervisord]
logfile = /var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
pidfile = /var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
childlogdir = /var/log/supervisordervisor ; ('AUTO&' child log dir, default $TEMP)

; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl = unix:///var/run/supervisor.sock ; use a unix:// URL  for a unix socket

; The [include] section can just contain the "files" setting.  This
; setting can list multiple files (separated by whitespace or
; newlines).  It can also contain wildcards.  The filenames are
; interpreted as relative to this file.  Included files *cannot*
; include files themselves.

[include]
files = /etc/supervisor/conf.d/*.conf

The important line there is files = /etc/supervisor/conf.d/*.conf: that line tells Super­visor where to find addi­tional config­u­ration files, and it means that we can cleanly seperate our global config­u­ration from our indi­vidual service config­u­ra­tions. Trust me when I say THAT’S SOMETHING YOU WANT TO DO.

To get my Graphite stack running, I just had to create a file named some­thing like /etc/supervisor/conf.d/graphite.conf and start or restart super­visord. Here’s the config­u­ration for my entire Graphite stack:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
[program:gunicorn-graphite]
command = /usr/bin/gunicorn_django -u www-data -g www-data -b unix:/var/tmp/graphite.sock --log-file = /opt/graphite/storage/log/webapp/gunicorn.log /opt/graphite/webapp/graphite/settings.py
process_name = %(program_name)s
autostart = true
autorestart = true
stopsignal = QUIT
user = www-data

[program:carbon-cache]
command = python /opt/graphite/bin/carbon-cache.py --debug start
process_name = %(program_name)s
autostart = true
autorestart = true
stopsignal = QUIT

[program:statsd]
command = /usr/bin/node /opt/statsd/stats.js /opt/statsd/localConfig.js
process_name = %(program_name)s
autostart = true
autorestart = true
stopsignal = QUIT
user = www-data

[program:unicorn-graphiti]
directory = /opt/graphiti
command = /local/rvm/bin/rvm ruby-1.9.3-p286 do bundle exec unicorn -c config/unicorn.rb -E production
process_name = %(program_name)s
autostart = true
autorestart = true
stopsignal = QUIT
stopasgroup = true
stopwaitsecs = 5
killasgroup = true
user = www-data

For those watching, I didn’t lump these into a process group or break the config file into smaller logical units. Both would have been good ideas, but for the sake of this proof-of-concept I admit that I was a little lazy with best-practice. In actual production, you should probably have a good reason to ignore best-practice, and “I didn’t bother” or “I’m lazy” are never good reasons.

How do I control this thing once it’s started?

Meet super­vi­sorctl, the general purpose interface to every­thing that super­visord manages. You run it as root (or via sudo), and when invoked inter­ac­tively it looks a little bit like this:

ryan@ubuntu1204-knockaround:~# sudo supervisorctl
carbon-cache                     RUNNING    pid 1672, uptime 19 days, 6:53:57
gunicorn-graphite                RUNNING    pid 1674, uptime 19 days, 6:53:57
statsd                           RUNNING    pid 1671, uptime 19 days, 6:53:57
unicorn-graphiti                 RUNNING    pid 1673, uptime 19 days, 6:53:57
supervisor> help

default commands (type help <topic>):
=====================================
add    clear  fg        open  quit    remove  restart   start   stop  update
avail  exit   maintail  pid   reload  reread  shutdown  status  tail  version

supervisor>

So, at a glance, that’s a quick dash­board with the status of whatever you’re running, and a prompt that I used to run the help command. help lists all of the other commands available if you invoke it without an argument, and it will provide you with details for each of those commands in turn if you invoke them together. One of the things I like about super­vi­sorctl is that it also works non-interactively – you can pass any of those commands to it on invo­cation, and it will simply run the command.

ryan@ubuntu1204-knockaround:~# supervisorctl help fg
fg <process>  Connect to a process in foreground mode
Press Ctrl+C to exit foreground
ryan@ubuntu1204-knockaround:~# supervisorctl help restart
restart <name>        Restart a process
restart <gname>:* Restart all processes in a group
restart <name> <name>   Restart multiple processes or groups
restart all     Restart all processes
ryan@ubuntu1204-knockaround:~#

Why would I do this instead of writing init scripts?

Well, that’s up to you but if I were to make the argument, I’d base it on the strength and flex­i­bility of the simple config­u­ration options available for any defined programs. The ability to define process groups, start standard executable processes, or manage FastCGI process, and the robust tools provided for managing those processes make using Super­visor a veri­table no-brainer on any stacks that I have to deploy home-brew daemons on.

Alter­na­tively, a case could be made that the simple ini-style config format makes it easier to program­mat­i­cally generate valid config­u­ra­tions for new daemons (lowering the barrier to imple­men­tation further still), with Super­visor providing XML-RPC and HTTP inter­faces for service management. If you’re into empow­ering your users, then empow­ering them to restart services without requiring you to do it for them should have you sali­vating at the thought of your decreased operational-task load.

There’s also a number of 3rd party plugins for Super­visor that can extend listener func­tion­ality, inte­grate Super­visor with Nagios, or scale the number of processes spawned to match the number of CPU cores on any server. That’s pretty POWERFUL stuff to mix into your init system right there, and tradi­tional init scripts don’t offer any of that func­tion­ality out of the box.

So what’s my subjective opinion?

I’ve been running this config­u­ration for my Graphite stack since earlier this summer and it’s been absolutely rock solid. There’s defi­nitely some quirks and caveats to keep in mind when using Supervisor:

  • If a daemon takes a while to start or stop and you don’t specify a timeout value then Super­visor will wind up stuck in a limited loop wherein it tries to spin up the same process repeatedly until it hits the restart threshold, always without killing previous attempts first. This causes a process pile-up. The config­u­ration direc­tives are very fine-grained, so you can tune the initial­ization and setup of each process you’re spawning, but there’s no way around the fact that a chance exists that one-size-fits-all config­u­ra­tions won’t work with every program you want to give Super­visor respon­si­bility for.
  • Some programs just seem to stub­bornly resist being passed off to Super­visor. The one that has given me the most trouble so far has been Resque (I am not alone in this). I still don’t have a perfectly func­tioning Resque config­u­ration, but I will. And when I do, I’m going to post that thing erry­where.

  • Super­visor is a daemon in its own right, and as such it will consume resources of its own. They’re negli­gible but it’s still some­thing you should keep in mind if you’re trying to absolutely maximize the resource usage of any given server.

Are these deal-breakers? Not for me – I love Super­visor. I wish I’d known abut it before I pulled a 48-hour marathon session of refac­toring a number of home-brew init scripts at work. I like Super­visor so much that I actually never finished my bake-off with Monit because Super­visor solved my problems the first time around (that is bad science, by the way, and the correct put-down here would be ‘my exper­i­mental rigor has been found lacking’). I think that the nicest thing I can say about a tool is this: it solved the problem I had at the time and let me work on solving new problems instead. And that’s exactly what Super­visor has done.


  1. StatsD has actually been imple­mented in a number of languages. Here’s a great post on ServerZone detailing some of the more popular imple­men­ta­tions. [return]
  2. If you care, I used RPMs from EPEL on the CentOS systems I needed to set this up on, and Canonical-provided .deb’s for the Ubuntu systems. If you’re not using a package manager, I’m not going to explain compiling software or installing Python packages to you because however you’re managing your installed software is probably wrong. [return]