Beware Rogue Passenger Orphans!

Awhile back, one of our apps started throwing 403 errors. We also noticed some blank data coming through which should've been filled in by JavaScript.

One other strange symptom was that every once in a while when you'd visit the site in a browser, it wouldn't have any styling. Muy raro.

After a good amount of digging into this issue, it finally became clear what was happening.

While looking through the log files to try to figure out what was going on, I ran across the superbly named "passenger-orphan-killer.log". My eye being drawn to this, I decided to peruse it.

This turned out to hold the missing clues as to what was happening. For the past few days, it was filling up with messages such as this:

Tue Oct 4 03:00:01 EDT 2011
Killed 1 orphaned ruby process(es): 21901
Thu Oct 17 03:00:01 EDT 2013
/usr/local/bin/passenger-orphan-killer: line 14: /opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/gems/1.8/gems/passenger-3.0.7/bin/passenger-status: No such file or directory
passenger-status failed. aborting
Fri Oct 18 03:00:01 EDT 2013

It was trying to make a call to passenger-3.0.7/bin/passenger-status, which didn't exist! Passenger must have been upgraded, as passenger-3.0.8/bin/passenger-status did exist.

After finding that, we were able to make sure the correct passenger-status was being called, and this got rid of a rogue orphan Passenger process. That rogue process hadn't been loading CSS or JavaScript correctly, which had led to missing data (that was usually calculated with JavaScript) as well as the missing style.

Once we'd successfully executed the Passenger orphan, and gotten the Passenger Orphan Killer working once more, everything was shipshape.

Hopefully this can help anyone else out there who notices strange, intermittent behavior such as CSS or JavaScript not loading.

Joe Peck

Joe is a former SciMed developer.