Flash Media Server Uptime Monitoring

I recently needed to implement a way to monitor Flash Media Server for Load Balancing purposes. FMS traffic can be load balanced easily but in this case we wanted to have it stop sending traffic to an FMS server that was down. We've had a couple of times where FMS servers have run out of memory and stopped responding.

The load balancer we are was set up to look for a static file to know if the server is running but since FMS isn't a web server that wasn't going to be very useful. Fortunately FMS comes with a command line tool called fmscheck that can test a stream so it was fairly easy to expand that into an easy way to keep track of the FMS going down.

First I setup a simple shell script with just one line.

/opt/adobe/fms/tools/fmscheck --host localhost --app appname --play testvideo 0 10 --logfile /var/www/html/fmtest.txt

This will basically attempt to play the first 10 seconds of a video called testvideo.flv from the appname application. Notice that the .flv extension is not in the command. The output of the command will be saved in fmstest.txt and if successful will look like this.

#2008-08-23 16:42:20 
#Software: Adobe Flash Media Server Check 1.0.0 r15  
#2008-08-23 
#Fields: date    time        x-event          x-sname
2008-08-23     16:42:20     Connect         rtmp://localhost:1935/appname
2008-08-23     16:42:20     NetConnection.Connect.Success   Connection succeeded.
2008-08-23     16:42:20     Success         StreamCreated
2008-08-23     16:42:20     Success         StreamCreated
2008-08-23     16:42:20     NetStream.Play.Reset     Playing and resetting testvideo.
2008-08-23     16:42:20     NetStream.Play.Start     Started playing testvideo.
2008-08-23     16:42:20     NetStream.Data.Start     -
2008-08-23     16:42:26     NetStream.Play.Stop     Stopped playing testvideo.
2008-08-23     16:42:26     Program complete successful 

If it fails to stream, presumably because FMS is down or a problem with the app then the output looks like this.

#Version 1.0.0 r15  
#2008-08-23 16:30:06 
#Software: Adobe Flash Media Server Check 1.0.0 r15  
#2008-08-23 
#Fields: date    time        x-event          x-sname
2008-08-23     16:30:06     Connect         rtmp://localhost:1935/appname
2008-08-23     16:30:06     NetConnection.Connect.Success   Connection succeeded.
2008-08-23     16:30:06     Success         StreamCreated
2008-08-23     16:30:06     Success         StreamCreated
2008-08-23     16:30:06     NetStream.Play.StreamNotFound   Failed to play 0; stream not found.
2008-08-23     16:30:06     Play fail 

The load balancer wants a simple file that it can check for any change so I added a trivial php file that reads in the file and looks for "Program complete successful" and outputs "FMS Success" if it finds it.

<?php

$file = 'fmtest.txt'; 
$handle = fopen($file, "r"); 
$input = fread($handle, filesize($file)); 
fclose($handle); 

if ( strstr( $input, "Program complete successful" ) )
   print "FMS Success";
else
   print "FMS Failure";

?>

Cron runs the shell script every 2 minutes and the load balancer is checking regularly for the "FMS Success" output. If it sees anything else or the web server doesn't respond then it will stop using this FMS for traffic. This is also useful for watch programs to notify you as soon as an FMS server stops responding.