Damned hackers!

If you've been trying to access my blog for the last couple weeks, I do apologize. Some POS hacker managed to find an exploit into my Wordpress blog and inserted links to some Russian malware sites. Rather than allow the exploits to be served up, I took my entire blog down.
 
I have now replaced my Wordpress blog with a Python CGI-based blog. I've moved the content over, but I still have to do some cleanup. So please bear with me - the blog will soon be back to it's former state, minus hacks.
Categories: hackers, blog
Date: 2012-11-10 14:40:49, 1 year and 165 days ago

See you in Chicago on Saturday, Sept 22!

Dan Mahoney will be one of the presenters at the BARcampChicago event at 1 p.m. Saturday, Sept. 22. His topic is How to Stream Live Video.
 
The talk, one of several scheduled for the day, will introduce participants to the tools needed to stream video, the various sources of software and services, and some of the pitfalls to avoid.
 
Check out the BARcampChicago site for info about the event, how to register, and how to suggest topics for the group.
 
Pumping Station One (3519 N. Elston Ave) will be hosting BARcampChicago. The doors will open at 9:30am on Saturday Sept 22 with an introduction starting at 10am and talks following. The space will be open until 7 p.m., says the BARcamp website.
 
See you in Chicago, Sept. 22!
 
Nancy
 
PS: Dan is systems engineer at Clickstreamtv.net, which provides an online video platform. i See his resume, etc, at http://www.furtechgroup.com/about-us/dan-mahoney-geek-and-cat-lover.html
Categories: streaming, python, Chicago
Date: 2012-09-21 13:47:32, 1 year and 215 days ago

Load testing streaming servers

One of the challenges we face trying to get a commercial–quality streaming server set up is how to test the server's capacity to respond to heavy stream loads. As handy as manufacturer spec sheets can be, it's one thing to read that your EC2–based server should be able to handle 150 Mbps of outbound stream and another to be willing to commit to that capacity in your business planning.
 
Let's say the live events you encode are being sent at a bit rate of 500 Kbps. Let's also say that you are running a streaming server on a single EC2 small instance. A single EC2 small instance should be able to deliver 150 Mbps, or about 300 streams at once. How do you verify this? It's usually not easy to call up 300 friends and have them all hit your server at the same time, and no matter how many computers you have at home or at the office it's unlikely you can get your hands on enough of them to make 300 requests at once.
 
This is where a load testing tool comes in handy.
 
The load testing tool I'll describe here is called Flazr, and it’s open source (GPL3). You can find it at http://flazr.com. There is also a wiki at with the specific content we want at http://sourceforge.net/apps/mediawiki/flazr/index.php?title=Main_Page.
 
If my Wowza (or FMS, or crtmpserver, or Red5) server is at mystreamserver.com, my live streaming application is named "live", and I'm pushing a stream with a name of “teststream”, I can impose a load of 100 connections on my server by running:
 
./client.sh –app live –host mystreamingserver.com –load 100 –port 1935 teststream
 
For example, if I encode the stream described above and send it to live.clickstreamtv.net/liverecord and use my stream name and auth key (stream name of 1084563), and use a command line of:
 
./client.sh –app liverecord –host live.cckstreamtv.net –load 10 –port 1935 1084563
 
then look at the "connectioncounts" page on the server (not accessible to our streaming customers but available internally) I see:
<Stream>
<Name>1084563</Name>
<SessionsFlash>10</SessionsFlash>
<SessionsCupegtino>0</SessionsCupegtino>
<SessionsSanJose>0</SessionsSanJose>
<SessionsSmooth>0</SessionsSmooth>
<SessionsRTSP>0</SessionsRTSP>
<SessionsTotal>10</SessionsTotal>
</Stream>

 
What are the limitations? If you’re doing this from your laptop while sitting at Starbucks, connected over their free WiFi, and trying to pull 50 connections you will be disappointed. In order to get meaningful results you will need to have:
  • a machine (or machines) with enough horsepower to run the JVM (Flazr is written in Java) and run potentially multiple threads
  • a connection with enough bandwidth to pull down the appropriate number of streams

 
The stream I usually use for testing is 500 Kbps of video and 96 Kbps of audio, for a combined total bit rate of 596 Kbps. If I am pulling down 10 connections to my live stream, I will be sucking down about 6 Mbps of content. If I increase the "–load" value to 100, my downstream bit rate increases to 60 Mbps.
 
One way to get access to affordable compute horsepower and bandwidth is to use an Amazon EC2 instance. A large instance goes for $.32/hour and offers 250 Mbps of bandwidth, so if all you need is to do a couple hours of testing you can move quite a bit of data on a large instance. The charges stop accruing when you stop or terminate the instance.
 
IMPORTANT NOTE regarding load balanced systems: If you are running Wowza in a load balanced origin/edge configuration you must remember that this test method will NOT exercises the load balancer! It just pulls a stream from the machine name in the command line and does not follow HTTP redirects.
Categories: streaming, testing, Wowza, flazr
Date: 2012-08-24 15:16:10, 1 year and 243 days ago

Live streaming setup using Flash Media Server on Amazon Web Services

Adobe and Amazon announced a few months ago that Flash Media Server would be available on Amazon Web Services by means of an FMS-enabled AMI. This was an addition to Adobe/Amazon's prior offering, that of doing live streaming through Amazon CloudFront by using FMS on Amazon's own servers. This new offering allows FMS to compete directly with Wowza for those companies wanting to use Adobe's software.
 
As of this writing (August 21, 2012) Adobe actually offers two versions of FMS on AWS – Flash Media Server (FMS) 4.5 and Adobe Media Server (AMS) 5. The FMS version doesn't offer server-side scripting, which my applications require, so I'm using AMS. Unfortunately I find Adobe's online documentation to be almost as good as Cisco's – that is, just a hair's breadth above useless. There is a lot of online documentation out there, but almost none of it directly addresses AWS. I am going to have to find my way through this morass of crappy documentation, and I will record here the important bits that I find to perhaps save you having to go through the same ordeal.
 
My first task was to figure out how to access the AMS administrative console. Once you launch and EC2 instance with an AMS-enabled AMI (in my case ami–766cce1f) and go to http:// you are presented with a pretty page with some sample videos and a link to the admin console. Clicking the admin link takes you to the admin console where you are presented with a user name and password prompt.
 
There is no default user name and password.
 
Note: the available documentation is written for FMS, but the commands for AMS use"“ams" in the name instead. So if a page you locate says to run "fmsmgr", chances are good that you actually need to run "amsmgr". And if you are told to edit a file named "fms.ini", you will probably have to edit "ams.ini".
 
I found the document at http://help.adobe.com/en_US/flashmediaserver/amazonec2/flashmediaserver_4.5_amazonec2.pdf and hoped it would help, but it didn’t. I tried:
vi conf/ams.ini ./amsmgr server ams restart ./adminserver restart but no joy. I posted a question on the Adobe support forums, and got a reply a couple hours later. The correct answer is at http://forums.adobe.com/docs/DOC-2334
 
The process: edit conf/Users.xml to set admin user name, then from command line run ./amsadmin -console -user admin and enter the admin user password.
Categories: streaming, Wowza, EC2, FMS, AMS
Date: 2012-08-21 15:30:47, 1 year and 246 days ago

User-agent based player switching, or streaming to Flash, iPad, and Android from a single link

Posted on June 11, 2012 by dan
 
A question I here pretty often is how to deliver content to multiple device types from a single player link. For example, “I am streaming a H.264 stream to Wowza and want to let users view my stream in Flash Player and on iPhones, iPads, and Androids. Do I really have to have three different links on my web page for people to click?”. The answer is no.
 
The easiest way to get automatic delivery to multiple devices is to use ClickstreamTV (http://www.clickstreamtv.net). The provided player automatically takes care of figuring out what device a user is trying to play from and delivering the appropriate stream. But for the moment let’s assume that you’re constrained by an external requirement and have to use a different player. This discussion will center around JWPlayer, but you can use the same technique with pretty much any player.
 
The basis upon which we make our decision about what stream to deliver is the player’s UserAgent. If a user is trying to play my stream on an iPad, the UserAgent header sent by the iPad will contain “iPad” somewhere in it. Likewise, an Android will have “Android” somewhere in the UserAgent header. Therefor, we’re just going to look at the UserAgent in a little javascript. If the user is coming from a mobile device we’re going to redirect it to the appropriate URL, otherwise we’ll deliver our standard player. For reference purposes, if my server is running an application named “liverecord” and I am encoding a stream named “1084563″, an iPad/iPhone/iTouch device will need to be redirected to “http://liverecord/1084563/playlist.m3u8″ and an Android device will need to be redirected to “rtsp://liverecord/1084563″.
 
And allow me to also preface the next paragraph with the warning that I AM NOT A JAVASCRIPT PROGRAMMER! I know just enough about javascript programming to get myself in trouble, so the code below is ugly and inefficient and any decent javascript programmer can clean it up greatly. So be warned:
 
Here’s my demo web page with the rough-cut javascript:
 
Test player for the “catfolks” stream
This text will be replaced

 
So here is a very simple, very ugly little demo of how to do your own delivery switching logic in javascript.
Categories: streaming, players, Flash, iOS, Android
Date: 2012-06-11 14:00:41, 1 year and 317 days ago

Lessons learned live streaming from an iPad


 
Two weeks ago the ADOPT animal shelter in Naperville, IL held a walk-a-thon. Nancy and I participated in honor of a recently deceased friend. Since I had a good handful of supporters I thought it might be a good idea to stream the walk live, from my iPad, using the livu app and streaming through the office video servers. It was easy to set up and the player looked really nice, but I learned some very important lessons. I’ll be glad to share them!
 
  • Make a harness. Holding an iPad in your hands while walking, and keeping it pointing towards the stuff you want to be shown, is a major PITA.
  • Check your connectivity. My iPad was 4G enabled, and I usually have good phone reception in that area, so I figured I’d be good to go. Wrong! My connection got dropped several times. If you are going to be streaming from a fixed location or a relatively small area you might be ok. If you’re going to be streaming from a larger area you might be screwed.
  • MAKE A HARNESS! That darned iPad gets really inconvenient really quickly.
  • Check the screen often. I learned the hard way that livu tends to stop apparently at random. 6 or 7 times during the walk I looked at the screen and found that livu was no longer running. This might be related to the connectivity issue. I haven’t had enough time to test thoroughly to verify this.
  • Do a lot of testing before the event. I’m not aware of whether livu is capable of saving a copy to what it streams to local storage. Unfortunately I had a minor misconfiguration on my streaming server that prevented it from saving an archive of the live stream, so I was unable to present the stream for on-demand viewing. Fortunately Nancy managed to capture some good video clips using her Flip.
     
    My next project is to either locate or write an Android app that can encode a live stream.
     
Categories: streaming, live, iPad, iOS
Date: 2012-05-18 14:06:28, 1 year and 341 days ago

Monitoring Wowza using JMX and Jython


 
Wowza (and other Java applications) allow you to monitor and control their operation by means of the JMX interface. This is nice and handy, but who wants to write a program (especially in Java) just to perform some simple management tasks?
 
Jython allows you to access the JVM from Python, making development of quick and dirty management scripts a lot easier. I documented a lot of what I learned on the Wowza forums (http://www.wowza.com/forums/showthread.php?15007-Monitoring-via-JMX) but I’ll also document that stuff here.
 
I can get a list of all applications currently loaded in my Wowza server with :
 
import java.io.BufferedReader; import java.io.IOException; import java.io.InputStream; import java.io.InputStreamReader; import java.io.PrintStream; import java.util.HashMap; import java.util.Iterator; import java.util.Map; import javax.management.MBeanServerConnection; import javax.management.ObjectName; import javax.management.openmbean.CompositeDataSupport; import javax.management.openmbean.CompositeType; import javax.management.remote.JMXConnector; import javax.management.remote.JMXConnectorFactory; import javax.management.remote.JMXServiceURL; from array import array import re import sys from sets import Set ad=array(java.lang.String,["admin","admin"]) n = java.util.HashMap() n.put (javax.management.remote.JMXConnector.CREDENTIALS, ad); jmxurl = javax.management.remote.JMXServiceURL("service:jmx :rmi://localhost:8084/jndi/rmi://localhost:8085/jmxrmi") testme = javax.management.remote.JMXConnectorFactory.connec t(jmxurl,n) connection = testme.getMBeanServerConnection(); mBeans = connection.queryNames(None, None) p = re.compile('applicationName=([^,]+),') iter = mBeans.iterator() appSet = Set() for m in iter: if hasattr(p.search(m.toString()), 'groups'): app = p.search(m.toString()).groups()[0] appSet.add(app) for app in appSet: print "We found application %s" % (app, ) testme.close()
 
I can see the total number of live streams being viewed with:
 
import java.io.BufferedReader; import java.io.IOException; import java.io.InputStream; import java.io.InputStreamReader; import java.io.PrintStream; import java.util.HashMap; import java.util.Iterator; import java.util.Map; import javax.management.MBeanServerConnection; import javax.management.ObjectName; import javax.management.openmbean.CompositeDataSupport; import javax.management.openmbean.CompositeType; import javax.management.remote.JMXConnector; import javax.management.remote.JMXConnectorFactory; import javax.management.remote.JMXServiceURL; from array import array import re import sys from sets import Set ad=array(java.lang.String,["admin","admin"]) n = java.util.HashMap() n.put (javax.management.remote.JMXConnector.CREDENTIALS, ad); jmxurl = javax.management.remote.JMXServiceURL("service:jmx :rmi://localhost:8084/jndi/rmi://localhost:8085/jmxrmi") testme = javax.management.remote.JMXConnectorFactory.connec t(jmxurl,n) connection = testme.getMBeanServerConnection(); object="WowzaMediaServerPro:name=Connections" attribute="current" attr=connection.getAttribute(javax.management.Obje ctName(object),attribute) print "We currently have %d players watching live streams" % (attr, ) testme.close()
 
and I can see the number of instances of the application named “dan” with:
 
import java.io.BufferedReader; import java.io.IOException; import java.io.InputStream; import java.io.InputStreamReader; import java.io.PrintStream; import java.util.HashMap; import java.util.Iterator; import java.util.Map; import javax.management.MBeanServerConnection; import javax.management.ObjectName; import javax.management.openmbean.CompositeDataSupport; import javax.management.openmbean.CompositeType; import javax.management.remote.JMXConnector; import javax.management.remote.JMXConnectorFactory; import javax.management.remote.JMXServiceURL; from array import array import re import sys from sets import Set ad=array(java.lang.String,["admin","admin"]) n = java.util.HashMap() n.put (javax.management.remote.JMXConnector.CREDENTIALS, ad); jmxurl = javax.management.remote.JMXServiceURL("service:jmx :rmi://localhost:8084/jndi/rmi://localhost:8085/jmxrmi") testme = javax.management.remote.JMXConnectorFactory.connec t(jmxurl,n) connection = testme.getMBeanServerConnection(); object="WowzaMediaServerPro:vHosts=VHosts,vHostNam e=_defaultVHost_,applications=Applications,applica tionName=dan,name=Application" attribute="instanceCount" attr=connection.getAttribute(javax.management.Obje ctName(object),attribute) print "I have %d instances of application 'dan'" % (attr, )
 
I hope those examples are of use to others. I’ve started writing a monitor app to keep an eye on the performance of my Wowza servers via a web interface.
 
Categories: streaming, Wowza, JMX, Java
Date: 2012-05-17 14:09:26, 1 year and 342 days ago

Remote logging for Wowza


 
Wowza makes use of the log4j Java module to handle logging. Log4j is a pretty capable logging system, and one could do a lot worse. It has the ability to log to a remote system, but it does so over unreliable UDP. If one is billing customers based upon the contents of log files, this could be undesirable. At Clickstreamtv.com we run multiple Wowza servers configured for load balanced origin/edge. This works very well for us, allowing us to rapidly adjust to changing demand levels. When demand spikes our code automatically spawns new Wowza worker instances that register themselves with the load balancing listener, quickly increasing our capacity. One of the challenges we were facing, though, is that each of these worker instances maintain their own set of log files, so if we have a couple dozen worker instances running our billing code would have to scan a couple dozen log files on each run. This lead us to look in to having all worker instances log to a remote server. As I mentioned before, log4j supports remote logging over UDP. We really needed remote logging over TCP. I investigated a few possible solutions, and what we finally decided to use was rsyslog. This decision was made for several reasons:
  • rsyslog is included in the Linux system upon which the Wowza AWS AMIs are built
  • rsyslog supports remote logging over TCP
  • rsyslog offers the ability to monitor another process’ log file (such as /usr/local/WowzaMediaServer/logs/wowzamediaserver_access.log), detect when a change has been made, and read just the new entries
  • rsyslog is easy to configure

 
One of our lightly loaded EC2 instances was chosen to act as the rsyslog server. In that machine’s /etc/rsyslog.conf we included: $ModLoad imtcp $InputTCPServerRun 514
 
to enable listening on TCP port 514. We described the format we wanted the log records to take with:
 
$template WowzaFormat,”%TIMESTAMP% %fromhost-ip% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n” $ActionFileDefaultTemplate WowzaFormat
 
We also decided that we would use the “local1″ facility for Wowza events, and specified the log file to use with:
 
local1.* /var/log/wowza.log
 
Then I modified the script executed by our startup file to copy a standard rsyslog.conf to all new instances. On the worker instances we tell rsyslog where to send it’s logs with:
 
local1.* @@
 
We also tell the worker instance rsyslog where to get the Wowza info and what to do with it with:
 
$InputFileName /usr/local/WowzaMediaServer/logs/wowzamediaserver_access.log $InputFileTag wowzaAccess: $InputFileStateFile statefile_access $InputFileFacility local1 $InputRunFileMonitor
 
When we restarted rsyslog on the worker instances, we saw the log messages appear on the rsyslog receiving server. Woohoo! However, we were also a bit disturbed – one of our first log lines looked like:
 
Apr 4 15:56:23 domU-12-31-39-14-F1-BF wowzaAccess: 2012-04-04#01100:07:02#011UTC#011disconnect#011session#011INFO#011200#0111628628511#011-#011_defaultVHost_#011liverecord#011_definst_#0111317.863#011[any]#0111935#011rtmp://50.17.117.175/liverecord#01174.243.203.120#011rtmp#011http://instance.clickstreamtv.net/service/flv-swf/2657028/2481/33033520#011WIN 10,1,52,14#0111628628511#0114965#01174078311#011-#011-#011-#011-#011-#011-#011-#011-#011-#011-#011-#011-#011-#011rtmp://50.17.117.175/liverecord#011-
 
It turns out rsyslog is very aggressive in escaping control characters – enough so that it even changes tab characters into “#011″. Not good! The fix was to add:
 
$EscapeControlCharactersOnReceive off
 
to the receiving server’s /etc/rsyslog.conf. We are now happily logging all of our Wowza traffic to a single log file on a single server. This makes log processing much easier. We are also now able to rotate our Wowza log every few hours to let us keep customer usage stats much more current than we had done before.
Categories: streaming, Wowza, logging
Date: 2012-04-20 14:13:38, 2 years and 4 days ago

Another video debugging player


 
A few messages back I posted on how to use JWPlayer’s wizard page to troubleshoot your live streams without having to have a web page created with an embedded player. Now I’d like to tell you about another such player, one that I like better.
 
osmf.org offers an excellent open source Flash player. They also have a wizard page that allows you to test your player configuration, and it offers some cool features as well. If you go to http://www.osmf.org/configurator/fmp, make sure you’re on the “Basic” tab, and enter your stream infomation, you can play your stream from there. In my case, if my Wowza server is at live.clickstreamtv.net, my application is named “liverecord”, and my stream name is 1084563, I would enter a “Video source” of “rtmp://live.clickstreamtv.net/liverecord/1084563″. I can then select an “Autoplay content” value of “Yes” to have my video start playing without having to click the “Play” button.
 
What’s extra cool, though, is what I get on the “Advanced” tab. If I select “Yes” for “Enable buffering overlay” and for “Optimize buffering” I can then experiment with different values for “Initial buffer length” and “Maximum buffer length” and see what kind of effect that has on the viewer experience. If I’m dealing with a source where the encoder might be having some bandwidth issues I can set larger values for the buffer lengths and see if that helps. I have also observed that with this player, when bufferering is enabled and a stream issue occurs I get a “Bufferering” dialog box when the stream issues are more than buffering can overcome. While this might be a distraction for your average stream viewer, I find that it helps me immensely – it lets me see when something is going on.
 
My experience over the last year and half is that Wowza is a tremendously powerful, very capable streaming server that is easy to run and easy to customize. It has also shown me that an awful lot of streaming customers (those who are originating the streams) have connectivity issues and aren’t aware of it. No matter how good the streaming server is, if the input stream is crap then so will be the output stream.
 
Categories: streaming, player, debug
Date: 2012-04-19 14:18:23, 2 years and 5 days ago

A simple gotcha on creating Wowza startup files


 
If you’ve spent any time at all creating Wowza startup files, you will have encountered the file size limitation issue. It’s not hard to get all of your required configuration files and jars in place and wrapped up in a zip file, but it’s really hard to do so without exceeding the 16 Kbyte size limitation.
 
There is an easy workaround to that, but it comes with a caution. You can always build a minimal startup file then have your tuning/tune.sh file grab whatever you left out of your startup file. I like knowing that the files I grab are going be somewhat secure from random strangers, so I use scp to do my files transfers. So let’s say you’ve got your tune.sh file all set to copy files from a server under your control:
 
scp -i root@myMainServer:~/downloads/* .
 
All’s well and good, right? Nope – that won’t fly. scp uses ssh as it’s underlying mechanism, and since the EC2 instance in question has just launched it doesn’t know anything about myMainServer so it will complain The authenticity of host ‘myMainServer’ can’t be established.
 
The workaround is simple: just don’t forget to add -o StrictHostKeyChecking=no to the scp command line. And don’t forget to chmod 600 as well.
Categories: streaming, Wowza, EC2, startup
Date: 2012-04-04 14:20:32, 2 years and 20 days ago