Sunday, October 10, 2010

Pushing Live Streams From Flash to iPhone

iPhone == anathema
So I've had to do some work with the iPhone regarding apps and that means looking at objective-C.  First let me suck yuck.  I started playing with OC code the other night when 1995 called and asked if it could please have its programming language back.  Anyway, as a Java programmer and a Flex programmer I have to say OC is nasty for gui dev and for basic app dev work.  Maybe it is really good for larger applications?  I don't know, but for little mobile apps it is cumbersome, ugly and overly fiddly.  That aside this post is actually about integrating an iPhone app into an existing Flex multi-media app, so I'll leave the OC coding to Mac Fanboys (as I'm a Java fanboy) and move on.

Flash Video Encoding
The existing app I have is a social networking site that allows you to stream live audio and video to an audience.  No, this is not a porn web cam, it is for musicians to broadcast their jam sessions or min-gigs to their fans.
So the performer logs on, clicks "yes allow flash to use my camera" and they start streaming to the server.  A fan logs on and clicks on the "watch my fave act" and they get the performer's stream playing for their pleasure.  The iPhone extension allows a person to watch the same performance on their iPhone rather than on their Flash enabled device (as in everything else in the entire universe, including TVs and refrigerators).
But, there is a problem:  iPhone only reads H264 encoded video with aac / mp3 audio.  Flash will also read that, but at present only creates Sorenson Spark / Speex (in my case) encoded video / audio.  That means the performer is producing a stream that is unintelligible to the iPhone.  There is, to my knowledge, no way to load additional codecs onto the iPhone and I don't think Apple will let you write an app that does.  So how the heck do you let an iPhone participate in these new fangled interwebs that Steve Jobs has heard so much about (and yet doesn't like)?  Oh and yes I am being contentious on purpose because I'm like that.

The magic word we are looking for is transcoder.  We take the flash video in one end and spit out h264.  This is easier said than done.  There is only one transcoder in the whole wide world, I know you don't believe me but it's true, and it is called FFMpeg.  Just try to find another application / library able to convert videos via a command line or with an API, go on I dare you.
Somewhere in my back end I need to use FFMpeg to create the iPhone stream otherwise I can just kiss my rich-media app goodbye.

Hooray for the Java open source community!  There is a lovely Java JNI wrapper for FFMpeg called Xuggler.  It even has some guts in it that let you read a stream from an rtmp source and publish back to an rtmp location.  As it happens my back end uses Wowza media server, also Java, so it is fairly easy to call Xuggler from a Wowza application.  What isn't so easy is finding any halfway decent documentation for Wowza. Also, Wowza isn't open source so it is a bit of a mystery as to what happens inside it.  I think there might be elves involved.  Anyway here is the basic glue I used (which I stole from a forum post on the Wowza forums, so thank whoever that was):

public class TheApp extends ModuleBase {
   ... stuff ...

   public void publish(IClient client, RequestFunction function, AMFDataList params){
      IMediaStream stream = getStream(client, function);
      stream.addClientListener(new StreamWatcher());
      ModuleCore.publish(client, function, params);
   ... other stuff ...

Ok so the above code adds a listener to the stream so that when the stream is actually published I get notified and a method in the StreamWatcher class is called (that's my class BTW)

public class StreamWatcher implements IMediaStreamActionNotify2 {
   public void onPublish(IMediaStream stream, String streamName, boolean isRecord, boolean isAppend) {
      H264Transcoder tc = new H264Transcoder(stream.getName());

Above you can see that when the stream is publish I create a new transcoder (which is a thread) and start it.

public class H264Transcoder extends Thread {

   public void run() {
try {
} catch (ParseException e) {
} catch (InterruptedException e) {

   public void testRTMPPublishH264() throws ParseException {
      Converter converter = new Converter();
      Options options = converter.defineOptions();
      String args = ""
         /* ffmpeg -i rtmp://
* -re 
* -acodec libfaac 
* -ar 22050 
* -vcodec libx264 
* -fpre default.ffpreset 
* -f flv rtmp://
      CommandLine cmdLine = converter.parseOptions(options, args);;

Wow, so the above isn't complete, but I've included the really important bits.  The reason it isn't complete is because I haven't perfected it yet.  Maybe I never will.  Oh, who am I kidding, of course I will, but not just now.

Important Points
The important points for this:
Xuggler can't connect to specific application instances, only to the _definst_ instance.  That means the rtmp url above has an application called "ppv" in which a stream called "streamName" is published.  Normally I'd have a named app instance that I'd publish into so I could track shared objects and the like, but that didn't work with Xuggler.  Also the output stream I've called streamName-iphone, is in a different application, again the _definst_ instance.  The reason for that is my ppv application has a bunch of rich user interaction stuff like chat and it gets info from the server about the performer and the people involved need to have valid accounts and so on.  I've only included the actual ffmpeg command line code that I have personally used and it works.  The preset I used is the default one that comes with Xuggler for h264 so I'll probably have to tweak that to get the best performance.

I may add an update post once I get all the kinks worked out.  For example I'd love to package this up as a simple jar + config arrangement, but I do have to work for a living so I may not get the time.


  1. Hi there,

    Nice POST. We have also the same questions about Flash VS Iphone streaming.

    By the way, what is the name of your current Facebook application for musicians ?

    Best regards


  2. A follow up on the flash and iphone streaming:
    I did manage to get the above working, but only just. There is a huge delay, like 3 minutes or so, due to the time it takes to transcode and the quality is not so good.
    ffmpeg has been successfully recompiled for iOS and my new efforts will be in using ffplay to read an rmtp flv stream directly. How cool would it be to have an iPhone app connect directly to a media server and consume flash streams? Very cool is how cool.
    The music man type app is not a Facebook app (yet) but rather an app that links with Facebook (and myspace and twitter) so you can share, like, comment and so on. My client is now signed up with a mob called Gigya who offer a single point of connectivity for social networks. The app itself is called and is, at the moment, completely written in flex.

  3. I've got the above code to work, but I'm having a hellova time testing it in html5/iOS. Do you have a successful test of wowza converting it to a http cupertino stream, and testing it from there?

  4. Bo, unfortunately I also had a hard time testing. There was such a delay in the transcoding that the client ended up saying "nah, stuff it." The only way I was able to test it was by running my app, stream some video to the server and connect via safari to the stream, wait for some time and if safari didn't time out the stream would commence.
    As it turns out I'm now hybridising the app into a flex / GWT monster with plans to eventually replace most of the gui with html keeping just the media streaming in flash.
    The only other way I was able to confirm a cupertino stream was being created was by watching the log files when an iPhone connected. It is an absolute pain how closed the iPhone is. I just wish I could read rtmp streams, I wish I could access the microphone and camera via flex (I know that one is Adobe not having finished the coding yet, so I won't really blame Apple for it)

  5. Hey, as a follow up on this one: Flash player 11 has h264 baked into it so you can capture iPhone friendly video streams through a flash client. That will render my above post meaningless, thank goodness. So you can either try to get the above working or wait a month or two until flash 11 is out.