Adobe's Real-Time Media Flow Protocol

By Brian Lesser
April 24, 2009 | Comments: 12

This is a long post. I wrote it while preparing a presentation on RTMFP for Toronto's FITC Flash festival next week. After I recover from the festival I'll post the sample files for the demo applications.

The Flash player is one of the most widely distributed virtual machines. Each new Player release includes new features that can have a surprisingly quick impact on how people use the Web. For example, Flash Player 6 was released in 2002 with support for the Sorenson Spark (H.263) video codec. Even in 2002 the Spark codec was not the most efficient available. At the time, Microsoft's Windows Media was the most popular way to deliver superior quality video on the Internet. But despite its initially inferior quality, Flash video took off quickly because of the popularity of the player and the quick-to-start in-page viewing of video it provided. Today the Flash player supports excellent quality video and can decode the VP6 and H.264 codecs. It is by far the most frequently used way to deliver everything from lower quality Webcam chats to high definition video on the Web.

When Flash Player 10 was released in October 2008, it included a number of new features such as support for 3D effects, reflowing text across multiple containers and around graphics, an improved sound API, and hardware accelerated graphics, among others. As with all new features, the impact of the release will not be felt until developers and designers have enough time to work with them and until the new player nears the 90% desktop/laptop adoption mark. While the impact of a change in one company's browser may take years to have a significant impact on the Web, new features in Flash may take as little as a year. I'm particularly interested to see what happens as people discover what they can do with the new text layout features because they address a longstanding limitation in Flash. The difficulty in flowing and reflowing text around graphics in Flash has regularly driven developers and designers, who might otherwise prefer to work in Flash or Flex, to work with HTML, CSS, and JavaScript. Browsers do a good job of laying out text and browser-based editors like Google Docs are surprisingly good. If Flash Player 10's new text features are going to have an impact we should see some signs of that towards the end of this year.

For anyone interested in the Real-Time Web, the most interesting feature in Flash Player 10 was the inclusion of a new, though initially limited, UDP-based peer-to-peer protocol. The Real-Time Media Flow Protocol (RTMFP) is based in part on the Media Flow Protocol (MFP) pioneered by a small company named Amicima that was purchased by Adobe in 2006. In its initial release RTMFP only provides for the direct transfer of audio, video, and data between Flash players and is not designed to relay media from player to player. However, just like the inclusion of the Spark codec in Flash Player 6, the initial release of RTMFP may only be the beginning of the use of UDP and peer-to-peer in Flash. At Adobe's Max 2008 conference Matthew Kaufman mentioned that Adobe knows how to create groups, dynamic self-organizing overlay networks of RTMFP peers, with RTMFP. The APIs to do this are not included in Flash player 10 or AIR 1.5. So Adobe is talking about the possibility of application-level multicast but has said nothing that I am aware of about IP-level multicast. In the enterprise space both approaches are interesting because enterprise clients often have more bandwidth which makes it practical for overlay networks to deliver media and enterprise routers often support IP-multicast.

You can see Matthew Kaufman's Max presentation via Flash Video here:

http://tv.adobe.com/#vi+f15384v1056

While the entire presentation is worth watching, pay special attention to the Future Possibilities slide just over 38 minutes into the presentation. Also, have a look at Michael Thornburgh's Max Session: Sneak Peek: RTMFP Application-Level Multicast in Flash Player:

http://www.youtube.com/watch?v=-c2pMOtw3EA

What is RTMFP and how does it work?

RTMFP is a protocol that provides a way to establish client-to-client communications over UDP. Before Flash clients can communicate with each other directly, they must make an RTMFP connection to either Adobe's Stratus service or to a forthcoming release of Adobe's Flash Media Server (FMS). Once the connection is made, each Flash client is assigned a unique ID and can publish "direct" UDP streams containing audio, video, and data that are associated with its ID. Any clients connected to Stratus or to the same application instance in FMS that know the stream name and client ID of a publishing client can arrange to receive the other client's UDP stream directly. The work required to receive a direct stream is coordinated via the RTMFP connection to FMS or Stratus.

rtmfpSmall.png

Figure 1. Three Flash clients streaming audio, video, and data directly to each other.

Figure 1 shows three Flash clients connected to FMS or Adobe's Stratus service. Each must know the unique ID of the other clients and the name of their streams in order to receive a direct stream from each of them. Only clients connected to Stratus or the same FMS application instance can receive these streams. In this case none of the clients need to send or receive a stream from the server. In a three-way chat or video conference, where each client must receive a stream from every other client, each client must send two streams and receive two streams. The server coordinates communications between clients. Both the client-to-server connections and client-to-client connections are RTMFP connections. Each stream is illustrated as a one way arrow that travels directly from client-to-client.

RTMFP vs. RTMP

How does all this compare to the various versions of the RTMP protocol that started to appear in Flash 6 and that are available in the Flash Media Server? RTMP is a TCP-based protocol designed to facilitate client-to-server-to-client communications. Figure 2 illustrates how it works at a high level. Since RTMP works over TCP, a persistent and bidirectional connection is established between each Flash client and the server. Audio, video, and data flows within unidirectional streams from clients to the server and back out to the other clients over RTMP connections.

rtmpSmall2a.png

Figure 2. Three Flash clients communicating with each other over RTMP.

In figure 2 all communications are relayed through the server. Each client sends one stream and receives two, while the server must receive three streams and then send each of them out to the other two clients. In this case the server must handle nine streams.

Advantages of RTMFP over RTMP

Allowing clients to communicate directly with each other using a UDP-based protocol like RTMFP has obvious advantages. First, the fact that UDP is used as the transport means that both client-to-server and client-to-client communications have less latency. Second, the direct client-to-client connections further reduces latency because streams do not have to flow from client-to-server-to-client. Third, the bandwidth cost, an impediment to building collaborative applications that incorporate audio and video, is moved off the server. Finally, Flash player 10 also includes the Speex audio codec, which provides better audio quality than the Nellymoser codec first, incorporated into Flash 6. Speex is an open-source and patent-free codec designed to perform well even when some audio packets are lost.

Two other less obvious advantages mentioned in Adobe's RTMFP FAQ include:

  • "Rapid Connection Restore: Connections are re-established quickly after brief outages. For example, when a wireless network connection experiences a dropout. After reconnection, the connection has full capabilities instantly."
  • "IP Mobility: Active network sessions are maintained even if a client changes to a new IP address. For example, when a laptop on a wireless network is plugged into a wired connection the connection will not be interrupted. This is a critical requirement for communication or live solutions."

Anyone who has worked with TCP-based RTMP connections knows how difficult it can be to recover from a dropped wireless connection and deal with IP mobility issues.

Further, because RTMFP is based on UDP it can do a better job and has been designed for better flow control and TCP-friendly congestion control.

Finally, RTMFP was designed to be a secure protocol. It is always encrypted using 128 bit Advanced Encryption Standard (AES) and provides features that can be used effectively to resist man-in-the-middle attacks.

Limitations

Using RTMFP has some limitations. Internet Service Providers (ISPs) limit both the downstream and upstream bandwidth available to clients. With direct streams the upstream bandwidth may be a particular problem when several clients must stream video to each other. For example, Figure 1 illustrates the streams required to have a three-way chat or video conference. Each Flash client must send a stream directly to every other Flash client and receive a stream from all the other clients. The upstream bandwidth required scales linearly with the number of clients whereas with RTMP only one stream has to be sent to the server. Another limitation with client-to-client communications is that an extra stream must be sent to the server if it must be recorded. To record a three-way video conference each client must send out 4 streams. For non-video streams, or low resolution video, or when enterprise clients have sufficient bandwidth, this may not be a problem. Where ISPs provide limited upstream bandwidth this can be a problem. By default RTMFP clients are limited to 8 outgoing connections but this limit can be changed using client-side ActionScript.

Another interesting though less important limitation regards direct client-to-client video that doesn't occur with client-to-server-to-client streams. When a client asks to play a video stream from FMS, the server can dummy up an initial key frame to send to the client. Adobe didn't build the capacity to dummy up a key frame into the Flash player. As a result, and depending on the key frame interval used, the initial display of video will not always look right. An extreme example of what you might see for a fraction of a second is shown in Figure 3.

interFrameKeyFrame.png

Figure 3. Initial frame coming from another client over RTMFP followed by a key frame

As soon as the first key frame arrives the video will look normal. Still, this feature may discourage developers from using longer key frame intervals to reduce bandwidth. Figure 3 is a little extreme. I had to use a key frame interval of 200 frames to make it easier to get a screen capture of the effect because it doesn't last long.

A related limitation has to do with the Sorenson codec. To avoid video artifacts appearing from dropped packets Flash Player 10 and AIR 1.5 send video, like data, with full reliability. This is unlike FMS which is capable of dropping frames rather than packets when it detects network congestion.

Connection Failures

Two hurdles must be crossed to make RTMFP work.

First, an RTMFP connection must be made to FMS or Stratus. Since RTMFP is purely a UDP-based protocol, the Flash client must be able to successfully send UDP packets to port 1935 and one higher port on FMS or Stratus.

Second, once an RTMFP connection is established, individual clients must also be able to connect to each other. RTMFP does an amazing job of connecting clients to each other behind firewalls and NAT devices. FMS or Stratus helps accomplish this by coordinating client-to-client connections. A client that wants to receive a stream can be instructed to respond to the originating client before the first packets arrive from the other client. The early response will be seen by many firewalls and NATs as initiating a request for a connection that creates a hole for the incoming UDP packets and the establishment of a peer-to-peer connection.

Still, you can find some cases where client-to-server RTMFP connections cannot be established through NATs or through firewalls that severely restrict or do not permit UDP traffic. Similarly, client-to-client RTMFP will not always work across two NAT devices and will be stopped by some firewalls.

If a Traversal Using Relays around NAT Proxy (TURN) is available the Flash player may be configured to use it without authentication. More information is available in Jozsef Vass's article: Stratus service for developing end-to-end applications using RTMFP in Flash Player.

Failover Options

No failover options are available if a client/server or client/client RTMFP connection cannot be made using Stratus there. However, with FMS failover to RTMP is possible.

If a client cannot connect to FMS using RTMFP it will very likely be able to connect to the server using one of the RTMP protocols. In the worst case it may have to tunnel RTMP over HTTP (RTMPT). Clients that "fail over" to RTMP cannot communicate directly with RTMFP connected clients - even if they are connected to the same application instance and could - in theory - communicate directly with each other behind a common firewall. However, that doesn't mean that all is lost. Both RTMFP and RTMP connected clients can still relay streams through FMS.

Some clients will be able to connect to FMS but not to each other. In that case clients can also send streams to each other via the server.

Figure 4 shows how three clients - one of which has failed over to RTMP - can still communicate.

rtmfprtmpSmall2.png

Figure 4. A mixture of RTMFP and RTMP connected clients.

In Figure 4 a client that failed to connect with RTMFP has connected to FMS via RTMP and can still send and receive streams to the other clients via the server. The other clients can send streams directly to each other but must also send a stream to the server for the RTMP connected client. While this imposes a load on the server, and very likely increases the latency between some clients, there may still be some improvement in overall latency whenever a client/server stream travels via RTMFP. Fail over to RTMP and server distributed streams is not automatic. Client-side code is required to detect that an RTMFP connection has failed and to make a new connection attempt via RTMP. Also, a directly published RTMFP stream cannot be automatically proxied by FMS to other clients. The publishing client must create a second stream which it publishes to the server.

There are other drawbacks to failover scenarios. The CPU load on the server is higher when it must act as a hub for RTMFP streams. On the other hand you can expect lower latency for client/server RTMFP streams. Also, even though only one of the three clients in the illustration connects via RTMP the impact on the server is quite high. Only two streams are sent directly from client-to-client. The remaining seven streams must traverse the server.

Stratus vs. FMS

I've already mentioned that Adobe's Status service does not support RTMP so you can't fail over to RTMP. Stratus has other limitations:

  • it does not support shared objects
  • you can't send a stream to or from Stratus
  • server-side ActionScript is not available
  • all connections to the service are accepted provided the service URI is correct and there are no extra NetConnection.connect parameters.

Stratus was designed from the ground up purely as a scalable hosted rendezvous service. Since it doesn't support server-side scripting, client/server streams, or shared objects there is no way to share Peer IDs between clients. Since Peer IDs are needed to establish a direct connection between clients, some other means is needed to share Peer IDs such as a separate XMPP server or Web service.

A Stratus URI always starts with:

rtmfp://stratus.adobe.com/xxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxx/

where the xxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxx is the developer key obtained from Adobe. Because the URI looks similar to FMS URIs its tempting to think of as the address of your application on Stratus. But that isn't really correct. In FMS applications can have separate instances. So you might think you can do this to separate your clients:

rtmfp://stratus.adobe.com/xxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxx /roomA
rtmfp://stratus.adobe.com/xxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxx /roomB

In fact, clients connected with the roomA and roomB URIs can still communicate with each other so there doesn't appear to be any practical value in using different URIs.

RTMFP and ActionScript

If you have written client-side ActionScript to work with Flash Media Server you'll find that the APIs you are already familiar with have not changed very much to enable RTMFP. More information is available from the NetConnection and NetStream classes (also NetStreamInfo) and some NetStream methods have new parameters.

To develop RTMFP enabled applications in Flex Builder you must have release 3.0.2 or later, the 3.2 SDK or later, and must set the compiler property of your Flex project to compile for Flash Player 10.

Making a Connection

The connection process is just like making a connection to Flash Media Server except the protocol is RTMFP.

Here's a snippet of code that illustrates the basic connection steps to a future release of FMS:

private function init():void{
  // Create a New NetConnection as usual:
  nc = new NetConnection();
  // Setup the usual event listeners:
  nc.addEventListener(NetStatusEvent.NET_STATUS, netStatus);
  nc.addEventListener(AsyncErrorEvent.ASYNC_ERROR, asyncError);
  nc.addEventListener(IOErrorEvent.IO_ERROR, ioError);
  nc.addEventListener(SecurityErrorEvent.SECURITY_ERROR, securityError);
}

private function connect():void{
  // Connect using rtmfp instead of rtmp:
  nc.connect("rtmfp://localhost/helloRTMFP/roomA", userInfo);
}

Other than the "f" in rtmfp the code is identical to how you normally connect to FMS over RTMP.

Connecting to Adobe's Stratus service is a little different. You must include your developer key in the URI and cannot pass a second connect parameter:

nc.connect("rtmfp://stratus.adobe.com/xxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxx/");

As mentioned earlier, extra path (instance) information has no meaning in Stratus. Regardless of the connection URI any client can communicate with any other client. In FMS clients must be connected to the same URI (application instance) to communicate.

Now, let's look at what happens when you successfully connect to a version of FMS that supports RTMFP. In client-side ActionScript you'll find some new and interesting properties on the NetConnection object (but only when you compile for Flash Player 10). Here's some trace output showing two of them after a successful connection (nc is the NetConnection object):

nc.farID: e936e23de6ca296cbe5306fb4a1eee597596f11d22cb0172adbdfc14015a9483
nc.nearID: c26b09e2755f3556b6e09c32ab674de1f1dw32693af3e473221c0d7dfc5218f08

On the Flash client the nc.nearID is the unique ID of the client and the nc.farID is the ID of the current application instance running in FMS.

On the server side the meaning of "far" and "near" is reversed. A client that connects to FMS will have these properties on the server's client object:

client.nearID: e936e23de6ca296cbe5306fb4a1eee597596f11d22cb0172adbdfc14015a9483
client.farID: c26b09e2755f3556b6e09c32ab674de1f1d32693af3e473221c0d7dfc5218f08

So, on the server the client.farID is the unique ID of the remote client and client.nearID is the ID of the current application instance.

IDs are not just simple values handed out by the server. Each ID is a 256 bit number, is guaranteed to be unique and is derived from strongly random values. When connecting to FMS the nc.farID is associated with an application instance's session. If an application instance is shut down and restarted it will have a new ID. In Stratus the nc.farID is associated with a node on the Stratus service. On the client the nc.nearID is associated with the connection to the application instance. If the client's NetConnection is closed and reopened a new nc.nearID will be issued to the client.

Since FMS knows all of its clients IDs, it can easily share them between clients that need to connect to each other. This isn't possible in Stratus so a Stratus connected client must send its ID to a Web Service or advertise it to other clients using an instant messaging system.

Publishing an RTMFP Stream

Publishing a direct stream is almost the same as publishing a stream to FMS. In the following code a second parameter is passed into the NetStream constructor. The constant NetStream.DIRECT_CONNECTIONS insures that the stream will be published directly to subscribing clients rather than sent to the server.

private function publishStream():void{
  
  ns = new NetStream(nc, NetStream.DIRECT_CONNECTIONS);  
  ns.addEventListener(NetStatusEvent.NET_STATUS, netStatus);
  
  camera = Camera.getCamera();
  if (camera){
    camera.setMode(240, 180, 15);
    ns.attachCamera(camera);
    video.attachCamera(camera);
  }

  microphone = Microphone.getMicrophone();
  if (microphone){
    microphone.codec = SoundCodec.SPEEX;
    ns.attachAudio(microphone);
  }
  
  ns.publish("streamName");
}

Also, note that the microphone codec is set to the constant SoundCodec.SPEEX to insure that the Speex codec is used. When there are no subscribers to the stream Flash doesn't send the stream anywhere. It just waits for another client to subscribe to the stream. To subscribe to the stream other clients must know the peer ID of the publishing client and the name of the stream.

Playing an RTMFP Stream

Provided another client has already connected to Stratus or the same FMS application instance, it can subscribe to a stream using the code below. In this case the NetStream constructor is passed the Peer ID of the publishing client.

private function playStream():void{
  ns = new NetStream(nc, "c26b09e2755f3556b6e09c32ab674de1f1dw32693af3e473221c0d7dfc5218f08");
  ns.addEventListener(NetStatusEvent.NET_STATUS, netStatus);
  video.attachNetStream(ns);
  ns.play("streamName");
}

The code sample is for illustrative purposes as the peer ID of the publishing client cannot be known in advance and so the peer ID cannot be hard coded. Here's a more likely example:

private function playStream():void{
  ns = new NetStream(nc, clientInfo.peerID);
  ns.addEventListener(NetStatusEvent.NET_STATUS, netStatus);
  video.attachNetStream(ns);
  ns.play("streamName");
}

where clientInfo is an object that contains information about the publishing client.

One thing to note is that a direct stream name does not have to be unique. Several clients could publish a stream named "main" and no conflict would exist, while only one client can publish a stream named "main" to an FMS instance.

Using Remote Shared Objects to Distribute Peer IDs

An easy way to share information about clients connected to FMS is put information about each client into a Remote Shared Object. Whenever a client connects to FMS a property is added to the shared object containing information about the client. When a client disconnects its property in the shared object is removed. Here's an abbreviated listing of some FMS server-side ActionScript that does that:

application.onAppStart = function(){
   clients_so = SharedObject.get("public/clients");
}

application.onConnect = function(client, credentials){
   
   // Build an object to send back to the client so the client knows
   // its FMS ID and so other clients know about this client:
   clientInfo = {
   	id: client.id,
   	protocol: client.protocol,
   	peerID: client.farID
   }
     
   clients_so.setProperty(client.id, clientInfo);
   application.acceptConnection(client);
   client.call("onConnect", null, clientInfo);
   return true;
   
}

application.onDisconnect = function(client){
   clients_so.setProperty(client.id, null);
}

The code creates a remote shared object at "public/clients" so that other clients can connect to the shared object and retrieve the ID of any of the other clients. Each time a client connects, a clientInfo object for it is stored in the remote shared object under the client's id. The client.id is a unique ID provided by FMS and has nothing to do with RTMFP IDs. Here's a table that illustrates the client data that is created in the remote shared object by the code listed above:

Remote Shared Object Properties

Property NameProperty Value
BCAoYlcB

{id: "BCAoYlcB" peerID: "4462cd6c614b9985921a514d35c356557b2e20abbcd2741c325f8959823c2471" protocol: "rtmfp"}

CJAQYFNJ

{id: "CJAQYFNJ"
peerID: "92d93e09b2a0b194d8688980120627ed3093e593a0b97b98ff7a8618571ea94a"
protocol: "rtmfp"}

DIA4o9NJ

{id: "DIA4o9NJ"
peerID: "14f298edc010f608999dc6e839054c4fbc7cf05fa701cdd4d8ed134433bb2aab"
protocol: "rtmfp"}

On the client a VideoConference component might have a method to subscribe to the shared object this way:

public function onConnect(nc:NetConnection, clientInfo:Object):void{
   this.nc = nc;
   myClientInfo = clientInfo;
   clients_so = SharedObject.getRemote("public/clients", nc.uri, false);
   clients_so.addEventListener(SyncEvent.SYNC, clientsOnSync);
   clients_so.connect(nc);         
}

Now, let's say we want to build a simple VideoConference application where every client automatically publishes a stream and plays a stream from each of the other clients. A convenient way to do this is to create something like a VideoPanel component for each client. The VideoPanel component is responsible for playing that client's stream. Similarly, a VideoPanel could be used to publish the stream of the local client. To play a stream each VideoPanel needs access to the NetConnection and information about the client.

After subscribing to the public/clients remote shared object, changes in the shared object cause a synEvent to be dispatched. When the VideoConference component receives a SyncEvent, it means that a client has connected or disconnected from the FMS instance. Here's an abbreviated listing of a method to handle shared object sync events. A SyncEvent always contains a changeList array. The array contains information objects about each property that has changed in the shared object. The code walks through the change list objects. In this case when it finds an info.code value of "change" it means a client has connected and when it finds an info.code value of "delete" it means a client has disconnected. The info.name property is the name of a property in the shared object that has changed. In this example, each client's clientInfo object is available in a property of the Remote SharedObject.

private function clientsOnSync(event:SyncEvent):void{
   var changeList:Array = event.changeList;
   var info:Object;
   var clientInfo:Object;
   var videoPanel:VideoPanel;
   for (var i:Number = 0; i < changeList.length; ++i){
      info = changeList[i];
      switch (info.code){
         case "change":
            // Get the clientInfo object from the shared object
            // info.name is the name of shared object property that contains the 
            // clientInfo object:
            clientInfo = clients_so.data[info.name];
            // If there is no child panel with the same FMS client id create one:
            if(! getChildByName(info.name)){
               videoPanel = new VideoPanel();
               videoPanel.name = info.name;
               // This ID test only works for pure RTMFP applications:
               if (nc.nearID == clientInfo.peerID){
                  // Create a new video panel for the current user.
                  videoPanel.publish(nc, clientInfo);
                  // Add it as the first child in the tile
                  addChildAt(videoPanel, 0);
               }
               else {
                  // Create a new video panel for a remote user.
                  videoPanel.play(nc, clientInfo);
                  // Add it to the tile
                  addChild(videoPanel);
               }
               
            }                  
            break;
         case "delete":
            videoPanel = getChildByName(info.name) as VideoPanel;
            if (videoPanel){
               removeChild(videoPanel);
               videoPanel.clear();
            }
            break;
      }
   }
}

In this case, the code compares the nc.nearID, which is the ID of the local client, to the clientInfo.peerID to see if it should publish a stream for the local client or play the stream of another client. If the IDs match, the clientInfo object represents the local client and a VideoPanel is created to publish a stream. If the IDs do not match a VideoPanel is created to subscribe to another client's stream.

Controlling Access to Streams

In the simple Video Conference room example, every client knows every other client's peer ID and stream name. You don't have to design your applications that way. It is possible to only share IDs between clients after the users have agreed to talk to each other. However, it is also possible to control access to a stream on the publishing client using the NetStream's onPeerConnect method.

For example, a client can maintain an access control object that contains the peer IDs of clients that are allowed to subscribe to a stream it is publishing:

var aclObject:Object = [];

//Allow access for this client:
aclObject[clientInfo.peerID] = true;

Whenever a remote client tries to subscribe to the local client's stream, the NetStream.client.onPeerConnect method is called and can return true to allow access or false to deny access:

private function onPeerConnect(subscriber:NetStream):Boolean{
   return aclObject[ subscriber.farID ];
}

There are different ways to setup an onPeerConnect method on the NetStream's client object. Here is one way to do it:

private function init():void{
   var aclPublisher:ACLPublisher = this;
   streamClient = {
      onPeerConnect: function(subscriber:NetStream):Boolean{
      return aclPublisher.onPeerConnect(subscriber)}
   }
   ns = new NetStream(nc, NetStream.DIRECT_CONNECTIONS);
   ns.client = streamClient;   
}

In this case I've arranged for the ns.client.onPeerConnect method to call the VideoConference component's onPeerConnect method that was listed earlier.

Disconnecting a Remote Client's Stream

In cases where the local client has allowed a remote client to subscribe to its stream, access can still be cut off as needed. The NetStream class contains a peerStreams array of NetStream objects. Each NetStream object in the peerStreams array represents a subscribed stream on a remote client. To disconnect the remote client's stream call close() on the peer NetStream. For example to disconnect the first subscribed stream in the peerStreams array:

var nsPeer:NetStream = ns.peerStreams[0]
if (nsPeer){
   nsPeer.close();
}

Sending Data Using NetStream.send()

After a connection has been established between clients, a published stream can be used to reliably send data to the other client. The data can be of any type supported by AMF3. That includes ByteArray which can be used to send small files from client to client.

To send data over a stream use the NetStream's send method. For example if ns is a NetStream that has been published:

ns.send("receiveFile", file.name, file.data as ByteArray);

The first parameter is the name of a method to call on the remote client's Netstream.client object. The remaining parameters are the data being sent over the stream.

To receive the data on a remote client, the client must be playing the stream and its NetStream.client object must have a method on it ready to receive the data. One way to set this up is to do this:

private var nsClient:Object;

//...

var filePlayer:FilePlayer = this;
nsClient = {
   receiveFile: function(fileName:String, fileData:ByteArray):void{
      filePlayer.receiveFile(fileName, fileData);
   }
}

ns.client = nsClient;

In this example the client object will call another function to actually receive the file.

Here's an abbreviated listing of code that sends the file out on the publishing stream:

// Ask the user to browse for a file to load:
private function sendFile():void{
   file = new FileReference();
   file.addEventListener(Event.SELECT, handleFileSelection);
   file.browse();
}

// Load the file:
private function handleFileSelection(event:Event):void{
   file.removeEventListener(Event.SELECT, handleFileSelection);
   file.addEventListener(Event.COMPLETE, handleFileLoad);
   file.load();
}

// Send the file over the NetStream.
private function handleFileLoad(event:Event):void{
   ns.send("receiveFile", file.name, file.data as ByteArray);
   file.removeEventListener(Event.COMPLETE, handleFileLoad);
}

Here's an abbreviated listing of the code that receives the file on an incoming stream:

private var nsClient:Object;

// Setup the 	receiveFile method on the NetStream client object:
private function init():void{
   var filePlayer:FilePlayer = this;
   nsClient = {
      receiveFile: function(fileName:String, fileData:ByteArray):void{
         filePlayer.receiveFile(fileName, fileData);
      }
   }
   ns.client = nsClient;
}

// This component function is called by the client.receiveFile method:
private function receiveFile(fileName:String, fileData:ByteArray):void{
   // Since files can only be saved after a user interaction:
   pendingFileData = fileData;
   pendingFileName = fileName;
   Alert.show("You have received a file that is " + fileData.bytesAvailable + 
        " bytes in size.\nDo you want to save the file?", 
        "Alert", Alert.YES | Alert.NO, this, saveFileChoice); 
}

// Finally, save the file if the user agreed.
private function saveFileChoice(event:CloseEvent):void{
   if (event.detail == Alert.YES){
      var file:FileReference = new FileReference();
      file.save(pendingFileData, pendingFileName);
   }
   pendingFileData = null;
   pendingFileName = null;		
}

There are no Client-Only Remote Shared Objects

Client-to-client RTMFP only provides a way to send audio, video, and data over a NetStream. It does not provide higher-level functions to keep client data distributed across clients in sync. Nor does it provide a way to save data that is passed between clients.

Flash Media Server's remote shared objects provide a great way to consistently distribute synchronized information to all clients while retaining a copy of all the information on the server. Late arriving clients can pick up all the information they need when they connect to FMS. A change in data by one client is automatically distributed to all the others.

Adobe does not provide a way to create pure peer-to-peer shared objects that remain synchronized as clients connect and disconnect from each other. However, clients connected via RTMFP to FMS can still make use of server-side remote shared objects. Also, there is no way to call a method on a remote client and receive a response the way you can with the NetConnection object used to connect to FMS. For direct client-to-client method calls, developers must use an outgoing and incoming stream to send out information and get a response. Instead of using remote shared objects to broadcast messages each stream must carry a message. Persisting information for late arriving clients requires keeping information on clients or storing it on an application server for later retrieval. A simple text chat is a good example. Messages can be sent to each client as they are typed. But a late arriving client that must read earlier messages must collect old messages from other clients. Otherwise each client must send each message to a server where they can be picked up later.

Out of the box, RTMFP is great for ephemeral audio, video, and data transfer directly between clients. Even when data must be persisted to a server, RTMFP may still significantly cut down on the overall server bandwidth required and will reduce latency. For example recording a three way video conference only requires three streams go to the server while the rest are delivered directly.

Will it Scale?

ISPs usually do not provide the same level of upstream bandwidth to consumers as downstream bandwidth. So when any client needs to send higher quality video to other clients the upstream bandwidth may quickly limit the number of streams that can be sent.

So RTMFP is highly scalable when manageable size groups can work independently without any one client consuming too much bandwidth. For example an n-way video conference may have to be limited to 5 to 10 people while an audio conference may hold many more. A low latency data-only game may accommodate even more.

These are n:n use cases. 1:n use cases are usually easier on servers than n:n cases. But for direct communications the individual upstream client bandwidth is the limiting factor rather than the total number of streams a server must relay.

The Future of RTMFP

Only Adobe knows what the next steps are for RTMFP. The sneak peeks we saw at Adobe Max in 2008 were not a commitment to implement application-level multicast. But now that Flash has UDP it's easy to imagine some places Adobe may take RTMFP. Right now RTMFP is limited by each client's upstream bandwidth even for 1:n scenarios or where the number of publishers is much smaller than subscribers. Application-level multicast could make a very large difference in scaling out few-to-many communications.

rtmfpTreeSmall.png

Figure 5. One client sending video to three other clients.

At Michael Thornburgh's Max Session: Sneak Peek: RTMFP Application-Level Multicast in Flash Player Michael demonstrated how each Flash player may one day be able to relay data to other players. Provided each player has enough bandwidth to forward enough data to two or more other players this could be a very scalable way to distribute live experiences to a very large number of people.

rtmfpMeshSmall.png

Figure 6. Application-Level Multicast

However, distributing very high bandwidth streams may not work very well in this sort of scenario. Another possibility that I'd like to see Adobe work on is IP-Level multicast where routers efficiently distribute streams instead of using Flash clients to do so.

I'd also like to see things like distributed shared objects built on top of RTMFP. But those things are for another day. RTMFP is here now via Stratus and will arrive in a future release of FMS. Here are some references you can check out now.

References

The most useful documentation for developers I've read on RTMFP is Jozsef Vass's Devnet article:

Stratus service for developing end-to-end applications using RTMFP in Flash Player

Both Matthew Kaufman and Michael Thornburgh have been answering questions about Stratus and RTMFP on the Stratus section in Adobe Forums:

http://forums.adobe.com/community/adobe_labs/stratus

Speex Codec: http://www.speex.org/

Flash Player 10 Features: http://labs.adobe.com/technologies/flashplayer10/ (It is interesting that the labs page does not mention RTMFP.)

And there doesn't seem to be any mention of RTMFP here either: http://www.adobe.com/products/flashplayer/features/

But there is here: http://www.adobe.com/products/flashplayer/features/all_features/

Which finally takes you here:

http://download.macromedia.com/pub/labs/flashplayer10/flashplayer10_rtmfp_faq_111208.pdf

Advanced Encryption Standard:

http://en.wikipedia.org/wiki/Advanced_Encryption_Standard

Amicima Links:

http://www.adobe.com/special/amicima/

The original Amicima web site is still available courtesy of the WayBackMachine Internet Archive:

http://web.archive.org/web/20061206084719/http://www.amicima.com/developers/documentation.html

The MFP protocol is described in detail here:

http://web.archive.org/web/20060906221358/www.amicima.com/downloads/documentation/protocol-doc-20051216.txt

Adobe Stratus

www.adobe.com/go/stratus

Credits

Thanks to Matthew Kaufman and others at Adobe for answering my questions and to Bill Sanders for cleaning up some ugly sentences. If I've messed up something here it's not their fault!


You might also be interested in:

12 Comments

Hi Brian,

Thanks for putting this together! I'm looking forward to implementing some of this goodness very soon.

Best Regards
Doug

Great article Brian. Thanks! This is definitely an amazing step for FMS.

Wow, that was a long post! FWIW, all of the problems you describe w/ RTMFP (incl peerID swapping / security, failover, detection of upstream limitation, detection of peer connection failure, etc) are handled in the AFCS framework (nee "Cocomo"). You can get started w/ RTMFP+AFCS ASAP here :

http://blogs.adobe.com/collabmethods/2008/12/try_rtmfp_and_clienttoclient_d.html

Very in-depth, thanks for all the detail. We've got to get this working in Red5 one of these days...

That's nice and useful article. I am also building similar kind of application for last 2months. Other then data transfer through NetStream it has audio/video conferencing.
My experience with Stratus is not that good. Sometimes when i don't get connected to Stratus i don't any useful information back i.e. the reason of failure.

And there is one strange problem i noticed few days back. I have 3 users in audio conference. All successfully connected to Stratus. The problem was that user1 and user2 were not able to talk on p2p while user1 was able to talk to user3 on p2p and User2 and user3 were able to communicate properly on p2p.
All 3 users were on different network/ISP. I didn't even get any stream failure event.

Can you please suggest what can be the reason/way out.

Thanks,
Vivek Lakhanpal.

Thanks Vivek,

I think you should always get a NetConnection.Connect.Failed if you can't reach stratus. I tried a few tests and noticed that once in a while I got a NetConnection.Connect.Closed, but that may be because of how I tested... Is that what you experienced? I think it should always be Failed but am not sure what else the event.info object should contain. If Player 10 knows more about the failure it would be useful to have that in another property of the information object.

The problem with some users being able to connect to each other while some cannot is likely because of the type of NATs each user has. One combination (symmetric to port restricted cone) may not work while the other NAT type combinations do.

Since RTMFP is using a NetStream object's API to make the actual RTMFP connect between clients, the NetStream class needs a new event to show that you didn't connect right away... Right now I think it implements the normal RTMP behaviour of simply waiting for the live stream to publish. That makes sense to me for cases where the stream is not published yet and you set up a subscriber first. So I think they need a "waiting for publisher" event or something similar. Or if they know a connection really failed a NetStream.Connect.Failed event.

Yours truly,
-Brian

Thanks for the awesome article. I'm interested in learning more about this part:

----------------------
Sending Data Using NetStream.send()
After a connection has been established between clients, a published stream can be used to reliably send data to the other client.
----------------------

Is it really reliable if it's using UDP? Is there some sort of error checking going on under the hood to ensure all packets are received?

Thanks!

Aaron

Hi Aaron,
Yes, direct client-to-client data transfer is via UDP and Adobe has built in the necessary higher level data transfer checking and retransmission logic to make it reliable.
Cheers,
-Brian

Thank you information

A recording of the FITC talk I gave is now available on Adobe TV's FITC channel:

http://tv.adobe.com/watch/fitc/playertoplayer-communications-with-rtmfp/

Unfortunately the audio level is very low.

The restriction where clients must be connected to the same application instance may be lifted in a future release of FMS. Instead clients can connect directly to each other if they are connected to any application instance running in the same FMS process. Check the docs when FMS with P2P is released.
-Brian

I think you can always reach a Net Connection.Connect.Failed you should not Stratus.Have you experienced this?
widows 8

News Topics

Recommended for You

Got a Question?