Concise media fallback model

[zach]

Proposal for media fallback. In other words, managing the fact not all browsers/plugins can play any video format. Components:

• Centralized way to filter levels.
• Logical scheme for managing file URL shortcuts.
• An overview of all modes and their supported files.
• Incorporating RTMP and custom media providers.

Levels

Content can be fed into the player by using levels. Here's an example:

jwplayer("container").setup({
  playlist: [
    { 
      levels: [ 
        { file: "/assets/video.mp4" },
        { file: "/assets/video.webm" },
        { file: "/assets/video.ogg" }
      ] 
    } 
  ]
 });

Levels inside JW Player are the equivalent of sources inside HTML. When multiple levels are provided, the player will select the first one that can be played back with the current playback mode on the current browser.

If multiple levels can be played back, and they have bitrate/width attributes, the player will resort to bitrate switching to deliver the optimal quality.

Shortcuts

Various shortcuts are available to load files into the JW Player. Consider this example:

jwplayer("container").setup({
  playlist: [
    { 
      levels: [ 
        { file: "/assets/video.mp4" }
      ] 
    } 
  ]
 });

The MP4 video can also be loaded using the following scenario:

jwplayer("container").setup({
  levels: [ 
    { file: "/assets/video.mp4" }
  ]
 });

This scenario leads to the same result:

jwplayer("container").setup({
  playlist: [
    { 
      file: "/assets/video.mp4"
    } 
  ]
 });

And so does this one:

jwplayer("container").setup({
  file: "/assets/video.mp4"
 });

If multiple scenarios are provided in a single embed code (error of the developer), the player will use the following interpretation:

• playlist » levels » file
• playlist » file
• levels » file
• file

Note this means that, if both a playlist and levels are provided, the levels are discarded in favor of a playlist.

Modes

A unique aspect of the JW Player is the integration of its multiple playback modes:

• flash
• html5
• download

The embedder determines which mode to use. It does so by running through the levels of the first playlist item. If at least one of the levels can be played by the preferred playback mode, the embedder will stick to that mode. If not, the embedder will drop down to the next mode, until the correct mode can be found. If no mode can be found, the embedder will not embed the JW Player, essentially leaving the original HTML alone.

Note the embedder only inspects the levels of the first playlistitem. It will not inspect all items in the playlist and provide means to hot-swap between modes inside a playlist. If the first video is MP4 and Flash is selected, a second video in WebM will spawn a player error.

The embedder uses file extensions to determine if a file can be played in a certain mode. Here is an overview of all allowed extensions in the available modes:

• flash: mp4, m4v, mov, f4v, aac, m4a, f4a, flv, mp3, jpg, png, gif.
• html5 (Android, iOS, Safari): mp4, m4v, mov (note: audio and image support coming).
• html5 (Firefox, Opera): ogg, ogv, webm (note: audio and image support coming).
• html5 (Chrome): mp4, m4v, mov, ogg, ogv, webm (note: audio and image support coming).
• download: mp4, m4v, mov, f4v, aac, m4a, f4a, flv, mp3, jpg, png, gif.

Providers

Inside the Flash mode, multiple so-called providers are be available to playback certain media formats. Here is a list:

• video; playing mp4, m4v, mov, f4v, flv, aac, m4a, f4a.
• sound; playing mp3
• http; playing mp4, m4v, f4v and flv.
• rtmp; playing mp4, m4v, mov, f4v, flv, aac, m4a, f4a.
• image; playing png, jpg and gif.
• youtube; playing youtube videos.

The Flash mode also has the option to playback media through pluggable media providers. Some examples are:

• livestream; playing streams from Livestream.com
• matroska; playing .mkv videos

Providers (and their settings) can be setup per level. Here is an example in which RTMP is setup for the first level. Only flash can play it, so html5 and download will fallback to the second or third level:

jwplayer("container").setup({
  playlist: [
    { 
      levels: [ 
        { file: "video.mp4", provider: "rtmp", streamer:"rtmp://example.com/app/" },
        { file: "http://example.com/video.mp4" },
        { file: "http://example.com/video.ogg" },
      ] 
    } 
  ]
 });

Here is another example, with the provider being set to all levels in a playlist item. In this case, both the html5 and download modes cannot play the playlist at all:

jwplayer("container").setup({
  playlist: [
    { 
      levels: [ 
        { file: "video_200.mp4" },
        { file: "video_800.mp4" },
        { file: "video_1600.mp4" },
      ],
      provider: "rtmp",
      streamer:"rtmp://example.com/app/"
    } 
  ]
 });

Again a higher-level shortcut is available. For this example, the player implies the RTMP provider is used for all playlist entries:

jwplayer("container").setup({
  playlist: [
    { file: "some_video.mp4" },
    { file: "another_video.mp4" },
    { file: "last_video.mp4" } 
  ],
  provider: "rtmp",
  streamer:"rtmp://example.com/app/"
 });

Note that media played through rtmp and through custom providers can only be played by Flash. Media played through video, sound, image, http and youtube can be played in html5 and download too.

[pablo]

Agreed; in fact, the embedder should allow complete alternate configs for different player scenarios. Here's one proposed approach:

<script type="text/javascript">
	jwplayer("container").setup({
	flashplayer: "player.swf",
	height: 270,
	width: 480,
	players: [
		{ type: "html5", file: "video1.mp4" },
		{ type: "flash", file: "video2.mp4" },
		{ type: "download" }
		]	
	});
</script>

[jeroen]

Not sure whether this is the scenario we want:

1. Alternative configs for players overall are completely contrary to our approach: the player should look the same on all platforms.
2. This will still not fix the problem where a video can be played in some HML5 browsers and not in others. The most important use case is HTML5/MP4 primary.
3. File / playlist settings should be separately settable from overall player settings. For one, it should be possible to set the different fallback scenarios inside an XML/JWP feed. Think about CDN load balancing, or BOTR.

Perhaps we should revisit the levels and see if we can either do something with that, or add yet another layer (sources) to the player as well.

[zach]

This specifies that one must sort by bandwidth, but what if you want to specify the order of use, ie "MP4 first, webm second, ogg third". This provides no mechanism. The implied data structure is something like:

levels: [{
    bitrate: 15000,
    width: 640,
    height: 480,
    sources: [{ file: "/assets/video.mp4", type:"video/mp4" },
      { file: "/assets/video.webm", type:"video/webm" },
      { file: "/assets/video.ogg", type:"video/ogg" }]
  },
  {
    bitrate: 30000,
    width: 1280,
    height: 720,
    sources: [{ file: "/assets/video-hd.mp4", type:"video/mp4" },
      { file: "/assets/video-hd.webm", type:"video/webm" },
      { file: "/assets/video-hd.ogg", type:"video/ogg" }]
  }
] 

An alternative data structure would allow one to specify order, but it's more verbose:

sources: [
  {
    type:"video/mp4",
    levels: [{
      file: "/assets/video.mp4,
      bitrate: 15000,
      width: 640,
      height: 480
    },{
      file: "/assets/video-hd.mp4,
      bitrate: 30000,
      width: 1280,
      height: 720
    }]},
{
    type:"video/webm",
    levels: [{
      file: "/assets/video.webm,
      bitrate: 15000,
      width: 640,
      height: 480
    },{
      file: "/assets/video-hd.webm,
      bitrate: 30000,
      width: 1280,
      height: 720
    }]},
   {
    type:"video/ogg",
    levels: [{
      file: "/assets/video.ogg,
      bitrate: 15000,
      width: 640,
      height: 480
    },{
      file: "/assets/video-hd.ogg,
      bitrate: 30000,
      width: 1280,
      height: 720
    }]}
] 

Not sure if that is a requirement, but we should determine if that's the case.

[jeroen]

This intentionally provides no mechanism for order of use. The player simply sorts all files it can play in a specific client by bandwidth, and that's all.

Different video formats are solely needed today because there's no format consensus in HTML5. This will go away. Which format to play in which browser is typically something the player will figure out and the publisher should not.

[zach]

From Jeroen:

• Throw away Ogg files if a browser can play WebM (newer FF/Opera)
• Throw away Ogg files if a browser can play MP4 (older Chrome)
• Throw away MP4 files if a browser can play WebM (newer Chrome)

The different formats are for providing alternatives for different browsers. One should not provide different bitrates in different file formats. This generally breaks horribly (see bitrate switching FLV+MP4 in RTMP).

[zach]

I actually think we should prefer MP4 over WebM (only effects the last item) because of the quality:size, but I'd defer to Daniel on that one.

[zach]

Another interesting suggestion: http://www.longtailvideo.com/support/forums/jw-player/setup-issues-and-embedding/16134/wanted-flash-rtmp-streaming-over-akamai-with-fallback-to-html5-for-idevices-playing-a-reduced-rate-mp4

[zach]

Dupes 1186