Introducing hls.js

Today, we are glad to open-source and release one of the key components embedded in our latest HTML5 Player.
This component is our streaming library, which handles interactions between your browser, our HTML5 player and our videos.

What is hls.js ?

hls.js is a JavaScript library which implements an HTTP Live Streaming client. It relies on HTML5 video and MediaSource Extensions for playback.
hls.js can also work directly on top of a standard HTML <video> element. It does not require any player to run on top of it.

You can view it in action here, running on top of a standard HTML <video> element :

The source repository is open-source and can be accessed here :

Why are we developing hls.js ?

Nowadays, web video playback on Desktop still remains Flash-centric. Indeed, Flash was the first to provide full control of the audio/video pipeline with Flash Player 10.1 in June 2010 and the introduction of a new API to control NetStream in data-generation mode. But Flash's days may be numbered... for security reasons: Flash security holes (5 security holes found per week) are becoming a serious concern for the web industry. Fortunately, HTML5 is now becoming ready to replace Flash for video playback, thanks to the introduction of MediaSource Extension. MediaSource, as described in the W3C spec, adds buffer-source based options to HTML5 video for streaming support. It also enables the management of all of the streaming logic via JavaScript. See browser compatibility here

As you can see, HTML5 MediaSource Extension is now becoming stable across browsers, and the timing is perfect to begin switching our users still using Flash to HTML5!

For this switch to happen, we needed a reliable JavaScript streaming component, that met the following criteria:

  • this component should support adaptive streaming
    • it should allow complete and fine-grained control of the adaptive streaming logic
    • adaptive switching algorithm should remain simple and easily tweakable
  • it should support both VoD and Live (with DVR) streams
  • it should support the current Dailymotion streaming delivery backend
  • it should be player-independent and expose a simple API
  • it should be resilient to errors (support redundant streams, failover on network and media errors, ...)
  • it should provide streaming analytics to gain knowledge about User Experience.
  • it should work cross browser / cross platform.
  • it should be optimized for performance and size
  • it should be easily extensible
  • and allow for P2P extensions (for hybrid CDN-P2P architecture)

Why choose HLS as a streaming protocol?

We would like to avoid protocol fragmentation and align viewers on a unique streaming protocol. This is necessary for the sake of simplicity:

  • to keep our system simple,
  • our codebase small,
  • and avoid redundant development.

This also helps to reduce delivery cost, as using one protocol leverages cache usage.

Since HLS is supported on iOS devices (on which the MediaSource Extension is not supported), HLS remains the lowest common denominator as of today. The HLS protocol currently covers our needs (adaptive streaming, free to air video, audio and video multiplexed in the same fragments) and it is already supported by our delivery infrastructure.

In any case, handling the streaming protocol, whether it be HLS/DASH/whatever, is a small part of the picture (even if it is remains a complex one, especially the transmuxing part), but the complexity of the implementation doesn't come from the protocol/signaling itself. The biggest efforts deal with the logic concerning buffer handling, error/failover handling, metrics reporting and so on.

Why rewrite from scratch and not reuse or contribute to any other existing library?

The 3 options we had were to either:

  • purchase
  • reuse an existing FOSS component
  • or write one new component from scratch

Purchasing was not an option, as nothing was available at the time we started the project. We then evaluated the different FOSS streaming libraries available :

  • videojs-contrib-hls
    • HLS/MSE support is available in a development branch but this component is deeply linked to videojs, thus not player independent. MSE support was really alpha.
  • hasplayer.js
    • HLS support is available, but the implementation is based on an old dash.js fork. It also supports DASH and MSS and as a consequence its size is significant and the overall dash.js framework is quite complex to come to grips with.

For these reasons, we decided to rewrite a streaming component from scratch.

Challenges we faced

The main challenges are due to Browser compatibility:

Although the MediaSource specification provides some guidelines on the way the API should behave, the devil is in the details, and having a streaming library that behaves identically across browsers is quite challenging.

Below are some of the areas in which we suffered:

audio codec switching

Some of our adaptive streams are encoded using two different audio codecs:

  • HE-AAC audio codec for low definition
  • AAC for medium/high definition

when switching from a low definition fragment to a medium or high definition fragment or vice versa, we want the audio decoder to switch seamlessly (without glitches) from one to another. This works perfectly with the Flash Player. HE-AAC has been designed to be backward compatible with standard AAC.

But this is audio switching is not straightforward to handle with MediaSource. Some browsers are incapable of switching properly: this raises a video error and playback freezes. In this case the only way to recover is to tear down and re-instantiate MediaSource... and this can result in a small audio/video glitch.

buffer handling
buffer eviction

Each browser has its own buffer eviction algorithm when the SourceBuffer maximum size is reached. The eviction algorithm is browser dependent and not specified in the W3C spec i.e if your library is buffering too much, your browser may start flushing near your current playback position.

forced buffer flushing

Browsers also behave differently upon calling SourceBuffer.remove(): this method is quite useful to partially flush the buffer and we use it to handle smooth quality level transitions.

SourceBuffer.remove() does not always remove the exact expected time range. Also some browsers are unable to flush multiple buffer ranges at once.

Transmuxing performance with High Definition Streams

HLS streams are multiplexed using an MPEG-2 Transport Stream, whose format is (usually) not supported by web browsers supporting the MediaSource Extension.

hls.js works by transmuxing MPEG-2 Transport Streams into ISO BMFF (MP4) fragments, which is supported by all browsers.

This transmuxing is performed as follow:

  • TS packets are extracted from the fragment payload, downloaded as Uint8Array.
  • PES packets are then decapsulated from TS packets.
  • Audio/Video data and timestamps are then decapsulated from PES packets
    • Presentation Time Stamp / Decoding Time Stamp (PTS/DTS)
    • AVC NAL units
    • AAC/HE-AAC frames

This transmuxing can be time-consuming, especially on High Definition content.
In order to guarantee correct performance, hls.js performs this task asynchronously, using Web Worker if the feature is available (check here) in the browser.
Web Worker offloads this heavy processing to a separate thread so that transmuxing is not a performance bottleneck (no frame drops, etc.)

Why Open Source it ?

Based on the experience we acquired by maintaining, we knew that OSS brings better quality and interoperability (across browsers, etc.)
In the end, we know that open sourcing will also benefit our users!

We would love to hear from you and help you begin to use and contribute to hls.js!