A year of building upon companion screen standards

It has been a busy year on our Companion Screens project, with new standards published, high profile demonstrations and other work helping the industry include companion features in TVs. Here is a round-up of progress from the past year.

Companion screens in HbbTV 2

We followed up on standardising synchronisation protocols at DVB by also working to include it in HbbTV 2. This is a standard that profiles the interactive and “connected” features for TVs across Europe. An early draft was released in spring 2015 and subsequently published by ETSI in October. Here in the UK, we are already starting to see some HbbTV 2 features included in the Freeview Play platform.

We made substantial contributions to HbbTV 2. We are very pleased that it incorporates many features needed to create companion screen experiences and media synchronisation, including the DVB synchronisation protocols.

Prototypes

In the meantime, the team was already starting to build prototypes to test and demonstrate these technologies. We created a prototype to stand in for the TV. This uses gstreamer on a Linux PC and can handle broadcast streams, allowing companions to synchronise with it. We also developed a native iOS tablet application that could synchronise audio, video and embedded web content with the TV.

During this process we have developed a better understanding of how to write these kinds of apps. We defined a software approach to representing the progress of timelines and clocks. We also developed modules that use the DVB protocols to control their timing and other modules that listen to them to drive media playback. This is evolving into a useful modular framework. Soon we hope to share the code and this knowledge both within the BBC and more widely.

We also prototyped other features from HbbTV 2, such as how a companion discovers the TV on the home network (this is based on DIAL) and a mechanism that allows apps on the TV to communicate with apps on the companion.

As our prototypes came together we were also able to use the tools we devised for timing measurement and calibration to edge towards, and eventually achieve, frame-accurate synchronisation.

Demo at IBC 2015

The culmination of this was a demonstration at the IBC exhibition in Amsterdam in September 2015. This was our first big opportunity to show the prototypes publicly. Thanks to DVB who hosted us on their stand.

This was a great opportunity to show the industry that this technology really works and performs well. The protocols and calibration processes did their job and delivered rock-solid frame accurate synchronisation all weekend despite a crowded WiFi environment.

The demonstration also showed that two independent organisations (BBC and Dolby) could interpret the specifications in the same way and succeed in creating systems that interoperate. Doing this shows that the specifications are fit for purpose and gives the industry greater confidence in the technology.

Also, with the whole industry there, IBC was a great place to reach a wide audience and get them thinking about the possibilities that this technology offers. 

We showed a range of use cases, including how alternative commentaries or Audio Description could be provided in a personalised way via a smartphone or tablet. We also showed the potential for providing extra information about programmes as well as linking through to online resources relevant to what is happening on screen.

IRT demo at IFA and IBC 2016

Our colleagues at IRT have been hard at work developing their own client implementations. They showed an early version of this at IFA 2016 and IBC 2016 using a prototype TV and Android tablets. They demonstrated how a user could use a tablet application to start content on a TV and then have the tablet synchronised to it, providing alternative language or clean audio. It is great to see others also working on bringing this technology closer to market. 

#HbbTV 2.0 : Neue Optionen für Audiodeskription, Clean Audio und Originalton via Tablet PC #ardifa16#IFA2016pic.twitter.com/5bJBRNCp0I

— IRT (@IRTpresse) September 3, 2016

Conformance Testing

The work on HbbTV 2 did not stop with the publication of the specification and building prototypes. We also contributed our expertise to help define the test harness system and suites of conformance tests. These help to ensure that TVs implement all the new technologies correctly. The HbbTV 2 Test Suite will be available to the wider industry in the coming months.

We have also created our own supplementary test materials and shared them with the industry. This helps with both their development and testing, particularly for features that are of most interest to the BBC. We crafted broadcast streams containing the extra data that is used for synchronisation, including timelines carried in a new format called TEMI that has been recently defined by MPEG.

Developing the User Experience

Getting the technology in place is only part of the story. The BBC must understand how to use it to create compelling experiences that match user needs. We are therefore also carrying out user experience research by putting some of our prototypes into the hands of viewers and observing how they respond.

This work is being done through our involvement in the 2-IMMERSE collaborative research project. This project is building on HbbTV 2 by developing and trialling systems and tools for creating flexible experiences that are spread across multiple devices in the home, and even between homes. In addition to our user experience work, we are also contributing our work on companion app prototypes and our expertise in HbbTV 2.

We have been sharing our work with the academic community too. A paper titled “Enabling Frame-Accurate Synchronised Companion Screen Experiences” was presented at the TVX 2016 conference where it was awarded an “Honourable Mention”.

As this technology gets closer to arriving in people's homes, we are excited by the possibilities it offers! But of course, there is still much for us to do to prepare and we look forward to sharing more of this with you.