gnome on kyte codes

GNOME Asia Feels

Tue, 13 Dec 2022 00:00:00 +0000

First and foremost I want to thank GNOME for sponsoring my visit to Kuala Lumpur, Malaysia.
It was nice meeting the people who have similar interests and also who actually build the GNOME I use and love.

All of them absolutely love open source and the work they do. And most of all, they were all excited and happy to showcase their art. Those would be one of the most engaging communications I’ve had in quite a while.

There were people from Singapore, Malaysia, India, Denmark, Hong Kong, Albania, Italy, and USA. I attended all that talks I could and apologize for the ones I missed owing to sleeping late in the morning due to my body clock. Malaysian time is 2h30m ahead of Indian time.
Still I enjoyed them a lot and hope that you all enjoyed my presentation as well.

Speaking of which, some people asked me “what was the presentation actually about?” Yeah.
To squash further FAQs and doubts, here is a simple explanation: I made an addition/extension to an already capable and working app. All it does it cast your screen to a bigger screen if it supports Wi-Fi and isn’t Apple owned. My addition revolved around Google’s Chromecast, while the existing one was Miracast.
Simply put, remove the HDMI cable and just use Wi-Fi in your next presentation.

Here is a pic of me at the conference. Thanks to Jona Azizaj (@jonatoni) for the amazing shot!

GNOME Asia Presentation

And not forgetting the main purpose of the whole trip.

White, Blue and Hearts

Lastly I would have to loved to see my mentors Benjamin and Claudio at the summit. I’ll probably BBB them sometime later :)

GSoC Finale

Sat, 17 Sep 2022 00:00:00 +0000

It definitely was a long journey. Although I didn’t write enough blogs to outline the exact shaping of it, here’s a final report on what has been going on.

The workings of the protocol and the connection specifics have already been discussed before, so let’s keep this short.

Discovery using mDNS
Connection over TCP
Messages are structured in JSON and encoded using protobuf
All the messages have at least a sender id, destination id, namespace, payload type, and the payload itself

Last time we connected to the Chromecast device, we opened a media receiver web app and played a video on it.
Our goal, as it remains, is to cast our desktop/window on the Chromecast display. We can choose from three different streaming protocols to have what we want: DASH, HLS, and Smooth Streaming.

Implementing one of those was the primary goal until a sparse study of Chromium’s code unveiled a namespace called urn:x-cast:com.google.cast.webrtc.
We also used Google Chrome’s (or Chromium’s) tab casting feature and recorded the logs. Looking at the logs virtually revealed the entire message exchange and the steps involved, but it was no documentation by any stretch. We had no idea what the possibilities were or the error cases, for that matter.
While trying to figure out the Chromecast protocol, we noticed many logs in the message exchange files that were not in Chrome’s logs, even with the highest level of verbosity. DLOG and VLOG, for example, did not print out the DLOG, where I suspect D stands for development and V for verbose.
It would have been easier to see what exactly happens with Chrome’s logs. We tried using the dev symbols that come with the Linux distribution, but Arch Linux had some issues at the time.
Nevertheless, I don’t suppose that would’ve worked for the logs.

So the only option left was to read the code and figure it out ourselves. And that’s what we did since no one was successful with the webrtc stream before on the internet.
We came across this directory. It contained all the knowledge there was. The proto files, the json schemas, the header files with all the constants, all the good stuff we needed. From the names of the files itself we knew how to send the stream to Chromecast, except that we didn’t (hint hint something to do with encryption).

Let me loop back to the negotiation part of the WebRTC stream. In normal WebRTC, there is a separate entity that relays messages between the two parties interested. Here that happens to be the sender itself.
We open the app called Chrome Mirroring (app id 0F5096E8). This is for audio and video streaming. There is also one for only audio streaming. You can find a few others here.
Now, when the app is opened, we get a receiver status message, and we start communicating with the app from this point on by addressing the messages to the session id of the app, after connecting to it first.

The first thing we do is send an offer message with the necessary information for the streams we wish to send and the encryption keys, among other details. Here is an example message:

{
 "source_id": "sender-990878",
 "destination_id": "a8a79f97-a63e-4555-baec-53e3f49e6df3",
 "namespace": "urn:x-cast:com.google.cast.webrtc",
 "payload_utf8": {
 "offer": {
 "castMode": "mirroring",
 "receiverGetStatus": true,
 "supportedStreams": [
 {
 "aesIvMask": "1D20EA1C710E5598ECF80FB26ABC57B0",
 "aesKey": "BB0CAE24F76EA1CAC9A383CFB1CFD54E",
 "bitRate": 102000,
 "channels": 2,
 "codecName": "opus",
 "index": 0,
 "receiverRtcpEventLog": true,
 "rtpExtensions": "adaptive_playout_delay",
 "rtpPayloadType": 127,
 "rtpProfile": "cast",
 "sampleRate": 48000,
 "ssrc": 144842,
 "targetDelay": 400,
 "timeBase": "1/48000",
 "type": "audio_source"
 },
 {
 "aesIvMask": "1D20EA1C710E5598ECF80FB26ABC57B0",
 "aesKey": "BB0CAE24F76EA1CAC9A383CFB1CFD54E",
 "codecName": "vp8",
 "index": 1,
 "maxBitRate": 5000000,
 "maxFrameRate": "30000/1000",
 "receiverRtcpEventLog": true,
 "renderMode": "video",
 "resolutions": [{ "height": 1080, "width": 1920 }],
 "rtpExtensions": "adaptive_playout_delay",
 "rtpPayloadType": 96,
 "rtpProfile": "cast",
 "ssrc": 545579,
 "targetDelay": 400,
 "timeBase": "1/90000",
 "type": "video_source"
 }
 ]
 },
 "seqNum": 730137397,
 "type": "OFFER"
 }
}

Note: only the payload_utf8 is structured in JSON, other sibling keys are part of protobuf.

You see that? All the details are right there in the offer message.
Other codecs are also supported, such as h264, vp9, hevc, and aac.
Although most of the things should be self-explanatory, focus on the key ssrc in both the audio and video streams. This is the identifier for the stream.
The AES keys and masks are passed down in the offer message still. The algorithm used however is AES-CTR-128.

We sent an offer. We must receive an answer to it.

{
 "source_id": "a8a79f97-a63e-4555-baec-53e3f49e6df3",
 "destination_id": "sender-990878",
 "namespace": "urn:x-cast:com.google.cast.webrtc",
 "payload_utf8": {
 "answer": {
 "castMode": "mirroring",
 "receiverGetStatus": true,
 "receiverRtcpEventLog": [0, 1],
 "rtpExtensions": [[], []],
 "sendIndexes": [0, 1],
 "ssrcs": [144843, 545580],
 "udpPort": 51810
 },
 "result": "ok",
 "seqNum": 730137397,
 "type": "ANSWER"
 }
}

You’d find it weird that the ssrc values have been incremented compared to what we sent. Apart from that, the keys that deserve our attention are updPort, result, and type. If the result is “ok”, we collect the port number and start streaming RTP packets and exchanging RTCP packets. For streaming RTP packets, we need a controller of the packets, either RTCP or RTSP. Found this out the hard way while messing around with the GStreamer launch command.

After these findings, we had some time left to reorganise the code and handle the edge cases well.
Daily routine was to read docs, write code, test, check logs, and repeat.

Now we had what you can call state management (just a variable and if checks) and some nice JSON string generators.
We jumped into media handling right on. Already having a pipeline to process video and audio in GNOME Network Displays made things easy, but tinkering with it was no easy job until the saviour lent a helping hand. The first few tutorials were all that was needed to get to speed.

The pipeline created for video (only) streaming looked like what follows:

Video Pipeline

You might want to zoom in a little.

This was an unencrypted RTP stream with unencrypted RTCP messages. The only thing to add was an encryption layer over the RTP packets. The plugin srtpenc provided by GStreamer did precisely this, except that it did not do AES-CTR, only AES-ICM and AES-GCM. Going deeper, it used the library libsrtp by Cisco, which, unfortunately, only has implementations for the same two algorithms.

There are two ways to go moving forward:

Add necessary headers in srtpenc and the AES-CTR implementation in libsrtp upstream
Implement a GStreamer plugin that does what both srtpenc and libsrtp do with some help from openssl

Despite that, we were enticed to test the unencrypted stream as is. We had the following broadcast message (and a blank Chromecast display to go with it).

{
 "source_id": "b7319b19-5641-472e-91d8-daeff9746a68",
 "destination_id": "*",
 "namespace": "urn:x-cast:com.google.cast.media",
 "payload_utf8": {
 "requestId": 0,
 "status": [
 {
 "currentTime": 0.0,
 "disableStreamGrouping": true,
 "media": {
 "contentId": "",
 "contentType": "video/webm",
 "metadata": { "metadataType": 0, "title": "Chrome tab casting" },
 "streamType": "LIVE"
 },
 "mediaSessionId": 0,
 "playbackRate": 1.0,
 "playerState": "PLAYING",
 "supportedMediaCommands": 0,
 "volume": { "level": 0.059999998658895493, "muted": false }
 }
 ],
 "type": "MEDIA_STATUS"
 }
}

What does all of this mean for the end-user? Do you wait patiently for us to code the plugin?
Of course not.
Remember that Chromecast also provides three streaming options with the media app, specifically the Deafult Media Receiver app (app id CC1AD845). We went with HLS.

Laptop screen mirrored on Chromecast

There is still some time till this lands on the gitlab. Some cleanup is left.
And the files (playlist file and the segment files) are served from an nginx server (with CORS header). So that needs to be inside the app as well. Soup is the answer to this one.

Now you won’t be able to competitive game on this or use it as a second display, but this surely can handle presentations well.

Huge thanks to my mentors Benjamin Berg (@bberg) and Claudio Wunder (@cwunder).
This project was practically impossible to complete without their help.

PS: In the offer message, we set castMode to mirroring. There is a second option called remoting. From what we gathered, it is a dynamic mode that does not need the codec declarations beforehand and supports RPC messages for some reason. Our best bet is that this has to do with Chrome Remote Desktop and is of no interest to us.

This isn’t the finale. This is the beginning.

Chromecast Protocol

Wed, 03 Aug 2022 00:00:00 +0000

After almost a month of reconnaissance through the study of Chromium’s code, VLC’s code and other people’s attempts, we finally have figured out the Chromecast protocol, and it works flawlessly and reliably!

Big Buck Bunny clip playing on my TV (Chromecast built-in) and the GNOME Network Displays app

Ignoring the previous blog posts, here’s a quick rundown of the Chromecast protocol:

There are two parties involved: the Sender and the Receiver
Typical sender apps are the Chrome browser, and the Android and iOS platforms
Chromecast devices consist of two different receivers (so to say):
- The in-built receiver app that accepts connection requests and handles all other communications with the Sender
- The Web Receiver app is optionally loaded and opened upon request by the Sender, again through messages handled by the former receiver
The first job is to make a Sender app
Now, we discover the available Chromecast devices in the local network through mDNS with the identifier _googlecast._tcp
Next step is to open a TCP and then a TLS connection over that to the Chromecast device on port 8009
After a successful TLS connection:
- The first thing we do is check the authenticity of the Chromecast (we fake it in our app)
- Next, a “virtual connection” is opened to the Chromecast
- At this point, we are eligible to query the status of the Chromecast device (which includes information such as the opened app, volume, and supported namespaces, among other details)
- We can send messages to open an app, Android apps such as the YouTube app or other “media player apps” either custom made or readily available from Google
- Whenever any app is opened or closed, that is, if the state of the Chromecast changes, a broadcast message is sent out to all the connected senders regarding the present status (you may have already seen it on your Android device when a media controller notification pops up if something is being played on Chromecast)
- The apps accept requests to play, pause, change to the next item in the queue, and other similar commands
- In addition to those, we can send requests to play supported contents on the apps (or custom data for custom web receivers – the second type of receivers. These should be hosted on HTTPS domains and registered on Google Cast SDK Developer Console)
Lastly, we close the app, the virtual connection (or as I have recently taken to calling it, VC), and the TCP/TLS connection.
Important point to note here is that an app can keep running even if the VC and the TCP/TLS connections are closed. We need to be explicit about closing the app with a message.

Default Media Receiver App (developed and hosted by Google)

Haah, that was a lot to take in! Fear not, there’s more.

Specifics

This is best explained using logs.
Here are cleaned bits of logs from the GNOME Network Displays app interacting with Chromecast:
(Only the part inside “payload_utf8” is structured in JSON, all parameters outside that like “source_id” are fields in the protobuf schema and only spread out nicely here for presentation purposes – JSON key-pair highlighting)

Sent message:
{
 "source_id": "sender-gnd",
 "destination_id": "receiver-0",
 "namespace_": "urn:x-cast:com.google.cast.tp.deviceauth",
 "payload_type": 1,
 "payload_utf8": (null)
}

Now, if you look closely, the first message sent has a sender_id as sender-gnd and receiver id as receiver-0. The sender_id can be anything prefixed with sender- but I doubt it allows more than six characters after the hyphen. It doesn’t really matter.
The receiver has a default id receiver-0 as found out through Chromium code, and again doesn’t matter since that is all the purpose it serves.

The first message we ever send is the “authentication challenge” (look for the further reading links below) that should ideally contain some binary information and has_binary_payload as 1.
We are not strict with the authenticity of the Chromecast device we are connecting to, so we send an empty request but on the correct namespace. The logs don’t print out the payload_binary field since it is only used once (here) and is empty with length 0 anyways.

Sent message:
{
 "source_id": "sender-gnd",
 "destination_id": "receiver-0",
 "namespace_": "urn:x-cast:com.google.cast.tp.connection",
 "payload_type": 0,
 "payload_utf8": {
 "type": "CONNECT",
 "userAgent": "GND/0.90.5 (X11; Linux x86_64)",
 "connType": 0,
 "origin": {},
 "senderInfo": {
 "sdkType": 2,
 "version": "X11; Linux x86_64",
 "browserVersion": "X11; Linux x86_64",
 "platform": 6,
 "connectionType": 1
 }
 }
}

The second message is the virtual connection message, where we include some random metadata and most importantly the type and connType keys. We indicate the connection type as 0 or strong. This comes from Chromium’s nomenclature for connection types: strong, weak and invisible, out of which weak is not used (don’t ask me why). The connection type must be strong if destination_id receiver-0 is used, as a comment says there. And it seems the default connection type is strong, but we choose not to take any chances.

Sent message:
{
 "source_id": "sender-gnd",
 "destination_id": "receiver-0",
 "namespace_": "urn:x-cast:com.google.cast.receiver",
 "payload_type": 0,
 "payload_utf8": {
 "type": "GET_STATUS",
 "requestId": 2
 }
}

Onto the next request. We manually request a status report with type GET_STATUS on namespace urn:x-cast:com.google.cast.receiver. With sharp eyes and a keen mind, you would have spotted a key called requestId in the payload. This is a unique identifier for every message we send, excluding those on special namespaces, such as the virtual connection or the ping message.
The easiest way to keep it unique is to initialize it to 1 and increment it for every outgoing message. Guess how Chromium and VLC do this.
The requestId field is also present in the received messages and is 0 for all the broadcast messages.

Received message:
{
 "source_id": "receiver-0",
 "destination_id": "sender-gnd",
 "namespace_": "urn:x-cast:com.google.cast.receiver",
 "payload_type": 0,
 "payload_utf8": {
 "requestId": 2,
 "status": {
 "applications": [
 {
 "appId": "2C6A6E3D",
 "appType": "ANDROID_TV",
 "displayName": "YouTube",
 "iconUrl": "",
 "isIdleScreen": false,
 "launchedFromCloud": false,
 "namespaces": [
 { "name": "urn:x-cast:com.google.cast.media" },
 { "name": "urn:x-cast:com.google.cast.system" },
 { "name": "urn:x-cast:com.google.cast.cac" },
 { "name": "urn:x-cast:com.google.youtube.mdx" }
 ],
 "sessionId": "0167f70a-6430-45ab-9714-fd2f09d70b2b",
 "statusText": "Youtube",
 "transportId": "0167f70a-6430-45ab-9714-fd2f09d70b2b",
 "universalAppId": "233637DE"
 }
 ],
 "isActiveInput": true,
 "isStandBy": false,
 "userEq": {},
 "volume": {
 "controlType": "master",
 "level": 0.10000000149011612,
 "muted": false,
 "stepInterval": 0.009999999776482582
 }
 },
 "type": "RECEIVER_STATUS"
 }
}

Hey, we received a message. It has the type RECEIVER_STATUS and is not a broadcast one. We specifically requested this of the Chromecast.

Let’s break it down byte by byte.
The usual fields source_id, destination_id etc. are nothing special, but the payload sure packs some exciting data. We have received the state of the Chromecast where it reports the volume details, stand-by state, userEq (?), and if the Chromecast is the active input (more info), and the running applications.
Only one app is currently running - YouTube (without YouTube Premium). A bunch of namespaces are supported for this particular app, and we should not dive into that for this post.

Sent message:
{
 "source_id": "sender-gnd",
 "destination_id": "receiver-0",
 "namespace_": "urn:x-cast:com.google.cast.receiver",
 "payload_type": 0,
 "payload_utf8": {
 "type": "LAUNCH",
 "appId": "CC1AD845",
 "requestId": 3
 }
}

Too bad we don’t need to watch YouTube videos now. We need to watch our Big Buck Bunny hosted on Google’s servers (not from Google’s CDNs, but Google’s hosted storage, makes sense?).
We launch the Default Media Receiver app (this is the same VLC and the CacTool use) by sending a LAUNCH message with its appId.

Received message:
{
 "source_id": "receiver-0",
 "destination_id": "*",
 "namespace_": "urn:x-cast:com.google.cast.receiver",
 "payload_type": 0,
 "payload_utf8": {
 "requestId": 0,
 "status": {
 "isActiveInput": true,
 "isStandBy": false,
 "userEq": {},
 "volume": {
 "controlType": "master",
 "level": 0.10000000149011612,
 "muted": false,
 "stepInterval": 0.009999999776482582
 }
 },
 "type": "RECEIVER_STATUS"
 }
}

Cool, we received a broadcast message that YouTube has been closed.
(I have configured my Android TV to limit background processes to at most two at a time (makes the system very snappy). This might have caused the YouTube app to close even when it wasn’t needed/intended in this case.)

Received message:
{
 "source_id": "receiver-0",
 "destination_id": "*",
 "namespace_": "urn:x-cast:com.google.cast.receiver",
 "payload_type": 0,
 "payload_utf8": {
 "requestId": 3,
 "status": {
 "applications": [
 {
 "appId": "CC1AD845",
 "appType": "WEB",
 "displayName": "Default Media Receiver",
 "iconUrl": "",
 "isIdleScreen": false,
 "launchedFromCloud": false,
 "namespaces": [
 { "name": "urn:x-cast:com.google.cast.cac" },
 { "name": "urn:x-cast:com.google.cast.debugoverlay" },
 { "name": "urn:x-cast:com.google.cast.broadcast" },
 { "name": "urn:x-cast:com.google.cast.media" }
 ],
 "sessionId": "014f4bf2-2c29-481f-ab50-ef7d6ff210ac",
 "statusText": "Default Media Receiver",
 "transportId": "014f4bf2-2c29-481f-ab50-ef7d6ff210ac",
 "universalAppId": "CC1AD845"
 }
 ],
 "isActiveInput": true,
 "isStandBy": false,
 "userEq": {},
 "volume": {
 "controlType": "master",
 "level": 0.10000000149011612,
 "muted": false,
 "stepInterval": 0.009999999776482582
 }
 },
 "type": "RECEIVER_STATUS"
 }
}

Ah, our media app has taken the stage now. This is a WEB application and runs on Chromium (maybe bare-bones, but it is Chromium).
It accepts messages with the shown namespaces where you’d recognize namespace ending in cac. The one with debugoverlay probably relates to this.
I suspect the broadcast one is responsible for passing on the events to be broadcasted to the internal receiver.
Finally, we are here for the media namespace. We need to send the URL of the media to play on this namespace. But wait, if multiple apps were open, how would the Chromecast or we even figure out for which app a particular message was meant? transportId serves as the receiver_id for those messages.

Sent message:
{
 "source_id": "sender-gnd",
 "destination_id": "014f4bf2-2c29-481f-ab50-ef7d6ff210ac",
 "namespace_": "urn:x-cast:com.google.cast.tp.connection",
 "payload_type": 0,
 "payload_utf8": {
 "type": "CONNECT",
 "userAgent": "GND/0.90.5 (X11; Linux x86_64)",
 "connType": 0,
 "origin": {},
 "senderInfo": {
 "sdkType": 2,
 "version": "X11; Linux x86_64",
 "browserVersion": "X11; Linux x86_64",
 "platform": 6,
 "connectionType": 1
 }
 }
}

We need a second connection to the CC1AD845 app to talk to it (it is what it is).

Received message:
{
 "source_id": "014f4bf2-2c29-481f-ab50-ef7d6ff210ac",
 "destination_id": "*",
 "namespace_": "urn:x-cast:com.google.cast.media",
 "payload_type": 0,
 "payload_utf8": {
 "type": "MEDIA_STATUS",
 "status": [],
 "requestId": 0
 }
}

Now we start receiving broadcast messages from the Default Media Receiver. For now, it’s doing nothing. See the dark image above.
Note that the type of message is MEDIA_STATUS and is on the urn:x-cast:com.google.cast.media namespace.

Sent message:
{
 "source_id": "sender-gnd",
 "destination_id": "014f4bf2-2c29-481f-ab50-ef7d6ff210ac",
 "namespace_": "urn:x-cast:com.google.cast.media",
 "payload_type": 0,
 "payload_utf8": {
 "type": "LOAD",
 "media": {
 "contentId": "https://commondatastorage.googleapis.com/gtv-videos-bucket/CastVideos/mp4/BigBuckBunny.mp4",
 "streamType": "BUFFERED",
 "contentType": "video/mp4"
 },
 "requestId": 4
 }
}

Let’s send our Google-hosted media file’s URL to the media app. Specifying the mime type of the file and the stream type is mandatory. Another option for the stream type could be LIVE, where seek functionality can be enabled or disabled as per wish, and there would be a visual clue on the screen to show that (may not be present on custom-made web receivers).
contentId and contentUrl have some differences but would not matter in our case, Docs for media key.

Received message:
{
 "source_id": "014f4bf2-2c29-481f-ab50-ef7d6ff210ac",
 "destination_id": "*",
 "namespace_": "urn:x-cast:com.google.cast.media",
 "payload_type": 0,
 "payload_utf8": {
 "type": "MEDIA_STATUS",
 "status": [
 {
 "mediaSessionId": 1,
 "playbackRate": 1,
 "playerState": "IDLE",
 "currentTime": 0,
 "supportedMediaCommands": 12303,
 "volume": { "level": 1, "muted": false },
 "media": {
 "contentId": "https://commondatastorage.googleapis.com/gtv-videos-bucket/CastVideos/mp4/BigBuckBunny.mp4",
 "streamType": "BUFFERED",
 "contentType": "video/mp4",
 "mediaCategory": "VIDEO"
 },
 "currentItemId": 1,
 "extendedStatus": {
 "playerState": "LOADING",
 "media": {
 "contentId": "https://commondatastorage.googleapis.com/gtv-videos-bucket/CastVideos/mp4/BigBuckBunny.mp4",
 "streamType": "BUFFERED",
 "contentType": "video/mp4",
 "mediaCategory": "VIDEO"
 },
 "mediaSessionId": 1
 },
 "repeatMode": "REPEAT_OFF"
 }
 ],
 "requestId": 0
 }
}

Yay!! Now would you look at that – it is loading our Google-hosted media file encoded in mp4.

Most of the keys should be self-explanatory :)
For supportedMediaCommands, I just hope that it is not a count.

If you want, you can put a bunch of videos in a queue and let them play or control them using NEXT or PREVIOUS messages or using your hand-held remote or using your IR-equipped Android device or using your Android TV remote (don’t have a clue if this works with other Chromecast devices).
Queuing the videos is left as an assignment for the reader and should be trivial to implement.

Received message:
{
 "source_id": "014f4bf2-2c29-481f-ab50-ef7d6ff210ac",
 "destination_id": "*",
 "namespace_": "urn:x-cast:com.google.cast.media",
 "payload_type": 0,
 "payload_utf8": {
 "type": "MEDIA_STATUS",
 "status": [
 {
 "mediaSessionId": 1,
 "playbackRate": 1,
 "playerState": "BUFFERING",
 "currentTime": 0,
 "supportedMediaCommands": 12303,
 "volume": { "level": 1, "muted": false },
 "currentItemId": 1,
 "repeatMode": "REPEAT_OFF"
 }
 ],
 "requestId": 0
 }
}

No idea why it would not send the complete status message here but whatever, it is buffering the video.

Received message:
{
 "source_id": "014f4bf2-2c29-481f-ab50-ef7d6ff210ac",
 "destination_id": "*",
 "namespace_": "urn:x-cast:com.google.cast.media",
 "payload_type": 0,
 "payload_utf8": {
 "type": "MEDIA_STATUS",
 "status": [
 {
 "mediaSessionId": 1,
 "playbackRate": 1,
 "playerState": "PLAYING",
 "currentTime": 0,
 "supportedMediaCommands": 12303,
 "volume": { "level": 1, "muted": false },
 "activeTrackIds": [],
 "media": {
 "contentId": "https://commondatastorage.googleapis.com/gtv-videos-bucket/CastVideos/mp4/BigBuckBunny.mp4",
 "streamType": "BUFFERED",
 "contentType": "video/mp4",
 "mediaCategory": "VIDEO",
 "duration": 596.474195,
 "tracks": [],
 "breakClips": [],
 "breaks": []
 },
 "currentItemId": 1,
 "items": [
 {
 "itemId": 1,
 "media": {
 "contentId": "https://commondatastorage.googleapis.com/gtv-videos-bucket/CastVideos/mp4/BigBuckBunny.mp4",
 "streamType": "BUFFERED",
 "contentType": "video/mp4",
 "mediaCategory": "VIDEO",
 "duration": 596.474195
 },
 "orderId": 0
 }
 ],
 "repeatMode": "REPEAT_OFF"
 }
 ],
 "requestId": 4
 }
}

Spot the difference challenge. Look at the message two messages above, the one that reports that our Google-hosted media file is being loaded, and then this one we got here.

With sharp eyes and a keen mind, you would have spotted that there are some additional keys here:

activeTrackIds: No idea what this does. See if you can make sense of this.
duration: It inferred this itself. We may provide this in the LOAD request itself though.
tracks: List of track objects (track as in an audio track) – docs
orderId: This is related to queue, see

breakClips and breaks refer to ad breaks in between the video, not even going to pick that up.

One category of messages you can’t see being sent and received are PING and PONG. We send PING messages every 5 seconds to keep the connection alive. A sub-bullet point in our TODO list is to drop the connection if a PONG is not received for 6 seconds in response.

GUADEC 2022

I presented my then latest findings and disclosed all of my plans for this month before all of the users and developers of GNOME located all over the world.
Would you believe this? Someone recorded that and called it Intern Lightning Talk?!
Since it is already sitting in the open internet, it wouldn’t hurt to show it to my dear readers as well (even if it’s not reader"s").

Timestamp to the Revelation
Timestamp to all the Intern Presentations

Follow up on the previous blog post

The bug that stood between a successful communication between the Chromecast and us turns out to have nothing to do with Chromecast itself but was a problem with C pointers. I would like to leave it at that, don’t wanna look stupid on the internet, right?
Maybe we don’t get to discuss Custom Web Receivers after all. In the very verbose logs of Chromium recorded for inspection of its communication with the Chromecast, where all we did was connect to Chromecast and cast a tab, we found out that it uses a receiver app named Chrome Mirroring with id 0F5096E8 and namespace urn:x-cast:com.google.cast.webrtc. This was a rather interesting find. We hopefully wouldn’t need to bother making, registering and hosting a Custom Web Receiver app.

Plan going forward

Clearing out some minor bugs
Handling of Chromecast states
Figuring out the WebRTC messages

Track the project’s progress here

My dear mentors

My mentors, Benjamin Berg and Claudio Wunder were ever present for me, day in and day out.
Stuck somewhere, need advice on some stuff about C, GObject, Chromecast, anything at all, they’d just jump in to my rescue.
This project couldn’t possibly be done without their consistent support.

* All the images used above were stripped off of all metadata using the Metadata Cleaner app
* Further reading on authentication of Chromecast devices: Blog post by Romain Picard
* A Custom Web Receiver in Action
* Chromium’s VirtualConnectionType enum
* Chromecast Web Receiver Docs
* Chromecast Web Sender Docs

Chromecast Connect

Tue, 05 Jul 2022 00:00:00 +0000

It has been a while studying and working with the Chromecast Protocol. This protocol is proprietary, and officially only the Chrome browser, Android, and iOS platforms support casting to a Chromecast-enabled device.
Now, Chromium is open source, and so is Android, but we would not want anything to do with Android for obvious reasons, not to mention this feature may be tucked away in Google Play Services.
So we settle with Chromium which is written in C++. Thankfully almost all of the hard work was done by many other open source projects, notably Node CastV2. Not only this project provides a simple implementation for the Chromecast protocol but also a written explanation of the underlying stuff and how things actually work.
All I need to figure out is how to implement that is C.

A brief overview of the connection and communication is as follows:

We start by discovering the Chromecast devices using mDNS (Avahi) for devices named “googlecast”
Next up we open a socket connection to the selected device on port 8009
We then negotiate a TLS connection on the same socket connection
Now comes the part where we open a “virtual connection” with the device
With that done, the Chromecast device expects a “PING” message every 5 seconds to keep the virtual connection alive
We are in the position to send and receive messages now

The messages are JSON encoded and serialized in protobuf according to the proto file conveniently found in the Chromium source code. Before sending the binary over, we prefix it with 4 bytes of packet length in Big Endian order. JSON is universal, good and easy to understand mostly because it is text, but what about protobuf? It is binary. Handy-dandy library protobuf-c save the day.

Here is what the proto file looks like:

// Copyright 2013 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

syntax = "proto2";

option optimize_for = LITE_RUNTIME;

package castchannel;

message CastMessage {
 enum ProtocolVersion {
 CASTV2_1_0 = 0;
 }
 required ProtocolVersion protocol_version = 1;

 required string source_id = 2;
 required string destination_id = 3;

 required string namespace = 4;

 enum PayloadType {
 STRING = 0;
 BINARY = 1;
 }
 required PayloadType payload_type = 5;

 optional string payload_utf8 = 6;
 optional bytes payload_binary = 7;
}

message AuthChallenge {
}

message AuthResponse {
 required bytes signature = 1;
 required bytes client_auth_certificate = 2;
}

message AuthError {
 enum ErrorType {
 INTERNAL_ERROR = 0;
 NO_TLS = 1;
 }
 required ErrorType error_type = 1;
}

message DeviceAuthMessage {
 optional AuthChallenge challenge = 1;
 optional AuthResponse response = 2;
 optional AuthError error = 3;
}

We can send many types of messages like CONNECT, PING, GET_STATUS, GET_APP_AVAILIBILITY, STOP, CLOSE, etc.
These messages are sent in their respective namespaces (some of them are: urn:x-cast:com.google.cast.tp.connection, urn:x-cast:com.google.cast.tp.heartbeat, urn:x-cast:com.google.cast.receiver) along with some other values such as sender_id, destination_id, payload_type, payload_utf8, etc.
Different namespaces don’t mean we create new channels for each one. They are just like the other values in the protobuf message.

We won’t bother ourselves to verify the authenticity of the Chromecast device we are talking to. Thus all the Auth- messages shall not be entertained. We are much more interested in the CastMessage.
Protobuf-c comes with a lot of helper functions, but it offers yet more. We use protoc, “a code generator that converts Protocol Buffer .proto files to C descriptor code”.

Having the TLS connection set up, we serialize a connect message and send it over.
For some odd reason when I sent { "type": "CONNECT" }, it replied with { "type": "CLOSE" }, replied { "type": "PING" } with { "type": "PONG" }, and { "type": "GET_STATUS" } with { "type": "CLOSE" }.
When I insisted on connection multiple times, it got mad and stopped replying to my messages :/
It still played ping pong though (replied to all the ping messages with pong).

We haven’t come to terms since then. Was it because I left it hanging, and I have a feeling it must have waited for my messages all night?
I tried to sever all ties and make them all over again, shut it down and revived it, changed my address (local IP), and changed my name (sender_id), but who knows what is going inside that little mind of it.

Fear not our next mission involves peering into that mind and getting to the root cause of this confusion (Custom Web Receiver).

We didn’t talk about the Custom Web Receiver this time and the protocols that would be implemented. Hang tight because that is what we are going to do in the coming days.

Many thanks to Benjamin and Claudio! I couldn’t have made it without their help and advice.

Relevant Links:

GNOME Introductory Post

Sun, 12 Jun 2022 00:00:00 +0000

I will be working as a Google Summer of Code Intern at GNOME Foundation, and my project will be to add Chromecast support to the already very cool GNOME Network Displays app that has Miracast support as of now. It can be installed through flatpak as well.

Details

Linux desktop users will be able to cast their screens to Miracast and Chromecast devices. For the Android TVs that feature both of them, it would be wise to opt for the Chromecast path when both the devices share the same router (i.e. are on the same network) since Miracast would require setting up a WiFi-Direct connection to function (may be faster with less latency, may not be worth the trouble).

I believe this will see the light of day with very helpful and experienced mentors, Claudio Wunder and Benjamin Berg.

Progress

We started by looking into the Chromecast documentation (an adventure in itself). So there are two ends to deal with - the sender one and the receiver one.

The receiver end

There are two options here: Styled Media Receiver and Custom Web Receiver.

The Styled Media Receiver does everything for us. It hosts the receiver application (an HTML5 app) and provides all the default styles and functionality. If we want to change the styles and/or the logo, we are free to host them on an HTTPS server, and we’ll be done.

With everything it offers, it has limited still quite adequate media support, including playing video files, audio files, streaming video, and audio, displaying images and opening specific apps on the TV (like YouTube).

Our main job is to “stream” our desktop to the Chromecast device. For this purpose, the Styled Media Receiver supports three streaming protocols:

Dynamic Adaptive Streaming over HTTP (DASH)
Apple’s HTTP Live Streaming (HLS)
Microsoft’s Smooth streaming

We drop considerations for the Smooth Streaming protocol here because of the lack of feature differences compared to the other widely supported protocols, except that HLS and Smooth Streaming support pre-loading by default.

So we end up here with two choices: DASH and HLS. Now, HLS is widely supported; on the other hand, DASH is codec agnostic for both video and audio. In terms of latency, both are said to have similar latencies that depend on the segment duration we decide upon.

As per the HLS’s RFC (linked below), the EXT-X-TARGETDURATION tag in the playlist file (or manifest file) accepts a decimal integer value. It dictates how much the maximum duration of each segment can be (rounded off to the nearest integer). It is suspected to be similar to DASH, although its RFC does not clearly mention so. We better find out by actually experimenting and streaming.

Reducing the segment size too much can cause the bitrate to bump up and the stream quality to degrade thanks to all the added keyframes for each segment, so we need to test what suits this live stream the best.

Next up, we take the discussion to the Custom Web Receiver and the conclusions from some of our tests with the Chromecast streamings using the Command and Control (CaC) Tool, which is a sender app provided by Google for testing and debugging Web Receiver apps.

We want to test out and know more about other protocols not supported out of the box in Chromecast but can work fine with a Custom Web Receiver: SRT, RTSP and WebRTC.

Thanks to Benjamin and Claudio for all the help!

Relevant Links: