Real Time Communications Bits

WebRTC Architecture that keeps working when us-east-1 goes down

2025-10-21T09:01:00.000-07:00

This week AWS had an outage in the northern Virginia region, probably the most commonly used region and many of the most popular services in the world were affected.

It was at least the third time in five years that AWS’s northern Virginia cluster, known as US-EAST-1, contributed to a major internet meltdown.

Every service is different, and nowadays, with so many dependencies and coupling, it is hard not to be affected by an incident like this. However, specifically for WebRTC platforms, it should be relatively straightforward to handle cases where one specific AWS region goes down.

Basic WebRTC Architecture

The basic architecture of most of WebRTC platforms comprises four components:

Load balancing service: The client application’s initial point of contact, selecting the appropriate media server for a session.
Database: Stores server information (status, load, location) and session-to-server mappings, often using a fast database like Redis.
Signaling service: Manages session state and forwards control messages. For this discussion, it’s assumed to be part of the media server and ignored in the following diagrams.
Media forwarding service: The core component, routing audio and video among participants. It’s typically stateless and reports its status.

Press enter or click to view image in full size

The whole solution usually includes other components to manage things like recording or firewall traversal but those are ignored for the purpose of this discussion and anyway usually they are stateless and easy to make them robust in case of a network issue.

Distributed WebRTC Architecture

When you want to provide the best possible quality, you deploy media servers in multiple regions, and your architecture typically looks like this:

Press enter or click to view image in full size

This is the most common deployment in my experience, but it is obvious that there is a central point of failure that can be the database or the Load Balancing service + Database depending on the way you deploy it.

Robust distributed architecture

If the Load Balancing service and the Database are the problem in case of outage in the region where they are located the solution looks obvious, let’s just duplicate them.

Replicating the Load Balancing service is easy, deploy it to as many regions as needed (ideally all of them) and use one of these approaches to failover to a different Load Balancer in case there is an outage in your closest region:

Rely on health checks and short DNS timeouts. That way if there is an outage the DNS servers will mark some region as unavailable and route client requests to a different one.. This can be in problem in case of an outage affecting the DNS service (like Route53).
Have clients configured to fallback to different url (for example api.domain.com and api-backup.domain.com)

Replicating the database is a little bit harder, the most common approaches are these ones:

The easiest way is to do it in an active/passive way, where multiple instances of the database are available, but only one is used at a given time, with the others just replicating the information. In case of an outage, we move the queries manually or automatically with some monitoring process to use one of those replicas instead.
A more evolved approach is active/active using sharding. It is more complex but possible and gives the additional advantage of distributing the load.
You can use an already distributed database like Azure Spanner. This can be tricky in case this distributed database has some central component, as apparently happened with DynamoDB in the case of the last AWS outage.

This is how the final architecture looks like:

Press enter or click to view image in full size

With the solutions proposed above your WebRTC infrastructure should be resilient to all the problems affecting a single region in your cloud provider.

It is possible to go further and for example deploy on multiple providers (AWS, GCP, Azure…) or using your own infrastructure but that’s not something I would recommend unless it is a really critical piece of software or you need it for other reasons like cost or compliance.

Reflection+Amplification attacks abusing TURN servers

2025-06-09T03:11:00.000-07:00

Last week there were many messages in the coturn issue tracker about TURN instances being blocked by some cloud providers because it was detected that those servers were being used to attack other hosts.

This is not new as Wire had already reported it and even suggested a mitigation some months ago.

The attacks being carried out in this case are based on two properties of TURN servers:

Reflection: A TURN server, by design, sends the responses to the TURN messages to the source IP of the Request. If an attacker is able to change his source IP address (spoofing) then it can direct the response of his TURN request to any other host he wants.
Amplification: TURN responses are usually bigger than the corresponding requests. This is especially problematic for authentication error responses that must include a NONCE value to be used to attempt authentication properly based on the standard authentication defined in the protocol.

We can see a real example of the second property in a commercial TURN service were the amplification factor is 2x (in other providers this amplification factor is even bigger):

Size of Allocation Request and Response in a TURN server

With those two properties in mind this is how the attack looks like:

The root cause of this problem is that the Unauthorized responses sent by the TURN server are bigger than the requests, otherwise the attacker wouldn’t get any benefit from the TURN server and would send the request directly to the attacked host. With the default coturn configuration (and similarly in other servers) this is around 2x-2.5x increase in size depending on your software name and realm configuration, so there is a big incentive to use coturn to reflect the attack instead of sending it directly.

The mitigation suggested by the Wire team was based on limiting the number of unauthenticated responses sent to a specific IP, and that’s effective as they have tested in production, and probably this featurewill be integarted in a way or another in coturn, but this approach has two limitations: 1) there can be other responses outside of 401s that could be used also for these attacks and 2) if the attacker is attacking a range of IPs instead of a single one this mechanism wouldn’t block it without adding more elaborated logic.

With all that in mind lets review the format of those error responses and see if we can optimize them. This is an example again from the same commercial TURN provider that uses coturn opensource server:

Looking at the content of those 401-Unauthorized responses we can see that we could try to reduce the size and because of that the amplification factor. These are the attributes sent in the response and what we could do with them:

24 bytes of SOFTWARE attribute → You should just remove it given that it is an optional attribute (this was already configurable in coturn, now with the latest version the default is off)
12 bytes of error phrase in ERROR-CODE → The error phrase is not required and it provides little value so we can just remove it (PR pending review)
16 bytes of NONCE → This probably could be shorter, specially when the type of credentials used are already valid for a short period of time, but the impact of changing this is not fully clear and requires more carefull review.
12 bytes of REALM → The realm is not providing any value, at least for WebRTC use cases, so we should make it as short as possible (for example just ”t”).

With those suggested changes the message should go from 142 bytes to 95 making the amplification “only” 1.35x that hopefully makes it much less worth it.

In addition to those suggestions, if you can change firewall configuration for the TURN incoming traffic following rules will help blocking illegal traffic:

Block traffic from illegitimate ports (for example privileged ports — below 1024 — cannot be used by normal clients and as such any traffic from those ports is illegitimate)
STUN/TURN (and underlying UDP) allow for very long messages. But in reality, messages longer than MTU are not used, at least for WebRTC use cases. That means that any message fragment is not a valid STUN/TURN message and should be blocked.
Monitor 401 errors from specific IPs and use fail2ban utility to block those IPs.

An interesting idea suggested by Philipp Hancke is to have STUN/TURN clients in browsers to send padding data in the request. TURN servers could ignore incoming packets without this attribute. This would make the protocol slightly less efficient but it bring the amplification factor below one. The other option could be to change the way authentication works in TURN to remove the need of those nonce attributes or rethink the authentication and authorization flows.

Let’s see how these mitigations go and if the attacks keep going maybe we should consider that. And remember to upgrade to the latest version 4.7.0 (which enables multiple options that helps mitigate this type of attacks by default) or at least review your configuration to apply some of the recommendations in this post.

Thank you very much to Philipp Hancke and Pavel Punsky for the review and contributions to this post. 🙇

You can follow me in Twitter if you are interested in Real Time Communications or want to comment about the topic of this post.

Another sneaky WebRTC optimisation only known by Google Meet (RemoteEstimate RTCP packets)

2025-03-05T04:07:00.000-08:00

Google Meet has consistently delivered superior quality among WebRTC applications (at least for web applications). This is especially true compared with typical open-source solutions, but it stands even for most commercial solutions.

The reason is that Google has the team that understands the media stack they have built very well and can make it behave in ways that solve their problems in the browser with knobs hidden in WebRTC that only they are aware of.

Some of us still remember how simulcast support was added to WebRTC approximately 12 years ago with SDP munging and without any documentation or note behind the x-google-conference flag, or how “Audio Network Adaptation” was added with a secret string encoding an undocumented protobuf schema to tune settings to improve audio quality.

Today I was trying to debug why a WebRTC application had a less stable bandwidth estimation than Google Meet. The scenario was quite simple: in a perfect network with plenty of bandwidth, add 50ms of extra latency suddenly. You see in all the WebRTC applications the bandwidth estimation calculated by the sending browser drops (left graph), while with Google Meet (right graph) it stays perfectly stable.

After spending hours adding logs to Chrome code and debugging the browser’s behaviour under those conditions, I ended up in the AimdRateControl class, which decides when to decrease the bandwidth estimation. For all applications (Google Meet and others), it was entering that point with a lower estimation. Surprisingly, for Google Meet, even if the estimation was lower, it would stay with the previous estimation. The reason is in this code clamping the final estimation.

new_bitrate = std::min(
 current_bitrate_, 
 std::max(new_bitrate, network_estimate_->link_capacity_lower * beta_)
);

So Chrome will never reduce the bitrate below “network_estimate_->link_capacity_lower * 0.85”. But what is network_estimate_->link_capacity_lower ?!?!

Some more debugging and I figured out that is some information sent by the server in proprietary RTCP messages of type APP with name “goog” and subtype 13.

bool RTCPReceiver::HandleApp(const rtcp::CommonHeader& rtcp_block,
PacketInformation* packet_information) {
  rtcp::App app;
  if (!app.Parse(rtcp_block)) {
    return false;
  }
  if (app.name() == rtcp::RemoteEstimate::kName &&
    app.sub_type() == rtcp::RemoteEstimate::kSubType) {
    rtcp::RemoteEstimate estimate(std::move(app));
    if (estimate.ParseData()) {
      packet_information->network_state_estimate = estimate.estimate();
    }
  }
  return true;
}

This extension is described with the following words inside the codebase:

// The RemoteEstimate packet provides network estimation results from the
// receive side. This functionality is experimental and subject to change
// without notice.

This RTCP message was added back in 2019. Back then it was wired up to the SDP but even though we have looked at Google Meet frequently since then we never saw the corresponding SDP and it is still not there today.

So the server is sending every 50ms a RTCP App message with a bandwidth estimation calculated from the server side.

That RTCP RemoteEstimation message includes two values encoded inside:

link_capacity_lower
link_capacity_upper

The lower value works as a cap to not react so fast when the sender side bandwidth estimation detects some sudden drop in the estimation but the server side estimation is still stable. The use of the upper value is disabled behind a flag in AimdRateControl but maybe it is used in other parts of WebRTC codebase.

And now the even harder question. How are those values calculated? They provide two different values and they look like different bandwidth estimations.

After sharing this with Philipp Hancke he suggested to check if it works with abs-send-time extension disabled and he was right, it doesn’t work without it so it looks similar to the old REMB implementation but providing two values (upper and lower) instead of a single one that acts as just a maximum for the sender.

In essence the reliance on abs-send-time suggests that a remote bitrate estimator is running on the server. Looking at this issue and the old emails this has quite possibly been happening since 2019, hidden in plain sight.

To verify this hypothesis, I recompiled Chrome to disregard those proprietary RTCP App messages and, there it was! Google Meet now performs as well as (or as poorly as) other implementations under the same testing network conditions.

So it looks like Google went from receiver-side bandwidth estimation (REMB; RTCP REMB could have been used to transport an upper capacity but not a lower one) to sender-side bandwidth estimation (transport feedback) to just do both :)

The necessity of the RemoteEstimate extension for quality in most scenarios is unclear. Probably is not critical given that no WebRTC application out there uses it apart from Google Meet as far as we know but it looks like it provides some value so other WebRTC applications should probably go and copy it. We only need to figure out how to calculate the lower and upper values and send them every 50ms to the sender.

Thank you very much to Philipp Hancke for his review and feedback.

You can follow me in Twitter if you are interested in Real Time Communications or want to comment about the topic of this psot.

OpenAI WebRTC API Review

2024-12-22T00:44:00.000-08:00

There is a new interface added to OpenAI RealTime models. Now it supports WebRTC!

Given the people working on it I'm sure it has to be great so as usual let’s take a look and see what is under the hood in terms of audio transmission.

Signalling or Establishment of the connection

There are two options for the establishment of a RealTime session with the OpenAI servers:

WebSocket signalling: much nicer API without ugly SDPs involved but less suited for public networks.
HTTP/WebRTC signalling: has an uglier API including SDP offer/answer negotiations but can work well in real networks that is critical for most of the use cases.

In the rest of the post we will focus only in the later (HTTP/WebRTC) that is the most interesting one.

Authentication

The first step to use these RealTime APIs sending audio data directly from clients to OpenAI servers is to obtain an ephemeral key using you OpenAI API Secret. This is a simple HTTP request that for testing you can do from the command line:

`curl -X POST -H “Content-Type: application/json” -H “Authorization: Bearer OPENAI_API_SECRET” -d ‘{“model”: “gpt-4o-realtime-preview-2024–12–17”, “voice”: “verse”}’ https://api.openai.com/v1/realtime/sessions`

One interesting thing is that those credentials retrieved are really short lived and valid only for 60 seconds.

The response of that API also include things like “audio_format”: “pcm16”, that look outdated as the API also support "opus" now with the new WebRTC API and some parameters about voice and silence detection that i'm not sure if they include some way to tune them by developers.

Connectivity

The WebRTC connection is established using UDP and only UDP. There are no TURN servers or ICE-TCP/SSLTCP candidates so don’t expect this connection to work in restrictive corporate networks.

This could be ok as you can fallback to websockets API in those cases, although in my opinion OpenAI should provide an SDK doing that fallback or just include iceServers or TCP candidates in the Answer.

All the examples also ignore the fact that the connection can get disconnected at some point and you will need to create it again. That’s another small feature that could be hidden in a future OpenAI SDK if needed.

As an interesting fact the UDP traffic uses the old trick of reusing the port 3478 to try to pass some firewalls that have STUN traffic open. Also in my case I’m in Europe and the WebRTC server assigned to me is in the US but I guess that will change once this API is out of Beta.

Audio transmission

As expected with a WebRTC the audio transmission is done using Opus Codec at 50 packets per second. PCM and G722 are also negotiated so probably they also work although I don't see a reason to include them.

The whole SDP Answer from the Server can be seen in the following snippet:

v=- 5732503706549353267 1734545489 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1
a=extmap-allow-mixed
a=msid-semantic: WMS realtimeapi
m=audio 3478 UDP/TLS/RTP/SAVPF 111 9 0 8
c=IN IP4 13.65.63.209
a=rtcp:3478 IN IP4 13.65.63.209
a=candidate:3461111247 1 udp 2130706431 13.65.63.209 3478 typ host generation 0
a=candidate:3461111247 2 udp 2130706431 13.65.63.209 3478 typ host generation 0
a=ice-ufrag:WGbWWofbaCZKJNIq
a=ice-pwd:SLqkjRNAwXPdKZsRmdlukcvcoPqUEsPw
a=fingerprint:sha-256 D7:A3:67:D9:49:5E:81:C5:A8:41:15:C1:A4:7A:FC:6F:51:8D:6F:6F:81:C2:A3:EB:BC:65:C0:B2:8A:26:4E:A7
a=setup:active
a=mid:0
a=extmap:3 <http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01>
a=sendrecv
a=msid:realtimeapi audio
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:9 G722/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=ssrc:2472058689 cname:realtimeapi
a=ssrc:2472058689 msid:realtimeapi audio
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:WGbWWofbaCZKJNIq
a=ice-pwd:SLqkjRNAwXPdKZsRmdlukcvcoPqUEsPw
a=fingerprint:sha-256 D7:A3:67:D9:49:5E:81:C5:A8:41:15:C1:A4:7A:FC:6F:51:8D:6F:6F:81:C2:A3:EB:BC:65:C0:B2:8A:26:4E:A7
a=setup:active
a=mid:1
a=sctp-port:5000

I was expecting discontinuous transmission enabled for audio to reduce the bandwidth usage given that the user is probably in silence most of the time but I guess the API is not yet optimised. You can see how it sends and receives 50 packets per second all the time:

The other option I would consider is reducing the uplink audio bitrate given that llm probably doesn’t need an amazing audio quality to understand the content and the emotions associated with it.

There is also transport-cc RTP extension negotiated that doesn’t have much use in this audio-only use case and probably could be eliminated to remove some tiny overhead in the transmission.

In terms of audio reliability, opus inbandfec is the only addition. There is no RED or Retransmissions. This is probably fine for this specific use case given that LLM should be able to infer missing parts of the sentences by the context. Still I don't see any reason to not include at least retransmissions.

Data

There is a reliable datachannel established to receive events from the server side. This is a great feature and provides a lot of flexibility as well as a channel to control things dynamically (like the voice used) without having to recreate the connection.

As soon as you start sending and receiving audio there are a lot of events received. Some of the most interesting ones could be:

input_audio_buffer.speech_started: Notifies when the user starts
speaking input_audio_buffer.speech_stopped: Notifies when the user
starts speaking response.audio_transcript.delta: Includes the deltas of the text that you are receiving also by audio.

It is interesting how (as expected) you receive the text much faster than the audio. That makes me wondering if for some use cases reading the text is not a more effective “conversation” than hearing the voice.

Conclusions

It is great to see new products adopting WebRTC and providing real time interfaces. The functionality looks great for a Beta version and I’m sure it will evolve quickly.

I’m still not convinced Opus and WebRTC are the best candidates for this specific type of use case where you are talking to a “machine” instead of "having a meeting" and I hope in the future we end up using more custom codecs and transports tuned for this use case but I could be wrong, and in any case this is a very simple and a great approach for the time being.

Target Bitrates vs Max Bitrates

2024-11-25T06:47:00.000-08:00

Not all the simulcast layers have the same encoding quality

When using simulcast video encoding with WebRTC, the encoder generates different versions or layers of the video input with varying resolutions. Using this techniques a multiparty video server (SFU) can adapt the video that each participant in a room receives based on factors such as available bandwidth, CPU/battery level, or the rendering size of those videos in each receiver.

How simulcast works with an SFU forwarding layers selectively

These different versions of the video have varying resolutions, but what about their encoding quality? For example, if a user is receiving a video and rendering it in a window of 640x360, would he get the same quality if he receives the 640x360 layer as if he receives the highest layer of 1280x720?

To answer this question about the quality of each resolution, we can examine first the bitrates used by each. But the interesting thing is that the bitrate of each resolution is not always the same and this is where we need to distinguish between min bitrate, target bitrate and max bitrate. Each resolution has those three parameters associated with it:

Min bitrate is the bitrate needed to be available to enable this layer.
Target bitrate is the bitrate used in ideal conditions when there are other higher layers enabled.
Max bitrate is the bitrate used in ideal conditions when this is the highest layer enabled.

For example, here are the values configured for different simulcast layers in VP8 (you can check them for other layers and codecs at simulcast.cc):

You can see how these things work with a quick test enabling and disabling different simulcast layers (in this case I used the simulcast-playground). See in the graphs the bitrates of each layer in different conditions. On the top is the lowest resolution layer, then the middle layer and in the bottom is the highest resolution layer.

Initially all the layers are enabled and they are sent: 320x180 (150kbps), 640x360 (500kbps), 1280x720 (2500kbps)
When the highest layer is disabled, part of that bitrate is used for the middle layer and they are sent as: 320x180 (150kbps), 640x360 (700kbps)
When the middle layer is also disabled only the lowest is sent as: 320x180 (200kbps)

Is this good or bad? 🙂

For the original purpose of simulcast — adapting video received to different receivers with varying bandwidth availability — this works well, since lower quality is expected when bandwidth is limited.

For other purposes, the results vary. With default bitrate settings, forwarding a lower layer means participants receive both lower resolution and reduced quality.

To measure the quality impact, I tested VMAF scores by encoding videos with target and max bitrates using ffmpeg’s real-time encoding settings. While this method differs slightly from WebRTC’s configuration, it provides a reasonable comparison.

Looking at the lowest resolution (320x180), we find good quality above 100kbps, with notably better results at the max bitrate of 200kbps compared to the target bitrate of 150kbps (a difference of 2.5 VMAF points).

This quality degradation affects both simulcast and SVC configurations. SVC faces an additional challenge: the bitrates for different resolutions are hardcoded in WebRTC and cannot be modified using the setParameters API.

In summary, a layer’s bitrate varies depending on whether higher layers are enabled. While requesting lower resolutions works well in most cases, remember that lower layers have both reduced resolution and lower encoding quality.

The Impact of Bursty Packet Loss on Audio Quality in WebRTC

2024-05-03T00:27:00.000-07:00

Ensuring high-quality audio in WebRTC encounters a pivotal challenge amidst less than ideal network conditions, predominantly driven by the burstiness of packet loss. This phenomenon is prevalent in congested networks, areas with low mobile coverage, and public Wi-Fi setups.

Within the WebRTC framework, an array of strategies exists to mitigate packet loss, yet their efficacy varies depending on the specific network dynamics. Among the most prevalent techniques are:

OPUS Forward Error Correction (FEC): Each audio packet incorporates low-bitrate data from preceding packet, facilitating potential recovery in the event of a single packet lost.
Packet Retransmissions: Leveraging standard NACK/RTX mechanisms, the receiver requests retransmission upon detecting packet sequence gaps.
Packet Duplication: Sending multiple instances of the same packet aims to compensate for potential losses. It is like sending preemptive retransmissions to mitigate the impact of potential packet loss.
Redundancy (RED): This technique entails embedding encoded audio for the current packet along with copies of previous packets within each packet, typically following the RFC2198 format.
Packet Loss Concealment: Employing various concealment techniques such as temporal audio stretching or neural network approximations to mitigate the impact of lost packets.
Deep Redundancy (DRED): Integrating a neural network encoder to provide lower bitrate copies of previous packets along with the current packet encoded with high quality as usual.

However, the effectiveness of these techniques hinges on network latency and the magnitude of consecutive packet losses. For instance, retransmissions suffer under high latency, while OPUS FEC and short RED patterns are not effective when faced with extensive consecutive packet losses. Hence, discerning the network’s characteristics is imperative to determine the most suitable approach for optimal results.

Analyzing Packet Loss Bursts in Real Networks

Exploring consecutive packet losses provides valuable insights into network behavior. How prevalent are these bursts in actual networks, and what magnitude do they typically reach?

Traditionally, answering these questions posed a challenge. However, with the advent of the new network simulator developed for the Opus 1.5 release and leveraging data from Microsoft, we now have a simple tool to estimate these parameters more readily.

I used that simulator to run some tests with different packet loss conditions and extract some statistics. The graph below illustrates the extent of loss bursts relative to the average packet loss percentage in a network.

For instance, in a network experiencing 1% packet loss, approximately 16% of losses entail more than one consecutive packet loss. Consequently, employing techniques like OPUS FEC or redundancy with a single packet may prove less effective in such scenarios. Similarly, in networks with 20% packet loss, approximately 5% of losses involve five or more consecutive packet losses so we would need to add a lot of latency to take advantage of redundancy in those cases.

Given that the efficacy of mitigation techniques depends on the nature of packet loss, the next logical query arises: How can we discern whether packet loss is bursty or not? Traditional RTCP feedback provides insights into the average percentage of loss but lacks details regarding loss patterns. To address this, several approaches exist:

Assuming Bursty Packet Loss: This prevalent approach assumes that the length of packet loss gaps correlates with the overall packet loss rate.
Transport CC Feedback: Although less common, this method involves measuring burstiness based on Transport Congestion Control feedback, offering a theoretically sound approach.
RTCP Extended Reports (XR): Leveraging XR reports enables the explicit inclusion of information regarding packet loss bursts.

Understanding the burstiness of packet loss is crucial for devising effective strategies to mitigate its impact on audio quality in WebRTC.

The decision regarding which techniques to implement is nuanced and far from black and white. Various combinations can be effective, but in my view, a blend of DRED, retransmissions, and PLC strikes a balance between maintaining quality, managing latency, and optimizing bitrate utilization across diverse network conditions. I’m eager to hear about others’ experiences and insights on this matter.

Loss based bandwidth estimation in WebRTC

2024-01-18T05:04:00.000-08:00

Measuring available bandwidth and avoiding congestion is the most critical and complex part of the video pipeline in WebRTC. The concept of bandwidth estimation (BWE) is simple: monitor packet latency, and if latency increases or packet loss occurs, back off and send less data. The first part is known as delay-based estimation, while the second part, less known, is referred to as loss-based estimation.

In the original implementation of WebRTC, the logic for loss-based estimation was straightforward: if there was more than 2% packet loss don't increase the bitrate sent and if it is more than 10% reduce the bitrate being sent. However, this naive approach had a flaw. Some networks also experience packet loss not due to congestion but inherent to the network itself (e.g., certain WiFi networks). We call that packet loss static or inherent packet loss.

To address this issue, the latest versions of Google’s WebRTC library introduced a more modern and sophisticated solution after several years of development and experimentation. This solution is known as loss-based bandwidth estimation version 2 (loss-based BWE v2) and it is the third iteration on the feature in the history of WebRTC. We can summarize that evolution of it in the following timeline:

Basic loss based bwe (~2011): Based on static thresholds for packet loss to increase / decrease bwe. Source Code
Loss based bwe v1 (~2019): Dynamic thresholds for packet loss and smoothing of measurements. Source Code
Loss based bwe v2 (2023): Maximum likelihood modeling from candidates based on current packet loss and trends. Source Code, Tracker Ticket

This post will focus on describing that newest solution (loss based bwe v2) enabled this year in Chrome and all the browsers and clients based on the same library.

Loss based BWE v2

At a very high level this diagram shows how loss based bwe is integrated in the estimation logic and what are the inputs taken into account. The loss based estimation takes as input the estimation from delay based estimation and based on that an the packet loss feedback received from the packets being sent it will try to provide a final estimation.

The new loss based BWE v2 runs the following steps every time it needs to provide a new estimation:

1/ First, it generates a list of potential candidates that describe the network's (also known as channel's) characteristics. In this case, the characteristics considered are the available bandwidth and the inherent packet loss of the network.

1.a. Bandwidth candidates are sourced from different areas. Essentially, delay-based estimations, the most recent loss-based estimations, and the bitrate being sent are all considered. These are then multiplied by certain factors (0.95, 1, 1.02) to generate potential additional new bandwidth candidates.

1.b. The inherent packet loss candidate is derived from the latest estimation. It is updated based on the trend of recent observations, using Newton's update method which is based on first and second derivatives. Additionally, there are limits based on the sent bitrate - at higher bitrates, the inherent packet loss should not be too high.

2/ Second, the algorithm selects the candidate that can most accurately explain the packet loss in the most recent observations. For that it calculates an objective score for each candidate and this is the maximum likelihood modeling aspect of the algorithm.

Those two steps can be seen in this snippet of the real code that summarize the concept of the algorithm:

  ChannelParameters best_candidate = current_best_estimate_;
  double objective_max = std::numeric_limits<double>::lowest();
  for (ChannelParameters candidate : GetCandidates(in_alr)) {
    NewtonsMethodUpdate(candidate);

    const double candidate_objective = GetObjective(candidate);
    if (candidate_objective > objective_max) {
      objective_max = candidate_objective;
      best_candidate = candidate;
    }
  }

The bandwidth of the chosen candidate with the highest objective score in this last step is then reported as the final estimation that will be used to decide how much video to encode and send to the network.

Although the algorithm appears simple based on this description, the coding process is far from trivial. Despite its relatively small size, it features over 30 tuneable parameters. If you're skeptical, you can review the issue tracker history. You'll see it took more than two years to develop, optimise, and deploy this new feature to production.

What do you think? As usual feedback is more than welcomed either here or in Twitter.

Audio Mixing or Forwarding

2023-08-27T11:17:00.004-07:00

How many audio streams should your WebRTC server forward to the participants in a room? There are various options, ranging from the simplest approach of forwarding everything, to the most extreme option of mixing all audio and sending just a single stream.

A few weeks ago, we engaged in a Twitter conversation about this very topic. Following that discussion, bloggeek also wrote a post on the subject. For me it is always interesting to see what different types of applications are doing because at least in some of those cases they have the ability to do A/B testing and compare the results with millions of users before making a decision.

The simplest way to determine the best approach is to enter a room with different applications and inspect the SDP (Session Description Protocol) in chrome://webrtc-internals. Within this tool, you can examine how many channels are being forwarded when you're in a room and look for potential clues within the SDP (some people use the "mixed" identifier for audio tracks, among other things). For example in case of Google Meet you can see in the SDP that joining a room you get 3 hardcoded audio streams created:

m=audio 9 UDP/TLS/RTP/SAVPF 111 63 9 0 8 13 110 126, mid=3
m=audio 9 UDP/TLS/RTP/SAVPF 111 63 9 0 8 13 110 126, mid=4

m=audio 9 UDP/TLS/RTP/SAVPF 111 63 9 0 8 13 110 126, mid=5

All the tests below were done with the web clients and Zoom was not included because it doens't make use of standard WebRTC and everything is binary&compiled so it would take more effort to reverse engineer it (for example analysing the audio bitrates when multiple participants speak).

What's the optimal approach for this task? There isn't a one-size-fits-all solution; it hinges on your particular use case and specific requirements. The most straightforward implementation is plain forwarding, which is what the majority of open-source projects and platforms employ. Interestingly, enterprise solutions tend to lean toward mixing. This inclination might be attributed to the fact that these services have evolved from earlier audio mixing solutions, the necessity for PSTN interconnection, or the need to accommodate unusual scenarios, such as devices with extremely limited available bandwidth.

Architecture for AI integration in conferencing applications

2023-07-11T07:47:00.003-07:00

With the latest improvements in ML technology, especially generative algorithms and large language models, more and more conferencing applications are adding these capabilities to their offerings.

This ML technology can be applied to conferencing applications at two different levels: the infrastructure level with improvements in media handling and transmission, and the application level with new features or capabilities for the users.

At the infrastructure level (codecs, noise suppression, etc.) most of the high-level ideas were covered in this other post. Some interesting recent advances are applying “ML codecs” for audio redundancy, and the next frontier is applying generative algorithms also to video, as well as general applications to photorealistic avatars.

This post focuses on the second level (the application part) and how to implement typical features such as summarization, image generation, or moderation. The idea here is to present a reference architecture that can be used to implement these services.

The architecture presented below is based on two core ideas:

1. The system is split into three different subsystems that have different execution patterns:

One on-demand subsystem that the application can use to request specific capabilities when needed.
One offline subsystem that processes files, events and recordings in the background for enhancement, filtering, or extracting relevant information. This system has a tradicional storage bucket as backend (f.e. AWS S3).
One online subsystem that processes all communication events (messaging, signaling, messaging) in almost real-time to make decisions, extract information, or enhance those events. This system has a streaming queue as backend (f.e Apache Kafka).

2. All the ML algorithms are exposed behind an API/Gateway to abstract details of the implementation and interface with hosted models and external providers.

With that architecture these are some typical features and how they could be implemented:

Media generation (images, music, memes…) can be requested from the application when requested by the user or automatically when needed by the application.
Enhanced recordings, for example with the typical super-resolution algorithms, can be generated by a background post-processing process ran every time there is a new recording generated in the storage bucket.
Real time extraction of relevant fields from the conversation (f.e. action items) can be implemented with a process reading the captions from the queue, grouping them in chunks by topic or speaker, and then sending those chunks to the AI Gateway. That information can be fed back to the queue so that other processes or the application can make use of them.

This post express my personal views. Feedback is more than welcomed either here or in Twitter.

Review of Signaling in different WebRTC applications

2023-05-24T22:05:00.007-07:00

This post provides a quick review of the signaling channel implementation in various popular WebRTC platforms. It examines the protocol used for the channel, how messages are serialized, and whether the applications use Session Description Protocol (SDP) as an opaque string over the wire, or if they instead send the required parameters in a custom format.

To provide a variety of platforms, I have included a mix of popular end-user applications, cloud providers, and open-source implementations in the table. If you would like, I am happy to add others to the list.

How was it tested?

To test it, join a room and check in Chrome Developer Tools whether there are WebSocket connections established or periodic HTTP requests being made. Then, inspect the messages of those connections and requests and check if the format is Binary/JSON/XML. In case of Binary messages, it's harder to see the content, and there's a chance that the information is compressed/encrypted, and there's some JSON or SDP inside.

Next, look inside those messages for the typical information in SDP description, such as ufrag, header extensions, and payload types, and see how they are sent.

Transport:

Most applications seem to use WebSockets, but some of the biggest ones use HTTP polling. This may be due to historical reasons, to better handle specific users with restricted networks or devices, or to react faster to disconnections in unstable connections.

In case of Google Meet it looks like part of the signaling goes over datachannels in addition to the existing HTTP long polling mechanism.

Serialization:

The serialization formats are all over the place between binary (the most efficient), JSON (the simplest) and XML in case of using XMPP. Using JSON can be less efficient over the wire but require less libraries/runtime to process it in a browser. The ProtoJSON approach used by Google Meet looks like a good compromise in the middle.

SDP Usage:

Most applications seem to not use SDPs over the wire. Instead they only send the parameters needed and keep the SDPs as an implementation detail in the client side. That's slightly more efficient and gives you more flexibility but it is less straightforward to ad those conversions if all your endpoints are libwebrtc based.

The conclusion could be that there is no single right solution for WebRTC signaling but different alternatives that all of them can work well. However two of the most common approaches seems to be using WebSockets and not sending SDP on the wire that is something that personally I'm very happy about.
Did you find something else interesting or maybe even something that is not accurate in this post? Feedback is more than welcomed either here or in Twitter.

Perfect Interactive Broadcasting Architecture

2023-05-15T22:10:00.003-07:00

While we might sometimes talk about low-latency or interactive broadcasting in a generic way, it's important to note that there are actually two distinct types of streaming use cases that require different levels of interactivity.

Conversational use cases where multiple participants are talking together and that conversation is being streamed to many other viewers. These viewers can potentially become speakers at some point too.
Single stream use cases where just one person is streaming their video feed (it can be their camera, their screen, or a combination of both) to many other viewers who can interact in different ways. The most obvious way is through chat messages, but it can also include emoji reactions or even bids on an auction being streamed.

The conversational use case has specific requirements. For instance, it demands effective synchronization of multiple streams, ultra-low latency (less than 250ms) only between the users who are speaking, and an element that performs video composition of all the streams prior to forwarding it to viewers. There are a variety of approaches to implementing this, each with its own set of pros and cons, as exemplified by top products like Streamyard or the way Twitch streamers use to integrate Discord into their streams.

This post is primarily focused on the second category, which pertains to single stream use cases. What makes this particular use case compelling is:

That single stream needs to be streamed with perfect quality if there are no network limitations. If there are network limitations, it should be streamed with the best possible quality in real-time and send the additional quality layer in the background/later to have at least the recording with the best possible quality.
There is a single stream being sent so there is no need for synchronization of multiple sources.
You usually don't need < 250ms latency and can prioritize providing a bit higher quality and have 1-2sec latency.

One potential requirement for these use cases is support for various endpoints such as browsers, mobile devices, smart TVs, consoles, and mobile phones. These devices have different capabilities in terms of networking, protocols, and codecs.

What would be the perfect solution in terms of quality and flexibility for this type of scenarios? (assuming infinite time to develop it and money to run it :))

Ingestion

Use the best available codec for the endpoint, taking into account device capabilities and status (such as battery load). In many cases, this could be the AV1 video codec with a high-quality Opus audio configuration.

If possible, send a single stream with perfect quality. However, if there are network issues in the sender's uplink, send two layers of video instead. One layer should maintain the low-latency requirement, while the other can be sent in the background with as much latency as needed. This ensures that the server will end up with a layer of perfect quality eventually.

To provide low-latency delivery in the presence of packet loss in the network, use a UDP-based transport. There are different alternatives available, such as SRT, WebRTC, RUSH, and custom protocols. Let's use RTS (Real Time Streaming) as a generic name for this. If the protocol needs to be supported by browsers as well, then either WebRTC or a custom QUIC/WebTransport protocol seem like reasonable candidates. However, in the case of WebRTC, it may be harder to implement the previous idea of always having a layer with perfect quality in background, and may require some workarounds.

Processing

There are a variety of audio and video processing algorithms that are gaining popularity, such as audio and video noise removal and superresolution. However, some of these algorithms are too resource-intensive to run on many client devices.

Furthermore, the most appropriate codec on the sender side may not always be the best or supported by all receivers.

For these reasons, a server processing pipeline that includes decoding, processing, and reencoding can be highly advantageous. There are also other tasks that can be incorporated into this pipeline, such as generating captions or adding watermarks to videos.

This is the most expensive component of the system, and for some applications, it may be excessive and should be eliminated or only applied to the most important streams. If there is no Processing layer in the server then the solution becomes much simpler with the sender generating video streams in multiple qualities and the server just forwarding them without reencoding them.

Delivery

Different devices require different qualities and codecs. Additionally, some devices may be limited to HLS support, so varying protocols may be necessary.

In most cases, server side recording is also necessary.

For use cases that really require real-time functionality (probably not that many), the RTS protocol can be used. This protocol can provide different media versions, including various resolutions, bitrates, and codecs, which can be distributed using a Real Time CDN. For example, WebRTC can use SFUs infrastructure with this protocol.

For use cases that require interactivity, but not real-time functionality (most of them?), or that require compatibility with legacy devices, the LL-HLS protocol can be used. Like the RTS protocol, LL-HLS offers different media versions that can be distributed using a Traditional CDN.

And that was the proposal for a possible “perfect” broadcasting architecture for interactive use cases would look like. What do you think? As usual feedback is more than welcomed either here or in Twitter.

WebRTC header extensions review

2023-05-01T07:51:00.011-07:00

WebRTC supports the concept of RTP header extensions to extend media packets with additional metadata. One of the most common use cases is to attach the audio level to audio packets so that the server can calculate active speakers without having to decode the audio packets.

Some of these header extensions are standard and have been used for a while, but there are others that are added by Google when needed and are only documented lightly in the website and the libwebrtc code. Those header extensions and its usage is not very well know and this post is an attempt to give visibility of them for the WebRTC community.

To discover some of these headers you can usually take a look at the Offer/Answer of a Google Meet session or take a look at the libwebrtc source code here: https://chromium.googlesource.com/external/webrtc/+/master/api/rtp_parameters.h

Audio Levels (urn:ietf:params:rtp-hdrext:ssrc-audio-level) Doc
[Very common]

The header contains the audio level (volume) of the audio contained in this audio package, as well as an optional bit (not usually utilized) that indicates if the packet is voice or another sort of audio.

Use case: A server not decoding audio can still detect the active and/or dominant speaker in a session.

Mixer to client audio levels (urn:ietf:params:rtp-hdrext:csrc-audio-level) Doc
[Not very common but Google Meet uses it]

This header extension includes information about the different sources contributing to the audio of this packet and their audio levels.

Use case: If a server combines or merges audio, an audio stream can portray various speakers simultaneously, and this header enables the receiver to identify the particular user who was speaking during that period.

Inband Confort Noise (http://www.webrtc.org/experiments/rtp-hdrext/inband-c) Doc
[Anybody using it?]

This header extension informs if the audio packet includes confort noise and its level.

Use case: A server could de-prioritize these packets or avoid including them in the mixing. A receiver could use it as a hint for discontinuous transmission being enabled.

Absolute Send Time (http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time) Doc
[Legacy]

This header extension includes an NTP timestamp for when the packet was sent instead of when the media was captured.

Use case: The main use case is for server side bandwidth estimation to be able to detect changes in inter packet latency and use it to estimate the bandwidth available and send it back to the sender (f.e. using a REMB RTCP packet) if needed.

Capture Time (http://www.webrtc.org/experiments/rtp-hdrext/abs-capture-time) Doc
[Not very common but Google Meet uses it]

This header extension includes an NTP timestamp for when the frame corresponding to this packet was captured from the input device.

Use case: Implement audio/video synchronization in servers, for example for mixing purposes.

Transport Wide Identifier (http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01) Doc
[Almost always used]

This header extension includes an unique identifier for each packet so that the receiver can then send feedback back to the sender with relative timestamps of those packets with those identifiers. There is a v2 version that gives more control to the sender on when to receive feedback.

Use case: It is only used to generate TransportControlFeedback packets including the inter packet delay for those identifiers included in the header extension.

Media Identifier (mid) (urn:ietf:params:rtp-hdrext:sdes:mid) Doc
[Almost always used]

This header extension includes an ID for the media stream that is also included in the SDP.

Use case: This way the server or the receiver can understand what media stream those packets are for. To save bandwidth this header extension is only included in the first packets of a stream or when the ssrc changes.

RTP Stream ID (urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id) Doc
[Very common]

This header extension includes the identifier of each encoding of a stream (f.e. a simulcast layer).

Use case: It is similar to the mid header extension described above but in this case to differentiate multiple encodings in case of simulcast.

Repaired RTP Stream ID (urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id) Doc
[Very common]

This header extension includes the identifier of retransmission packets that provide redundancy for packets in other RTP stream.

Use case: Differentiate and associate retransmissions to the corresponding RTP stream.

Time Offset (urn:ietf:params:rtp-hdrext:toffset) Doc
[Anybody using it?]

Includes the offset from when the packet was captured to when it was sent.

Use case: Allow more accurate jitter calculation in the receiver side.

Video orientation (urn:3gpp:video-orientation) Doc
[Commonly used, the only problem is firefox]

This header extension includes the rotation that needs to be applied to the video frames before rendering them (0, 90, 180, 270 degrees). This way a phone sending a 640x480 video in landscape mode can keep sending 640x480 with a value of 90 in the header extension when rotating the phone to portrait mode.

Use case: Used by the sender to avoid having to change the aspect ratio of the stream when a device (usually a phone) goes from portrait to landscape. This is not supported by Firefox making it tricky for some use cases.

Playout Delay (http://www.webrtc.org/experiments/rtp-hdrext/playout-delay) Doc
[Not very commonly used, mostly for gamestreaming use cases]

This header extension includes a delay that should be applied to this audio or video stream before rendering it.

Use case: Used by the sender or a server to indicate the receiver that it needs minimum latency (playout delay=0) or it can tolerate some latency to have better quality (f.e. playout delay 500ms). This behaviour can also be controled with the javascript property playoutDelayHint

Video Content Type (http://www.webrtc.org/experiments/rtp-hdrext/video-content-type) Doc
[Not very commonly used]

This header extension includes information about the type of content (camera or screensharing). Chrome also includes some even more proprietary stuff here to define the spatial layer of this simulcast stream.

Use case: Used by a server to apply different . It could potentially be used by a receiver too to tune some latency or retransmissions behaviour but apparently it is not used for that in Chrome.

Video Timing (http://www.webrtc.org/experiments/rtp-hdrext/video-timing) Doc
[Not very commonly used. Google Meet only maybe?]

This header includes different timestamps for the media corresponding with a specific frame (encode start, encode end, packetization time, pacer time).

Use case: I think the main use case is for monitoring, testing and statistics purposes.

Color Space (http://www.webrtc.org/experiments/rtp-hdrext/color-space) Doc
[Not very commonly used. Stadia?]

This header extension includes the color space used to represent the video frames information.

Use case: Encode video represented in different color spaces.

Generic Frame Descriptor (http://www.webrtc.org/experiments/rtp-hdrext/generic-frame-descriptor-00) Doc
[Legacy. Mostly replaced with the Dependency Descriptor hdrext below]

This header extension includes the information about layers and dependencies of a frame for scalable video.

Use case: Used by a server to be able to filter different frames and resolutions properly in a code-independent way.

Video Layers allocation (http://www.webrtc.org/experiments/rtp-hdrext/video-layers-allocation00) Doc
[Not very commonly used. Google Meet uses it.]

This header extension includes information for streams using scalable video coding or simulcast about the spatial layers, the temporal layers, their resolutions, framerates and bitrates.

Use case: Otherwise the server needs to guess some of this information (f.e. bitrates) to make decissions on what layers to forward to each participant, leading to less accurate values and more complexity.

Dependency descriptor (https://aomediacodec.github.io/av1-rtp-spec/#dependency-descriptor-rtp-header-extension) Doc
[Not very commonly used. Google Meet uses it.]

This header extension includes the information about layers and dependencies of a frame for scalable video. It was designed for AV1 but can be used for other codecs (f.e. VP9).

Use case: Used by a selective forwarding unit to be able to filter different frames and resolutions properly in a code-independent way. It is the same purpose as the legacy Generic Frame Descriptor included above.

Video Frame Tracking Id (http://www.webrtc.org/experiments/rtp-hdrext/video-frame-tracking-id) Doc
[Only for testing]

Header extension including a unique frame identifier.

Use case: For testing purposes to be able to correlate the frames in the sender and receiver side.

What do you think? Any other info or feedback about these or other header extensions? Feel free to share those here or in Twitter.

Existing WebRTC is not great for broadcasting use cases

2023-04-12T07:30:00.016-07:00

WebRTC was originally designed for real-time communication with a small number of participants, where latency requirements are extremely strict (typically <250ms). However, it has also been utilized for broadcasting use cases, such as YouTube Studio or Cloudflare CDN, where protocols used in the past have been different, typically Adobe’s RTMP and protocols based on HTTP.

WebRTC offers a new range of broadcasting use cases, particularly those requiring hyper-low latency, such as those with audience interactivity, for instance, user reactions or auction use cases. However, choosing WebRTC comes with tradeoffs, including increased complexity, scalability challenges, or lower quality. While it's possible to address the first two with enough time and effort, the primary concern should be how to obtain the best possible quality.

Why do we have lower quality when using WebRTC?

First of all, a clarification. In a perfect network with infinite bandwidth there is no much difference in quality but in other restricted or flaky networks these are some reasons why WebRTC provides lower quality than RTMP&HTTP based solutions:

Smaller buffers: WebRTC implementations prioritize latency over quality by using small buffers, which can sometimes be too small to receive retransmissions in time. TCP-based protocols with large buffers prioritize quality over latency and do not face this issue.
Aggressive congestion control: WebRTC employs aggressive congestion control that quickly reduces video transmission when congestion is detected to prevent increased latency and potential packet loss. This helps to have more predictive low latency but in many cases being less aggressive would maintain a higher quality at the expense of a higher latency.
Only real time reliability: If you lose connectivity for some seconds WebRTC is not going to queue the media to deliver it whenever you recover. Same way it is not going to recover a previous frame after a newer one has been decoded. The most obvious impact on this is in case of server side recordings.
Fast Encoding: WebRTC codecs configuration are optimized for speed and do not use two-pass encodings or bidirectional predicted frames, instead using conservative settings to ensure fast encoding. However, this means that the encoding efficiency is not as good as that of broadcasting applications, resulting in slightly worse encoding for a given bitrate.
Unpredictable Bitrate: The fact that WebRTC cannot use 2 pass encoding makes the quality/bitrate decisions by the codecs less optimal and unstable.

To be honest some of these limitations can be solved if you control the WebRTC implementation (f.e. in a mobile or desktop native app) but that is not so easy or even possible nowadays for web applications where the WebRTC stack is embedded in the browsers.

What should we do for interactive broadcasting?

Ideally we need a protocol where we can configure the acceptable latency in a range from real time (<250 ms) to traditional broadcasting (5 - 10secs). Once that constraint is set the protocol needs to provide the best possible quality in those conditions. That is doable and we will probably see it in one or both of these ways:

More tunneable WebRTC. With some more flexibility in WebRTC APIs it should be possible to overcome the limitations described in the previous section. For example things like configurable congestion control with settings like RTCPeerConnection.congestionControl = "low latency" or some lower level APIs in the new WebCodecs API.
New protocols working over UDP and reusing some of the WebRTC techniques but providing more flexibility (f.e. new QUIC protocols like RUSH). Implementations using those protocols created from scratch can overcome most of those limitations because they leave part of the problem (buffering or encoding) to the application. Although they would also need the ability to control the congestion control behavior.

So should I use WebRTC or not?

I would use it only when it is really required and provides enough business value to make it worth the additional complexity and potential quality degradation. For streaming a traditional soccer game it is probably not needed while for a stream with interactivity with the audience (f.e. reactions) it can provide real value and the quality loss can be worth the benefits in terms of engagement.

Also we need to keep in mind that we don't need to have an end to end WebRTC. For example in cases where there is a need for low latency only between speakers it could make sense to use WebRTC for ingestion but keep using HLS for playback. Or in some cases use RTMP for ingestion from broadcasters with good connectivity and RUSH for playback.

Media Ingestion protocols Review

2023-02-18T09:37:00.019-08:00

This post summaries the most relevant protocols using for media ingestion. In these media ingestion flows a client that is broadcasting some content sends it to an endpoint (typically a server) that will re-format it for distribution to a potentially large audience, using a different protocol optimised for media playback.

The following sections show the main ideas behind each protocol as well as some of their advantages and disadvantages. The protocols are organised into three buckets (tradicional protocols, modern protocols and next generation protocols) that correspond to three waves of protocols with the first one based on TCP, the second one on UDP or HTTP and the third one with QUIC. You can see a summary of those characteristics as well as the timeline and adoption in the following diagram.

Traditional protocols

RTMP

RTMP (Real-Time Messaging Protocol) is the most widely used protocol for media ingestion nowadays. It was developed more than 20y ago and popularised by Adobe/Flash initially as a playback protocol. It is supported by the biggest Internet video platforms like Youtube, Facebook or Twitch.

The protocol transports media over TCP, includes an initial handshake and support for arbitrary content (metadata, RPC and media) using a proprietary binary format. The feedback mechanisms to control the media transmission is limited as well as the codecs supported. There are some variations of RTMP developed for proxy traversal or P2P but they are not that popular.

It is a protocol used for ingestion and not playback so in most of the cases it requieres other protocol and some repacketization for distribution (typically using HLS protocol).

RTSP

RTSP (Real-Time Streaming Protocol) is another protocol developed in the 90s but in this case it was fully standard from the beginning and reuses other internet standard protocols like RTP (Real-Time Protocol), RTCP (Real-Time Control Protocol) and SDP (Session Description Protocol). The main motivation was its standardisation and the lower latency compared with RTMP. It is not used that much today and mostly limited to specific use cases like IP cameras.

It is a protocol that separates the control and the media plane. The establishment part of RTSP is similar to HTTP and uses SDP to describe the media content and the media part uses RTP over UDP. It could support most of the feedback and capabilities of RTP extensions but usually the implementations are limited to the basic RTP profile.

Modern protocols

SRT

SRT (Secure Reliable Protocol) is a protocol invented in 2012 and released in 2017 as a replacement for the tradicional protocols to be better suited for low latency transmission over noisy networks. It is used by many professional broadcasters and some internet services like Caffeine.tv.

(source)

The protocol makes use of UDP for the transmission of control and data packets and includes feedback for retransmissions under packet loss. The protocol syntax is based on a previously existing protocol for file transfer called ULP and has different modes to enable use cases from reliable file transfer to low latency streaming.

RIST

RIST (Reliable Internet Stream Transport) is a protocol similar to SRT in terms of features and motivations. It was developed to increase the reliability against packet losses and the security included in SRT as well as to support higher bitrate streams.

This protocol is also transported over UDP although in this case it uses RTP instead of inventing a new protocol for that. It includes feedback for retransmissions under packet loss.

It is a slightly better protocol that reuses some existing protocols but so far it had lower adoption than SRT.

WebRTC

WebRTC is a project that was designed for videoconferencing use cases but that has seen also some adoption for streaming applications. Its real time nature makes it suitable for use cases where ultra low latency is more important than getting the highest possible quality and also the native support in browsers makes it very simple to integrate for applications. Different streaming services use it in different degrees like for example Youtube Studio or StreamYard.

The protocol doesn't define the signalling/control plane and that's left to each application although recently a new specification (WHIP) has defined a very simple HTTP signalling protocol for basic ingestion use cases. The protocol does define the media plane that is composed of different layers for connectivity (ICE protocol), encryption (DTLS&SRTP protocols) and media itself (RTP&RTP procols). That makes it a complex protocol even if it is mostly a composition of protocols existing for many years. It includes feedback mechanisms for retransmissions and bandwidth estimation as well as many other extensions for FEC or buffers control.

The WebRTC protocol stack can be used also for playback if needed although in many cases the applications still rely on traditional protocols for playback to prioritise reachability and quality over the lowest possible latency.

FLT

FLT (Faster than Light Protocol) is a protocol designed to provide a very low latency replacement for RTMP. The protocol was designed for Microsoft Mixer service and many applications and services implemented it although it is mostly unused with the shutdown of Mixer.

The protocol was making use of UDP and RTP to provide ultra low latency in a way similar to WebRTC but without the ICE and DTLS parts and it includes a control protocol over TCP to establish and negotiate the media connection.

HTTP ingestion (HLS, DASH...)

HTTP ingestion consist on reusing the protocols that are mostly used for media delivery (HLS, DASH...) but can also be used for media ingestion. The most popular of these protocols is probably HLS (HTTP Live Streaming) and it is supported by Akamai or Youtube Live in addition to RTMP for ingestion.

These protocols use HTTP as transport to send the media segments encoding pieces of audio+video of a small number of seconds duration and also to send the playlists with the metadata and the references to those segments. They make use of the delivery and encryption mechanisms of HTTP and doesn't include any specific feedback mechanism to adapt the ingestion to the network conditions beyond controlling latencies and buffers.

Next generation protocols

RUSH

RUSH (Reliable (unreliable) streaming protocol) is a bidirectional media protocol designed by Facebook as a replacement for RTMP with the goal to provide support for new audio and video codecs, extensibility ini the form of new message types, and multi-track support. This protocol is only used by Facebook at this point both for ingestion and for media distribution inside their network.

The RUSH protocol is based on two core ideas:

Use QUIC as the transport protocol. The QUIC protocol, built on UDP, serves as the foundation for HTTP/3 and offers robust security measures in addition to affording granular control over packet priorities and reliability.
Use a different QUIC stream for each video frame so that they can be prioritized independently in case of congestion based on the latency and the importance of each frame.

The specification includes a control protocol and the media protocol both running on top of a single QUIC connection. The messages have a binary encoding and include fields similar to the ones in RTP.

WARP

WARP (Live media transport over QUIC) protocol is also a bidirectional protocol but in this case designed by Twitch as a replacement for RTMP for the ingestion but also for existing playback protocols like HLS. Twitch is already using it for the distribution of media chunks in their backbone network.

The ideas behind this protocol are similar to the ones in RUSH with a configurable latency vs quality trade-off. The main difference is that the unit of transmission is not a frame but a segment. A segment is a group of encoded frames packages in a MP4 container. Another difference is that WARP is defined on top of WebTransport instead of just QUIC (WebTransport is a thin layer over QUIC intended for web browsers support)

The specification includes a control protocol and the media protocol both running on top of a single QUIC connection. All the messages (control and media) includes a type-length encoded binary header that should allow any intermediaries in the distribution pipeline to prioritise and filter messages based on their content.

The protocol can also be used for playback and the use of segments makes it very simple to do the conversion to other segment-based protocol like HLS if needed.

Conclussions

After many years where most of the ingestion has been happening with RTMP there is finally some alternatives getting enough traction. WebRTC and SRT where the first ones and have already been adopted for many use cases while the new protocols designed over QUIC (RUSH, WARP or QUICR) are likely to be widely deployed in the next years.

The use of one protocol over the others depends mostly on two factors:

Support and interoperability. The most widely supported protocol is still RTMP and some streaming equipment only support that protocol or the newer SRT/RIST protocols. However, if we ignore the new QUIC protocols, most of these protocols are already supported in the widely used opensource OBS streaming client.
Latency vs quality trade-off. Depending on the use case it is possible to select a protocol that is better suited for ultra low latency (like WebRTC) or when that is not strictly required other protocol can be selected that can provide the highest possible quality (like HLS).

As a final note this is the result of the wowza report on protocols being used by for ingestion:

Different types of latency measurements in WebRTC

2022-10-04T11:05:00.004-07:00

When building WebRTC services one of the most important metrics to measure the user experience is the latency of the communications. The latency is important because it has an impact on the conversational interactivity but also on video quality when using retransmissions (that is the most common case) because the effectiveness of retransmissions depend on how fast you get them.

And to be fair at the end of the day latency is what differentiates Real Time Communications from other types of communications and protocols like the ones used for streaming use cases that are less sensitive to delays, so it is clear that latency is an important metric to track.

However there is no single measurement of latency and different platforms, APIs and people usually measure different types of latency. From what I've seen in the past we can see differences in these four axis described below.

One Hop latency vs End to End latency

When there are multiple servers involved in a conversation the native WebRTC stats only provides information about the latency in the first hop (from the client to the server). That doesn't give you the full picture and usually what you prefer is to measure the end to end delay including the latency from the first client to the first server, then from each server to the next one and finally from the last server to the other client.

This can be seen in this great diagram from webrtchacks.

Mesuring the End-to-End latency is much harder as it is not provided automatically by WebRTC stacks and requires some additional ping-pong messages to be sent (f.e. using datachannels) or a mechanism relaying in synchronized clocks that it is not very practical.

One alternative way to measure it is to measure the latency of each hop and then aggregate the averages per source and destination regions. This doesn't give you the same thing but it can be an approximation and good enough to detect issues in specific regions.

Real Users latency vs Synthetic Traffic latency

Some people use synthetic tests to measure the latency provided by their platform. That's great for alerting purposes but doesn't give you the full picture of how users are experiencing your service. For example maybe there is one specific region from where users are experiencing high latencies but you don't have your test clients probing your platform from that very specific region.

When measuring latency for real users you typically have to use the WebRTC getStats API or some datachannels ping-pong messaging but when using synthetic traffic you have more freedom to use special custom RTP packets or a video source with a timestamp watermark.

Network latency or Capture-to-Render latency

When measuring the latency typically you measure the delay of the packets in the network but that's not the whole latency that a user will experience for his voice and video. Apart from that network latency there are also delays introduced by the software doing the capture and rendering. This is specially relevant in case of the buffers used in the receiver side to mitigate network issues like jitter, packet loss or packet reordering.

To measure the whole latency including the delays introduced in the capturing and rendering sides you can include in your calculations some additional metrics provided by the WebRTC getStats API like the jitterBufferDelay, playoutDelay, packetSendDelay...

Round Trip Time or One Way delay

Another small difference is if the latency is measured as round trip time latency (the sum of both directions A->B + B->A) or just one of the ways independently. In most of the cases what is measured is the round trip time and then just divided by two as a good estimation of the one way delay.

Conclussions

As a summary, next time you provide numbers on the latency of your WebRTC solution please explain what type of latency you are measuring and how are you measuring to make it more clear 🙏.

If you can you should probably try to measure the end-to-end delay but even the hop-by-hop latency is very relevant for monitoring and troubleshooting. If you have enough traffic it is possible that real users latency is what you want to measure, but if you don't have enough users or you want to have some more data then using synthetic tests to estimate the latencies is also a good approach.

Screensharing content detection

2022-08-15T02:57:00.004-07:00

One interesting feature in WebRTC is the ability to configure a content hint for the media tracks so that WebRTC can optimize the transmission for that specific type of content. That way if the content hint is "text" it will try to optimize the transmission for readability and if the content hint is "motion" it will try to optimize the transmission for fluidity even if that means reducing the resolution or the definition of the video.

This is specially useful when sharing documents or slides where the "crispiness" of the text is very important for the user experience. You can see the impact of those hints in the video encoding in this screenshot taken from the W3C spec:

This is very useful but there is a small problem. What happens when we don't know the type of content being shared? How do we know if the browser tab being shared has some static text slides or a youtube video being played?

One possible option could be to do some type of image processing to detect the type of content in the video frames and check for text content. That's possible doable and not that expensive if run only once every few seconds. But right now I was more interested on using existing WebRTC statistics as a proxy to detect the type of content so I tried a couple of approaches along those lines.

The other motivation to do this kind of detection could be for analytics purposes to understand better how users are interacting with the product and make further changes in the application to optimize it for those use cases.

Simplest approach

The idea is that when you are sending static content (slides) WebRTC notices that the frame didn't change from the previous one and skips sending a new one so it ends up sending ~1 frame per second only.

With this approach just checking the framerate sent using the getStats() API we could have a high level of confidence on the content is static. For example, we could use a threshold of 2-3 fps to differentiate static and dynamic content.

Special cases:

You are sending a low framerate because the available bandwidth is too low and not because the content is static. I don't think libwebrtc will send such a low framerate in that case but anyway we can differentiate this case by checking the "availableOutgoingBitrate" and/or "qualityLimitationReason" the WebRTC stats.
When you move the cursor over the slides, or your slides have a small animated logo in one corner or you are sharing a text editor but you have a small blinking cursor. In those cases this approach won't work because the framerate sent will be high.

Elaborated approach

The idea is that when you are sending static images you can send a higher quality with less number of bits because the image doesn't change from one frame to the next one so it needs fewer pixels to be encoded.

We can use the framerate, qp (quantization parameter), and resolution statistics from WebRTC and calculate a number that intuitively shows Quality / BitsUsedPerPixel.

You can see the implementation in this jsfiddle: https://jsfiddle.net/zb8w4nd5/3/

I tested both approaches using a document with two slides, one static with text and one image and one with a youtube video embedded. The results are shown in the next graph.

One thing we can see in the graph is that the framerate is a good proxy apart from when there is some "artificial" movement like the mouse cursor. In that sense, the Quality factor looks like an approach with fewer issues.

As you can see in the graph after switching from the video slide to the text slide again it took a bit to recover in terms of quality. Combining both approaches and using the framerate in that case could make the detection faster.

Additional considerations:

Probably is a good idea to ignore the first few seconds as it can take a bit to stabilize.
Add some hysteresis or window average to avoid detecting when switching slides or scrolling in a text document.
The Quality factor is assuming everything is linear and maybe other approaches (logarithmic, quadratic...) could have more accurate results.
I haven't used this in a real product yet, so take it just as a list of ideas to explore and not a finished reliable algorithm.

New look at WebRTC usage in Google Meet

2022-06-05T02:47:00.012-07:00

I hadn't looked at Google Meet webrtc internals for a while so while I was having a meeting last week I decided that it was a good time to check what were the latest changes that had been added.

P2P Connections

One of the first things that I checked was if Google Meet was using P2P connections when there are only two participants in the room and I was surprised that it was not the case. P2P support was included in the past (P2P-SFU transitions discussion) but apparently has been removed.

This increase the infrastructure cost (not an issue for Google) and increase a bit the end to end latency for the 1:1 calls but given that Google Meet is probably deployed in many points of presence that's probably not a big increase and the simplicity of not having to handle another type of connections and the transition between them (P2P <-> SFU) is a big benefit so it looks reasonable.

ICE candidates and (NO) TURN servers

Google Meet is not configuring any ICE servers anymore and their servers are providing candidates for both IPv4 and IPv6 and types: UDP (random port) + TCP (same random port as UDP) + UDP (3478 port) + SSLTCP (443 port).

This is the iceServers configuration that is empty as you can see:

https://meet.google.com/_meet/xxx-xxx-xxx, { iceServers: [], iceTransportPolicy: all, bundlePolicy: max-bundle, rtcpMuxPolicy: require, iceCandidatePoolSize: 0, sdpSemantics: "unified-plan", extmapAllowMixed: true }, {advanced: [{googScreencastMinBitrate: {exact: 100}}]}

The fact that it doesn't use any TURN servers shouldn't be a huge issue given that the servers have support for SSLTCP and was seen also also in other apps in the past like Jitsi or Houseparty. The main implications could be the lack of support in other browsers like firefox and some very strict proxies/firewalls blocking the pseudo-SSL connection used by SSLTCP.

New RTP header extensions

Google Meet has always used many header extensions and this time and saw two that I hadn't seen in the past.

One header extension is related to the new AV1 video codec and used to provide scalability information that the server can use for selective filtering and forwarding of video layers. It is interesting because AV1 is not used by Google Meet but maybe this is an indication of some work in progress or the support of the new codec in internal versions of the product.

a=extmap:15 https://aomediacodec.github.io/av1-rtp-spec/#dependency-descriptor-rtp-header-extension

(@murillo&@HCornflower mentioned in Twitter that even if this header extension was defined for AV1 it can also be used also for VP9 and that is apparently the case based on a quick test with a sniffer)

The other header extension provides information about target bitrates, resolutions and framerates of the layers so that the server can use that information to decide which layer to forward instead of making those decissions based on the bitrate received like all the SFUs do today. The advantage is that this value is more stable and reliable than the calculated one that can have temporal spikes depending on. the network and pacing implementation and even the content being sent.

a=extmap:12 http://www.webrtc.org/experiments/rtp-hdrext/video-layers-allocation00

Special RTP Control Messages (RTCP)

There were a couple of RTCP features in the SDP negotiation that got my attention.

The first one is a new RTCP message called RRTR. Well, it is not really new as it was standardized a long time ago (RFC3611) but I haven't seen it used till now. The purpose of the RRTR message and the corresponding to DLRR is to be able to calculate the round trip time for receiving streams in a similar way to how it is calculated for sending streams with SR and RR messages. Basically RRTR sends a timestamp and DLRR sends back the original timestamp plus the delay between RRTR and DLRR messages.

a=rtcp-fb:96 rrtr

The other interesting thing is that Google Meet enables transport-cc RTCP messages to use audio packets also for bandwidth estimation and not only video packets like most of the platforms do. This was problematic in the past but apparently it is safe enough now and in that case it looks like a very good idea to have it enabled.

a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc

Video Encoding

Google Meet uses VP9 with profile 0 although for some reason it falls back to VP8 whenever a Firefox browser joins the room. One interesting thing about the video encoding is the attribute "useadaptativelayering" included in the SDP negotiation. The attribute has been there for a while and there are two versions of it but there is nothing about it in the opensource version of Chromium/WebRTC and neither in the Chrome binary 🤔, so I would assume that is something only used in an old or custom version of libwebrc (mobile?).

a=fmtp:96 useadaptivelayering_v2=true; useadaptivelayering=true

Audio Encoding

Google Meet is using opus as expected (with dtx and inbandfec enabled) but the most interesting part is that the usage of redundancy encoding (red) is also negotiated for the audio channels. This encoding allows sending multiple copies of the same packet when needed, to provide better robustness against packet loss (detailed analysis of RED in WebRTCHacks).

But even if it is negotiated it is not really active for sending because it is included in the codecs list after opus. I also tested it introducing a 15% packet loss and checking the packets sent and received with a sniffer.

a=rtpmap:111 opus/48000/2

a=fmtp:111 minptime=10; useinbandfec=1; stereo=0; sprop-stereo=0; usedtx=1
a=rtpmap:63 red/48000/2

Media Acquisition

The last thing I took a look at is at the configuration requested from the microphone and camera. The most interesting thing I found was the fact that the framerate is limited to 24fps explicitly when the default is usually 30fps.

Video constraints: {deviceId: {exact: ["xxxxxxxxx"]}, advanced: [{frameRate: {min: 24}}, {height: {min: 720}}, {width: {min: 1280}}, {frameRate: {max: 24}}, {width: {max: 1280}}, {height: {max: 720}}, {aspectRatio: {exact: 1.77778}}]}

There are other things that can be interesting from Google Meet like the usage of a single peer connection (apart from the extra one for the screen sharing), the creation of offers always from the browser side, the multiplexing of audio over 3 transceivers or the codec selection (VP9) but those has been like that for a while and there are other places where they have been discussed.

What do you think? Any other interesting features that you found in Google Meet? Feel free to share those here or in Twitter.

Handling back-pressure in RTC services

2022-01-30T07:48:00.004-08:00

In the context of software, back-pressure refers to actions taken by systems to “push back” downstream forces. As such, it is a defensive action taken unilaterally while under duress or if the aggregate call pattern exhibits too many spikes, or is too bursty.

This approach is commonly used in microservices infrastructures but we don't talk too much about it in the context of Real-Time Communication platforms even if they can be equally important to handle spikes of load while keeping the quality of service inside some reasonable limits. During times of spikes or burstiness, the quality could be slightly degraded but it should still be usable and the servers should never go down.

The typical techniques to apply back-pressure and get some protection against those unsustainable patterns of traffic
are buffering, dropping, and controlling the sender:

Buffering: Messages can be queued and processed at a lower rate when needed to limit the impact of spikes or high load.
Dropping: Messages or even connections can be dropped when it is not possible to process them fast enough. This can be problematic in case there are retries from the clients as it could create a multiplicative effect making the problem even worse, so it has to be applied carefully.
Controlling the sender: The receiver can provide some feedback to the sender to limit the amount of traffic that it is sending to temporarily reduce the impact on the service quality.

The next two sections present some ideas and implications of those techniques applied to the two planes that usually compose the RTC platforms (the signaling plane and the media plane).

Signaling Plane

Typically we use a websocket connection to send and receive messages related to session establishment and often also some type of presence and messaging. Given that it is a persistent TCP connection we already get some buffering for free as part of the operating system implementation that can handle some spikes in traffic. In addition to that buffering it is convenient to have a more strict buffering at connection and at room/session level to avoid a single room overloading the whole server in case of an unexpected surge in traffic in that room.

Dropping messages is not very common for the signaling plane and usually it is not an option given that we don't want to lose critical messages. But in some specific cases, it could be an option and reasonable like for example in the case of telemetry messages or sending cursor positions in a screen-sharing application.

Controlling the sender could be done explicitly (sending a feedback message to the client to suggest the client send fewer messages by adding some batching or reducing the number of videos for example) or can be done implicitly by reading slower from the buffer so that the corresponding buffer in the client-side fills up too.

Media Plane

When running media servers you get some buffering by default as part of the OS sockets implementation. The size of those buffers is something that you usually want to tune and gives you some flexibility to deal with spikes trading off delay for packet loss. For traditional use cases you probably don't want to increase it much but for high bitrate use cases where a keyframe can be really big (f.e. Stadia-like use cases) it is likely that you need to do increase those buffers. Those buffers are the first protection against small spikes but shouldn't be abused because they also increase the latency and jitter of media packets.

Dropping messages could be an option, but usually doesn't save that many resources once you have already read them also because that would trigger retransmissions (at least in case of video). So instead of dropping packets it is a better idea to disable the forwarding of specific video layers to reduce the amount of packets or in case of audio to enable dynamically dtx (discontinuous transmission) or to increase the packetization time.

Controlling the sender is probably the most common technique for the media plan. The sender can be controlled explicitly by configuring a lower sending bitrate when needed (for example using RTCP REMB messages) or can be done implicitly. In the later case, when the media servers start to get overloaded the the processing time of packets is less predictable and that introduces some jitter that as a consequence reduces the bandwidth estimation because jitter is a signal of congestion. This behaviour is something that happens automatically in most of the existing RTC solutions that include a bandwidth estimation algorithm based variability of packets delay like is the case of WebRTC.

Those were some of the techniques used in signaling and media servers that I'm aware of. What do you think? Any other ideas do you apply today to your platform? Feel free to share those here or in Twitter.

The role of Real Time Communications in the Multiverse

2021-09-13T06:15:00.008-07:00

Recently many companies have been talking about something that they usually call the metaverse. There is no single clear definition of what the metaverse is but something like this sounds close enough: "A highly connected environment with lots of interactive players and complex simulation creating rich experiences, something more than a game but less than the real world" (What is the metaverse, and why is it worth so much money?)

For me, the easiest way to understand it is to think about something similar to the world shown in the Ready Player One movie where users can interact with each other in a digital environment making use of virtual reality devices and sensors.

Perhaps the biggest company speaking publicly about the metaverse is Facebook and is betting hard on it saying that they want to move from being a "social company to a metaverse company" but there are also many gaming companies working on that direction or at least talking about it. In this blog post we can find a nice representation of the market around this trend.

With that context in mind, I started to wonder what is the role of Real-Time Communications (RTC) in this ecosystem or more precisely what are the differences or customizations that we could need to do to our existing RTC solutions to provide the features and quality needed for this futuristic metaverse vision.

To try to frame the problem I started to think about this quadrant with two types of communications (those face to face and those with remote people) and two types of worlds where they can happen (the real and the virtual worlds). Given this classification, the ones that are more interesting or at least specific of the metaverse are the face-to-face communications in the virtual world.

High-quality spatial audio

One obvious requirement will be the use of high-quality and spatial audio. To have an immersive experience in the metaverse the audio quality has to be crystal clear and the voice of the people needs to be heard coming from the place where they are supposed to be in the virtual space and with the volume corresponding to the attenuation due to the distance between user avatars in the virtual world.

All the technology needed for this is already in place
and there are many applications already doing something similar as you can see from simple apps like hubbub to fancy products like Google Starline. Some introduction to spatial audio can be found in this interesting presentation from Dolby: Improving intelligibility with spatial audio

Audio effects

The voice coming from participants also needs to be adapted to the environment where you are placed in the virtual space to sound realistic. For example, if you are in a cave the reverberation and echo need to be different than if you are in an open space.

In addition to that if you can impersonate or take different forms inside the metaverse (a robot, an animal...) it would be cool if some of those customizations also apply to your voice to change your appearance or just to "improve" your voice when you are tired or hoarse.

For the former (adaptation to the space and conditions) those effects could be applied on the receiver side while the later (voice tunning) could be applied probably in the sender side.

Large scale routing infrastructure based on virtual locations

While being in the metaverse you are communicating with people around you instead of or not only with people in predefined rooms. That means that the architecture for audio routing has the following particularities:

Routing must be based on location instead of room identifiers.
Routing must be able to scale to a very large number of participants (imagine somebody singing/shouting in a concert venue or thousands of people talking at the same time in that venue).

To address these requirements we need an architecture where the participants connect to a voice server close to their physical location to have as low latency as possible but the audio routing is based on the virtual location instead. So your server will subscribe to the audio of users with a virtual location (the geographical position in the metaverse) close to yours and when receiving the audio from those servers filter it (or even mix it) based on the distance between the source and the destination of the audio.

In addition to the distance, there are other parameters (like the presence of walls in the middle) that need to be taken into consideration to decide what are the most relevant audio streams for a user so probably the voice servers need to get access to more information from the 3d world servers than just the location.

Given the tightly coupling required with the 3d environment and the low latency overlay already required it is possible that we don't really need specific RTC servers but just to add some new capabilities to the existing gaming servers to properly route and filter audio packets in addition to other 3d interactions.

Video is secondary and replaced with the transmission of face features/expressions

My expectation would be that traditional video transmission will become less relevant and even marginal. It would be mostly replaced with very detailed face features extraction on the sender side combined with very realistic 3d rendering of faces on the receiver side. Something like super high-quality 3d animojis that will be even photorealistic for some use cases.

The technology for that is almost there and you can see demos of videoconferencing doing this way from NVIDIA and amazing digital humans in this Epic product.

Could this apply also to voice so that instead of sending our voice we only need to send its "features"? We have already seen some codecs like Lira going this way to try to reduce the bandwidth consumption and maybe that's what the future looks like in terms of audio codecs but I am not an expert and don't have a clear opinion about it.

Arbitrary data channels are already in place

A third dimension of RTC apart from audio and video is sometimes data (for example in the case of WebRTC that's provided by DataChannels). In the case of the metaverse by definition, it needs to have very low latency and high scale messaging overlay to distribute and synchronize the state of the world so I don't expect any need for data capabilities in the RTC layers.

Other features

Some standard requirements like captioning for accessibility or even recording could also apply (hopefully we don't end up in a metaverse where everything is recorded) but don't look like they have different challenges than today.

Wrapping-up

It is clear that the metaverse vision is coming one way or another and looks like one of the critical components will be the Real-Time Communications and specifically the voice communications support. Our current solutions for RTC need to evolve and go to the next level especially in terms of audio manipulation (effects and adaptation to the 3d environment) and large-scale architectures while it is possible video communications will be less relevant in these metaverse use cases.

And as usual feedback is more than welcomed either here or in Twitter.

WebRTC Video Codecs performance evaluation

2021-02-04T00:55:00.019-08:00

The standard and most popular codecs in WebRTC are VP8 and H.264 but those are not the only options we have. VP9 has been available for a while and some big services are using it and AV1 support has been recently added to Chrome.

When comparing codecs there are interesting considerations like interoperability and licensing but probably the most important factors are how good the codec is in terms of compression and how cheap the codec is in terms of cpu&memory usage.

The compression ratio is usually the first thing we look at and there are many comparisons available for that, but the resource consumption is equally important if we want to be able to use the codecs for real time use cases.

Given that AV1 is available in the Chrome Canary versions I decided to run some tests to get estimation of where we stand in terms of cpu usage for the 4 available codecs in WebRTC ecosystem. The idea of the test is to compare the whole video pipeline with those 4 codecs and not just the codecs in isolation.

Testing environment

The tests were done with a simple web page establishing a connection between 2 PeerConnections (one sending and the other receiving). The test page is here in case you want to run the tests yourself:

https://jsfiddle.net/tvo7czxs/

The tests with that page were performed changing 3 variables:

Codec: VP8, VP9, H264, AV1
Resolution: HD, VGA, QVGA
Bitrate: 200Kbps, 800Kbps, 2Mbps

If you look at the test page it is easy to change those 3 parameters to run the tests with other configurations or in other devices.

The Chrome version used was the latest one synced from git this week (1/2/21) and the tests were ran in a MacBook Pro (2.4 GHz 8-Core Intel Core i9).

To check the CPU usage I just looked at the average CPU consumed by the Chrome process in the system Activity Monitor after waiting 30 seconds to give time for the stabilisation of bandwidth estimation and resolution/framerate adaption inside WebRTC. When the results below say 100% that means 1 full core of the machine.

Nothing too fancy but hopefully easy enough to understand.

In that environment I ran the 36 combinations of the parameters a couple of times, averaged the results and summarised them in the following sections:

QVGA results

For QVGA resolution the results were as expected, with VP9 requiring a bit more CPU than VP8 and AV1 requiring almost 2x CPU as VP8. H.264 was the one requiring less CPU usage as it is using hardware acceleration for that.

% of CPU usage with different codecs
QVGA	200Kbps	800Kbps	2Mbps
VP8	18	22	28
VP9	20	28	33
H264	10	14	15
AV1	36	46	50

VGA results

For VGA the results were not very different but at low bitrates only VP9 was able to maintain the resolution and AV1 was using more than 1 core when increasing the bitrate limit to 2 Mbps. Quality of H.264 at 200Kbps was really bad with very low framerate and lot of blocking as adaptation is apparently not working well in this case in Chrome.

% of CPU usage with different codecs
VGA	200Kbps	800Kbps	2Mbps
VP8	18	30	46
VP9	26	38	40
H264	8	13	16
AV1	33	80	150

HD (1280x720) results

HD results were similar to VGA but AV1 was not able to encode the original resolution and downscaled it in the tests with all the bitrates. H264 also behaved very poorly at low bitrate and the difference between VP8 and VP9 cost was much higher than with VGA.

% of CPU usage with different codecs
HD	200Kbps	800Kbps	2Mbps
VP8	17	30	65
VP9	30	50	80
H264	8	12	16
AV1	35	48	90

(In addition AV1 with HD resolution crashed often with a memory issue in a Mac related code, but maybe the bug is fixed at the time you read this)

Encoding vs Decoding cost

I did one more test to split the cost between encoding (Sender side) and decoding (Receiver side). That test was performed for VGA at 800 Kbps and the results were the next ones for the four codecs under consideration.

The results are not very different but the decoding in case of VP9 and AV1X was relatively cheaper compared with the encoding.

Comparing just the decoding cost with different codecs it looks like AV1 is around 2x more expensive than the others. VP9 is slightly more expensive than VP8 and VP8 slightly more expensive than H264 but without big differences between those three.

Conclusions

Having new codecs is amazing and there is no doubt AV1 is the future of real-time video communications but it looks like we should wait a bit to be able to use it for general purpose videoconferencing applications. In the mean time we can probably use it for specific use cases like broadcasting, dedicated powerful devices or to encode the low resolution versions of the video streams when using simulcast.

For other use cases VP8 and VP9 look like still the best options unless you don't worry that much about low bitrate cases or you are using high resolutions and battery/cpu consumption is a big concern and you can consider H264.

Also there is apparently a new libaom patch coming that could improve the performance by a 15% so it would be good to repeat these tests with future versions of Chrome and different devices (AV1 could be potentially more optimized for ARM CPUs).

If you want to replicate the test and share your data I will update these tables/graphs with the average of the measurements. And as usual feedback is more than welcomed either here or in Twitter.

Thx a lot for the early feedback from @murillo, @elminiero, @dan_jenkins, @maniksach.

Speech recognition in Hangouts Meet

2019-10-24T06:21:00.000-07:00

There are many possible applications for speech recognition in Real Time Communication services live captions, simultaneous translation, voice commands or storing/summarising audio conversations.

Speech recognition in the form of live captioning has been available in Hangouts Meet for some months but recently it was promoted to a button in the main UI and I have started to use it almost every day.

I'm mostly interested in the recognition technology and specifically on how to integrate DeepSpeech in RTC media servers to provide a cost effective solution but in this post I wanted to spend some time analysing how Hangouts Meet implemented captioning from a signalling point of view.

At a very high level there are at least three possible architectures for speech recognition in RTC services:

A) On device speech recognition: This is the cheapest option but not all the devices have support for it, the quality of the models is not as good as in the cloud ones and it requires some additional CPU usage that can be a problem for limited devices.
B) Recognition in a separate server: This is very inefficient in terms of network because the client needs to send the audio twice in parallel, it is expensive for the service provider but doesn't require changes in the media server.
C) Recognition from the media server: This is very efficient from the client and network point of view but requires changes in the media server and it is expensive for the service provider.

Given the fact that Google owns its own Speech Recognition service and because of that the cost is probably not their biggest concern it looks like the most reasonable approach for Hangouts Meet could be the Option C. So I tried to confirm it and see how the transcriptions are sent between the media server and the browser.

First thing I did was to take a look at the minified code and to the HTTP requests sent by Hangouts Meet web page during the conversation but I couldn't find and reference to stt/transcription/recognition or anything like that. I also ran a very simple WebSpeech snippet in the browser console while connected to Hangouts and saw the results where different in the Hangouts page and in the on-device recognition (It was better in the page).

So next I decided to take a look at the DataChannels in chrome://webrtc-internals and voilà, there were a lot of messages received while I was talking so those had to be the messages with the speech recognition content.

After that I checked in webrtc-internals events and I saw there was a specific DataChannel created for the captions the first time you enable them in addition to the default DataChannel that it is always open in Hangouts Meet.

To try to figure out the exact options of that DataChannel and the format of the information sent in those messages I replaced the RTCPeerConnection.createDataChannel API with my own to be able to intercept those calls using this simple snippet in the browser console.

With that in place I saw the unreliable DataChannel for the captions is created with maxRetransmissions=10 and the payload is binary data probably in protobuf format. Still we can convert it to string and see a couple of fields with the user id and the text inside:

And that's all for today. In the next post I'll try to share some information around my DeepSpeech experiments. As usual feedback is more than welcomed either here or in Twitter.

Implementing P2P-SFU transitions in WebRTC

2019-04-12T14:05:00.004-07:00

One of the more disruptive aspects of WebRTC is the ability of establishing P2P connections without any server involved in the media path. However this doesn’t scale well for multiparty audio/video calls as the bandwidth and cpu required for a full mesh of N:N P2P connections is too much in most of the cases.

But the fact that you support multiparty calls doesn’t mean that you shouldn’t consider using P2P connections when there are only two participants in the room and switch back to the SFU when the third participant joins or when you need to enable some recording or broadcasting capabilities only available in the SFU.

In fact this approach of using P2P when only two participants are connected and switching to the SFU has been successfully implemented in many products in the last years (Jitsi, Hangouts, Facebook). The implementation of this feature is not trivial but it provides enough advantages to be worth it:

Minimise the network hops reducing the end to end delay and the chances of having congestion. This is specially interesting in countries where you don't have datacenters or meetings inside corporate networks.
Be able to use all the features included in WebRTC. For example even if your SFU only supports REMB based congestion control you can use the newer transport feedback based congestion control at least while in P2P mode.
Reduce the infrastructure costs (specially the bandwidth cost).

There are different ways to implement this feature but at a high level the biggest decision is probably: Are you still connected to the SFU while having the P2P connection or not?

Having the SFU connection always established makes the switching faster but on the other side not having that connection open all the time is more efficient both for the endpoints and the infrastructure.

I did a quick review of the most popular WebRTC services / platforms to see how they do it and this is what I saw:

As you can see there is no obvious winner based on the choices of those popular platforms and both approaches have its merits.
Note: I tested other services, platforms and opensource implementations and couldn’t find this feature. If I’m wrong please let me know the name and I will update the table.

Apart from that decision about maintaining the SFU connection open or not you also need to decide when and how to do the switching and there are many optimisations that can be done here specially in mobile native apps to minimise the transition and make it as seamless as possible. But that can be a topic for other day... :)

Sending Packet Loss Feedback in WebRTC SFUs

2018-12-10T05:53:00.003-08:00

One of the responsibilities of WebRTC SFUs is to receive and send RTCP packets. RTCP packets include different types of feedback about audio and video streams and one of the most important RTCP packets is the Receiver Report (RR).

RR packets are sent from the receiver of the media stream towards the sender of that media stream. In case of an SFU the RR are generated and sent from the SFU to the media stream Sender and also from every stream Receiver to the SFU (Figure 1).

The feedback sent inside RR packets include fields to calculate the round-trip-time delay, the jitter and the packet loss introduced by the network.

The packet loss reported in these RR packets is important because the audio and video being sent will be adjusted based on that parameter:

In case of audio streams the packet loss in the network modifies the level of robustness of the OPUS codec. In presence of high packet loss the sender increases the level of redundancy of the forward error correction (FEC) included in the audio packets.
In case of video streams the packet loss modifies the video bitrate encoded and sent. In presence of high packet loss the sender decreases the bitrate sent to reduce the possible congestion in the network.

Based on that behavior the question is... How should a SFU calculate the packet loss reported in the RR packets sent from the SFU to the Sender to get the best possible user experience?

In the next sections you can find a suggestion for how to handle this in three different types of streams: Audio, Video with simulcast and Video without simulcast.

Audio

Opus FEC is sent inband and because of that FEC is end to end (cannot be added/updated in an SFU) and everybody in the room will receive the same level of FEC.

If we want audio FEC to work well for the participant with the weakest link then we have to make sure we report the worse packet loss to the sender.

Because of that the packet loss reported from the SFU to the sender should be the worse packet of the receivers (max(PL1, PL2)).

For example the packet loss reported to the sender will be 20% if one of the receiving participants is experiencing a 20% packet loss in the downlink even if the sender and rest of receivers are in perfect network conditions.

Video with Simulcast

When using Simulcast the SFU has the ability to send a different video quality to each participant so there is no need to adapt the sender stream to any specific participant.

Because of that the packet loss reported from the SFU to the sender should be the packet loss in that link (Sender-SFU) no matter what the packet loss received the Receiver 1 and 2 is.

For example the packet loss reported to the sender will be 0 if the sender uplink is good even if the receivers are experiencing big packet loss in the downlink. That will be corrected with retransmissions and bitrate adaptation done in the SFU by selection a lower resolution/layer to be forwarded to those participants.

Video with No-Simulcast

When there is no Simulcast the SFU has to send the same video quality to every participant. Because of that it is important that the sender adapts the sending bitrate to the participant with the weakest link (and/or disable the video for some participants).

This bitrate adaptation is mostly controlled by REMB RTCP packets where the SFU includes the bandwith available in the worse receiver. But packet loss has also an impact on top of the bitrate reported in the REMB packets so we need to decide what packet loss to include in the RR packets.

In this case I think both approaches described in the previous sections can work reasonably well. Either a) the SFU reports back the worse packet loss of the receivers or b) use the packet loss of each receiver to estimate its bandwidth and report only the sender side packet loss in the RR packets.

Note: If you use P2P connections when there are 2 participants in the room and switch to the SFU with Simulcast when there are 3 or more participants in the room you should be almost never in this Video with No-Simulcast and multiple participants case.

You can follow me in Twitter if you are interested in Real Time Communications.

Measuring WebRTC video quality for different bitrates - Playing with VMAF

2018-10-05T07:19:00.003-07:00

I've been wanting to play with Netflix Video Multi-Method Assessment Fusion (VMAF) for a while and yesterday I found the time and the motivation to give it a try.

Netflix VMAF is an algorithm to generate a video quality score by comparing a reference image/video with a distorted image/video. To do that VMAF calculates scores using tradicional image quality metrics like VIF or DLM and then aggregate them using a Machine Learning model (SVM) trained with the videos and scores coming from real users. Smart, isn't it? (You can see a high level description of those metrics that are aggregated in this Netflix post or the Wikipedia page)

It is important to notice that VMAF works in a per-frame base so it is NOT a good tool to measure the quality impact of many artefacts happening in Real Time Communications (delays, reduced/frozen framerate, audio/video desync). However we can use it to measure the impact of different encoding settings like the average bitrate of the encoding.

As you can expect coming from Netflix VMAF was designed for video streaming and trained using images from movies. Anyway I was interested on seeing how it performs with mobile videoconference like videos so I recorded a short typical VGA video with a talking face not very stable.

I reencoded that sample video in VP8 using ffmpeg with different bitrates (50kbps, 100kbps, 200kbps, 400kbps, 600kpbs, 800kpbs, 1.2mbps, 2mpbs) and then used the ffmpeg2vmaf command line tool to calculate the score of those videos and presented them in the following graph:

VMAF scores for

What we can see in this test is that beyond 600 (or even a little bit lower) the quality improvement is not that high and beyond 1200 it is barely noticeable. Remember that these results are based on VMAF default model (not tuned for videoconference videos) and for my specific test video but the results don't look very different that what our experience with real users in production tells us.

This kind of test can help us decide the max bitrate we want to use for our WebRTC conferences, although there are other implications beyond quality like battery consumption and the results depend on the type of video, use case and how picky your users of your application are. That's the reason why we have to be careful when playing with video bitrates in production. As an example Facebook explained how increasing the bitrate lead to lower user scores because of the implications in battery consumption. In Houseparty we always do A/B testing to quantify the impact of any relevant change like this and decide the optimal video bitrate for our specific use case.

Google is including VMAF in the WebRTC test suite and implementing some frame alignment to overcome the limitation of having to compare a specific reference frame with the corresponding distorted one. It would be nice if in the future we could expand the VMAF idea including new metrics in the ML algorithm to account for delays, framerate or video desynchronization. That core idea of multi-method fusion looks very powerful!

You can follow me in Twitter if you are interested in Real Time Communications.

How are Images/Videos sent in WhatsApp?

2018-05-08T00:08:00.003-07:00

I've been involved in the development on different IM&P services in the past and always one of the core features was the ability to share media files (audio, video, pdf...) with other users or groups.

Probably the most popular of these services was TU Me that was based on SIP but used HTTP to upload and download the files from a central storage. SIP was used to send the message with the url to the other end so that he can download the media file.

In cases where I had to use XMPP there were typically two approaches at least if you need to provide offline messaging:

Some inefficient inband alternatives where the media files are sent using the XMPP connection.
Custom (or not widely supported like xep-0363) extensions based on HTTP Uploads.

But today I was curious on how this is done in WhatsApp. I found this opensource python implementation (yowsup) that looks great and got an idea on how it could be done but I wanted to confirm what is the protocol used by the most recent WA clients and also get some captures of the messages to see how everything looks like.

Obviously the easiest way was to use to WA Web Client because you have the Chrome developer tools in Chrome so I tried sending and receiving some files from the browser.

Sending a file

1/ Request a url to upload the media file

The first thing the client does when it needs to send a media file is to request an upload url from the server. This is done using the signaling channel over WebSockets with a message with the action "encr_upload", the type of the content ("image" or "video") and an long id that looks like a hash of the file to be sent.

The server replies with a "status=200" and a dedicatdd url where the file has to be uploaded. In my case the url was: https://mmg.whatsapp.net/u/f/CIX33...._kOA/Ajlc....78Xf

2/ Upload the media file

The file is uploaded to the url received in the previous message using a HTTP POST as you can see in the next capture:

It looks like if the file is uploaded encrypted (appart from using HTTPS) probably as part of the E2E encryption feature in WhatsApp.

3/ Send a message to the recipient of the media file

After the file is uploaded to the mmg.whatsapp.net server the client sends a message using the WebSocket connection to send the actual message to the other participant (yellow line in the first capture). This message was bigger than expected (2-10K) so I thought it was very likely to include a thumbnail/preview.

I confirmed that in the receiver side. The size of the preview image in base64 shown in the UI was more or less the size of that signaling message.