Giacomo Vacca

Understanding WebRTC State Machines

2026-02-02T11:07:00.001+01:00

Understanding WebRTC State Machines

Specific transports, aggregate states, and the Chrome/libwebrtc mental model

WebRTC is one of those technologies that appears deceptively simple on the surface. You call createOffer(), exchange some SDP, and suddenly two browsers are streaming video to each other. But beneath that simplicity lies a set of interlocking state machines that govern every aspect of the connection's lifecycle—from signaling and ICE candidate exchange to DTLS handshake completion.

If you've ever stared at a WebRTC debugging log wondering why connectionState reads "connecting" while iceConnectionState says "connected", this article is for you. The key insight, and the one the Chrome/libwebrtc codebase is built around, is that WebRTC state is organized into two distinct tiers: specific state machines that live on individual transports, and aggregate states on RTCPeerConnection that are derived—computed—from those transports.

This article walks through each layer in detail, explains how the aggregate states are calculated, and ties it all together with a narrative of a typical connection's lifetime.

The Two-Tier Architecture

The easiest mental model for reasoning about Chrome's WebRTC behavior is to think of it in two layers.

Specific state machines live on individual transports. Each ICE transport, each DTLS transport, and the SCTP/DataChannel layer maintain their own state, tracking their own progress through their respective protocols independently.

Aggregate states live on the RTCPeerConnection object and have no independent transition logic of their own. Instead, they are computed from the set of currently-relevant transports (the active transceivers plus the optional SCTP transport). Think of them as dashboards—read-only views that reduce the complexity of multiple transports into a single summary value using well-defined precedence rules.

This distinction matters enormously for debugging. When an aggregate state like connectionState reports "failed", the question is never "why did connectionState fail?"—it doesn't fail on its own. The question is "which specific transport entered the failed state, and why?"

The Specific State Machines

SDP/JSEP Signaling State

The signalingState property tracks the progress of the offer/answer exchange—and nothing else. It is a common misconception that signaling state reflects connectivity. It does not. A peer connection can be in signalingState: "stable" and have no working media path whatsoever; "stable" simply means that the last offer/answer round-trip has completed successfully.

The signaling states are:

State	Meaning
`stable`	No offer/answer in progress; the default and resting state.
`have-local-offer`	A local offer has been set, awaiting a remote answer.
`have-remote-offer`	A remote offer has been received, awaiting a local answer.
`have-local-pranswer`	A local provisional answer has been set.
`have-remote-pranswer`	A remote provisional answer has been received.
`closed`	The peer connection has been shut down.

Every state can transition to closed when close() is called. The key transitions to remember are: calling setLocalDescription(offer) from stable moves to have-local-offer, and receiving the remote answer returns to stable. The same pattern holds symmetrically for the answering side.

Key Takeaway: signalingState is purely about offer/answer progress. It tells you nothing about whether ICE has found a path or DTLS has completed its handshake.

ICE Transport State

The RTCIceTransport.state property is where the real connectivity story lives. Each ICE transport independently manages candidate pair checks, consent verification, and failure detection.

State	Meaning
`new`	The transport exists but has not begun checking candidates.
`checking`	Candidate pair checks are underway.
`connected`	A viable pair has been selected, though additional checks may still run.
`completed`	Gathering is done, end-of-candidates has been signaled, and a final pair is selected.
`disconnected`	A transient loss of connectivity; the transport is still trying to recover.
`failed`	All checks are exhausted and no working pair could be established.
`closed`	The transport has been shut down.

What makes the ICE state machine interesting—and occasionally surprising—are its back edges. Unlike a simple linear progression, ICE can move backwards:

connected → checking: When consent is revoked on the active pair, the transport drops back to re-check alternatives.
completed → checking: An ICE restart (triggered by renegotiation) resets a completed transport back to checking.
connected → disconnected: A transient network interruption moves the transport to disconnected, from which it may recover or eventually fail.
disconnected → checking: If new candidate pairs become available during a disconnected period, the transport re-enters checking.

These back edges are critical for understanding connection lifetime. A connection that appears "stable" in the completed state can regress to checking during an ICE restart, and a brief network blip can trigger a disconnected → checking → connected cycle without the user ever noticing.

Debugging Tip: The disconnected state is transient by design. If you see it in logs, wait before concluding there's a problem—ICE may be in the process of recovering. The failed state is the terminal one to watch for.

DTLS Transport State

The RTCDtlsTransport.state property tracks the DTLS handshake that secures the media path. It is comparatively simple—a mostly linear progression from new through connecting to connected, with failed as the error terminal.

State	Meaning
`new`	The DTLS handshake has not started.
`connecting`	The DTLS handshake is in progress.
`connected`	The handshake completed and the fingerprint was verified.
`failed`	The handshake failed (e.g., fingerprint mismatch, DTLS alert).
`closed`	The transport has been shut down via close_notify or peer connection closure.

Failure in DTLS is almost always a security-related event: a certificate fingerprint mismatch (which can indicate a man-in-the-middle attempt or an SDP error), a DTLS alert, or a timeout during the handshake. Unlike ICE, DTLS has no recovery path from failure—once it fails, the transport is done.

The Aggregate (Derived) States

The aggregate states on RTCPeerConnection exist as a convenience for application developers. Rather than requiring you to iterate over every transport and compute the overall status yourself, the spec defines derivation rules that collapse the individual transport states into summary values. There are three aggregate states to understand.

Aggregate ICE Connectivity: iceConnectionState

The iceConnectionState property is computed from the set of all currently-relevant ICE transports using the following precedence rules, evaluated in order:

closed — if the peer connection itself is closed.
failed — if any ICE transport is in the failed state.
disconnected — if any ICE transport is disconnected.
new — if all ICE transports are new or closed (or none exist).
checking — if any ICE transport is new or checking.
completed — if all ICE transports are completed or closed.
connected — if all ICE transports are connected, completed, or closed.

The precedence ordering is important. Notice that failed dominates: a single failed transport poisons the entire aggregate, regardless of how many other transports are healthy. Similarly, disconnected takes precedence over positive states. This is a "worst-case-wins" model.

One subtlety worth noting: when ICE transports are created or discarded due to signaling changes—bundle policy changes, RTCP multiplexing adjustments, or adding new media lines—the aggregate state can "jump" forward without any individual transport changing state. A new transceiver added mid-session can momentarily pull the aggregate back to checking even if existing transports are all connected.

Overall Session State: connectionState

The connectionState property is the highest-level health indicator on RTCPeerConnection. It combines the aggregate ICE connection state with the DTLS transport states. Its precedence rules are:

closed — if the ICE aggregate is closed.
failed — if the ICE aggregate is failed or any DTLS transport has failed.
disconnected — if the ICE aggregate is disconnected.
new — if the ICE aggregate is new and all DTLS transports are new or closed (or no transports exist).
connected — if the ICE aggregate is connected and all DTLS transports are connected or closed.
connecting — otherwise (the catch-all bucket).

This is the state that explains the puzzle from the introduction. connectionState can read "connecting" even when iceConnectionState shows "connected" because the DTLS handshake hasn't finished yet. The ICE path is established, but the secure channel over that path is still being negotiated. Since neither the failed nor disconnected rules apply, and DTLS isn't yet connected, the catch-all connecting state kicks in.

Key Takeaway: connectionState = "connected" means both ICE has found a working path and DTLS has completed its handshake. It is the definitive signal that the media channel is fully operational.

Aggregate ICE Gathering: iceGatheringState

Separate from connectivity, the iceGatheringState tracks candidate gathering progress across all transports. It follows a simple progression: new → gathering → complete. It can return to gathering if new network interfaces or STUN/TURN servers become available. In practice, with trickle ICE, the gathering state often reaches complete well before the ICE connectivity state finishes checking all pairs.

A Connection's Lifetime

Tying all these state machines together, here is the typical progression of a successful WebRTC connection from offer to media flow:

Layer	State Progression
Signaling	stable → have-local-offer → stable (after answer)
ICE Gathering (agg.)	new → gathering → complete
ICE Transport	new → checking → connected → completed
ICE Connection (agg.)	new → checking → connected / completed
DTLS Transport	new → connecting → connected
Connection (agg.)	new → connecting → connected

Notice the ordering: signaling completes first (the offer/answer exchange), then ICE gathering begins. As candidates trickle in, ICE connectivity checks start. Once ICE finds a viable pair (connected), DTLS begins its handshake over that path. Only when DTLS also reaches connected does the aggregate connectionState finally report "connected".

During this progression, there is an inevitable window where ICE is connected but DTLS is still negotiating. This is normal and expected. The aggregate connectionState will show "connecting" during this window—not because anything is wrong, but because the derivation rules correctly reflect that the full secure channel is not yet established.

Practical Debugging Advice

Armed with an understanding of the two-tier architecture, here are some practical principles for debugging WebRTC connections.

Always drill down from aggregate to specific. When connectionState or iceConnectionState shows something unexpected, don't try to reason about why the aggregate changed. Instead, inspect the individual RTCIceTransport and RTCDtlsTransport objects to find which specific transport triggered the change.

Respect the back edges. ICE is not a one-way street. Consent revocation, ICE restarts, and transient network issues can all cause state regressions. Build your connection-monitoring logic to handle backwards transitions gracefully, especially the connected → disconnected → checking → connected cycle.

Distinguish transient from terminal. The disconnected state is transient; failed is terminal. Don't show error UI on disconnected—give ICE a chance to recover. Reserve error handling for the failed state, or for disconnected states that persist beyond a reasonable timeout.

Remember what connectionState = "connected" actually means. It means ICE has a working path and DTLS has completed its handshake. If you need to know that media can flow securely, this is the state to watch—not iceConnectionState alone.

Watch for aggregate jumps from signaling changes. Adding a new transceiver or changing bundle policy mid-session can cause aggregate states to shift even when no individual transport changed. If your state-monitoring code triggers unexpectedly during renegotiation, this is likely why.

Conclusion

WebRTC's state management is a layered system. Specific state machines on individual transports do the real work of establishing connectivity and security. Aggregate states on RTCPeerConnection provide a convenient summary by applying precedence rules across the set of active transports. Understanding this two-tier architecture—and especially the derivation rules that connect them—is the key to reasoning confidently about WebRTC connection lifecycle, debugging unexpected state transitions, and building robust real-time communication applications.

References

W3C, WebRTC: Real-Time Communication in Browsers (W3C Recommendation). https://www.w3.org/TR/webrtc/. Sections referenced: RTCSignalingState enum, RTCIceConnectionState derivation, RTCPeerConnectionState derivation, RTCIceGatheringState, and non-normative signaling transition diagrams.
MDN Web Docs, RTCIceTransport: state property. https://developer.mozilla.org/en-US/docs/Web/API/RTCIceTransport/state. Referenced for per-transport ICE state definitions and back-edge transition descriptions.
MDN Web Docs, RTCDtlsTransport: state property. https://developer.mozilla.org/en-US/docs/Web/API/RTCDtlsTransport/state. Referenced for DTLS transport state definitions and failure conditions.
IETF RFC 8829, JavaScript Session Establishment Protocol (JSEP). https://www.rfc-editor.org/rfc/rfc8829. Background reference for the offer/answer model and SDP handling that drives signalingState transitions.
IETF RFC 8445, Interactive Connectivity Establishment (ICE). https://www.rfc-editor.org/rfc/rfc8445. Background reference for ICE candidate pair checking, consent verification, and the state model underlying RTCIceTransport.
IETF RFC 6347, Datagram Transport Layer Security Version 1.2. https://www.rfc-editor.org/rfc/rfc6347. Background reference for the DTLS handshake and alert mechanisms underlying RTCDtlsTransport.

This article was written in collaboration with GPT-5.2 and Claude Opus 4.5. The technical research and notes were developed with GPT-5.2, and the article was drafted and formatted with Claude Opus 4.5.

A rudimentary dissector for Wireshark (Lua)

2025-12-21T20:29:00.005+01:00

Wireshark offers a simple but effective option to extend its capabilities, by using Lua dissectors.

Just to give an example, recently received a pcap file containing some traffic (it was RTP) encapsulated inside a UDP header. The reason for encapsulation was transporting over a VPN.

I'm in such a habit to look into RTP streams on Wireshark, that I have a setting that tells it to try an interpret any UDP packet automatically as carrying RTP (I wrote how here). That also failed.

So Wireshark was not able to interpret those frames as RTP (or anything else, for what matters), and I remembered time ago writing a custom dissector. Wireshark allows to do that simply in Lua and add it as a plugin.

The code is available here. I just had to make it available inside '$HOME/.local/lib/wireshark/plugins/`.

It's not a sprint

2025-11-20T22:05:00.006+01:00

You’ve surely heard at least once in your life somebody saying “Calm down: it’s not a sprint, it’s a marathon”.

People use this phrase to try and slow things down, but it misses the point.

The first misconception is that a marathon is something you do slowly. A marathon is fast; it’s just as fast as possible for that athlete in that moment.

Most people couldn’t hold an elite marathoner’s pace for even 100 meters. It’s a marathon, but it’s not slow.

It’s easy to make a point that running a marathon is tough, not just for the distance but for the speed you try to keep.

What’s maybe harder to see is that a marathon is much easier than many life challenges.

You can simply prepare for a marathon. You get to the starting line with months or years of preparation. You tried progressively longer and tougher sessions, simulating the marathon effort. That’s a privilege. Many hurdles you’ll face in life will just appear in front of you while you were thinking of something else.

A marathon has a fixed length and a fixed course that’s known in advance. Life challenges are typically faced only once, making you very unprepared.

You can run the same marathon many years in a row, each time learning something more about that specific event. In life, it’s never clear when a challenge will end. Or if it will end.

Nobody tries to make you fail at completing a marathon, apart, I guess, very rare exceptions. That’s not the case for many life endeavours.

You can quit a marathon. Nobody will blame you for it. There’s basically no risk that somebody will be harmed if you fail.

If you try to run in the wrong direction, somebody will likely tell you and help you get back on track. In life, you’d be lucky if somebody even notices. And the ones who notice may just stay quiet.

And finally the only way a marathon can disappoint you is if you have unrealistic expectations. People can disappoint you even if the expectations are modest.

So maybe next time somebody will tell you “Calm down: it’s not a sprint, it’s a marathon” you’ll want to answer “You wish”.

Decrypt SDES SRTP from pcap

2025-03-12T10:27:00.005+01:00

If you have a pcap file with encrypted RTP (SDES SRTP) and have access to the SIP signalling to see the keys, these instructions will help you decrypt the RTP payload and save it as raw audio. Optionally, depending on the codec, you can then import the raw audio in Wireshark and save it as an audio file.

Steps

Build ‘srtp-decrypt’

These are the steps:

Build ‘srtp-decrypt’
Get crypto key for each direction
Use the decrypt.sh bash script to decrypt into raw audio
Import the raw audio into Wireshark as hex dump
Generate the wav file
Generate the wav file as two separate channels, one per direction

Follow instructions here:

https://github.com/gteissier/srtp-decrypt

Assume the binary will be produced in a path like ‘/root/code/srtp-decrypt/srtp-decrypt’

Get crypto key for each direction

Access the SIP involved and take note of the ‘crypto’ line from the SDP for each direction.

e.g.:

v=0

...

m=audio 16398 RTP/SAVP 96 8 97

c=IN IP4 xxxxx

...

a=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:K7oAZZ5Fm9fePS5/t1ac00000000000000000000

...

The crypto key for the RTP stream produced by this party is K7oAZZ5Fm9fePS5/t1ac00000000000000000000

Use a bash script to decrypt into raw audio

Use this simple bash script to perform the actual decryption:

#!/bin/bash

set -x

SRTP_DECRYPT_BIN=/root/code/srtp-decrypt/srtp-decrypt

HEADER_SIZE=48

CRYPTO_KEY=5eKSVpIOePOgjjfep21R3b000000000000000000

RTP_PCAP=outbound.pcap

RAW_DATA=outbound.raw

${SRTP_DECRYPT_BIN} -d ${HEADER_SIZE} -k ${CRYPTO_KEY} < ${RTP_PCAP} > ${RAW_DATA}

Where:

CRYPTO_KEY needs the value of the crypto key seen earlier.

RTP_PCAP has the pcap with the RTP stream that you want to decrypt.

RAW_DATA is the output file where to put the decrypted raw audio.

HEADER_SIZE may vary, depending on the presence of the SLL header in the capture.

Import the raw audio into Wireshark as hex dump

In Wireshark:

File → Import from Hex Dump
Select the file with the raw audio
Offsets: Hexadecimal
Timestamp format: %M:%S.%f
Encapsulation type: Ethernet
UDP: source port 10000, destination port 20000
Import

Then go to:

Telephony → RTP → RTP Streams
Play
Select stream
Export as Synchronised audio file

If for example you have two RTP streams, one per direction for the same call, you can generate the wav file with two separate channels, one per direction.

Once you have the raw audio for both directions, you can use Audacity (https://www.audacityteam.org/) to import the audio.

Open the first file.
File → Import Audio for the second file.
Select the first audio, right click, “Set as Stereo file”
Save

Wireshark setting to interpret UDP as RTP automatically

2024-07-11T16:01:00.003+02:00

Before I forget again, a Wireshark setting that can help saving time by trying to interpret any UDP as RTP, if possible:

Analyze --> Enabled Protocols... --> Search for RTP and enable at least 'rtp_udp'

Without that change, when SIP signalling is not present (or it's encrypted) Wireshark would not understand automatically that UDP packets may be RTP.

This is particularly true for example for WebRTC calls, where signalling happens elsewhere and is not available to Wireshark.

This will also save the reader some time if you're used to right click and 'Decode As...' to achieve the same.

About ICE negotiation

2022-10-27T13:32:00.005+02:00

Disclaimer: I wrote this article on March 2022 while working with Subspace, and the original link is here: https://subspace.com/resources/ice-negotiation . This post in my personal blog is a way to ensure it doesn't get lost. There is nothing service-specific in it, I've made only minor edits and I hope it can be a good technical reference on the topic.

WebRTC is a set of protocols that allow applications, typically running on Web browsers, to exchange media (audio, video, data) with other entities.

Before media can flow, however, the WebRTC entities need to discover what type of connection is possible, and among the possible connections, what’s the best to be used. This needs to happen as fast as possible, so that users can perceive the service as instantaneous as possible.

WebRTC includes protocols like STUN and TURN that are designed to facilitate the establishment of connections when a direct connection is not possible. The typical case is a computer inside a home or office network, with a private IP address, and able to reach the public Internet only through an address translation (NAT).

STUN helps in discovering the IP address and port from where a computer enters the Internet, and in some circumstances that IP address and port can be used by other entities to reach that computer. STUN is also used for keeping such bindings alive.

TURN provides a way for two entities to communicate when they are behind two different symmetric NATs, or when one is behind a firewall that restricts outbound traffic to only some UDP or TCP ports. TURN uses STUN as the underlying protocol, adding requests, responses and indications to accomplish media relay.

STUN and TURN play a role in the ICE negotiation process. ICE, Interactive Connectivity Establishment, is a protocol that allows the dynamic discovery of the best way to establish a connection for entities that may be behind NAT.

All WebRTC clients use ICE before media can flow.

There are three main phases: the gathering of candidates, the connectivity checks, and the nomination of the candidate pairs to be used.

The ICE candidates are simply transport addresses (IP address, port and transport type) that can potentially be used to communicate (send and receive media) and that the ICE client collects and shares to the other party through some form of signalling.

There are three main types of ICE candidates: 'host', 'server reflexive' and 'relay'.

'host' candidates refer to transport addresses that are directly visible by the client, where the client can start listening for incoming connections or packets. Computers behind NAT may only have private IP addresses as 'host' candidates, but they are potentially usable if the other party belongs to the same network, or depending on the type of NAT.

'server reflexive' candidates are the ones discovered through the interaction with a STUN server. The client sends a Binding Request to the STUN server and receives a Binding Success Response with a MAPPED-ADDRESS containing the source IP address and port from where the request was received. There may be more than one level of NAT, and that transport address represents the outmost one.

'relay' candidates refer to allocations reserved on a TURN server. The ICE client requests an Allocation of a relay, and after successful authentication the TURN server provides a RELAYED-ADDRESS containing the transport address allocated on that server for the client.

These types of ICE candidates have different priorities, 'host' being at highest priority and 'relay' at lowest priority; this is a way to privilege direct interconnection when possible (but that not necessarily represents the best solution in terms of connection quality and in general of Quality Of Experience).

This diagrams shows a typical process where ICE candidates are gathered and sent to the other party:

In this example the candidates are communicated as soon as they are retrieved. This technique is called Trickle ICE and was designed to ensure that the connectivity checks can happen as soon as the candidates are available.

Without Trickle ICE, the WebRTC client would need to wait for all the candidates to be collected before sending an offer or an answer, increasing the session set up time.

The candidates are transmitted over a signalling system established between the two parties. This is outside of the WebRTC specifications and application-specific.

The WebRTC client will then receive the ICE candidates from the other party, and it will build a list of “candidate pairs”: each local candidate will be paired with each remote candidate.

After this operation the connectivity checks can begin.

Let’s see it with an example, assuming UDP as transport for all cases. The WebRTC client has a local host candidate, IP1:port1, and has received a remote host candidate, IP2:port2. It builds a “candidate pair” with the two: {IP1:port1, IP2:port2}.

The WebRTC client will start sending STUN Binding Requests with source IP1:port1 and destination IP2:port2. These requests use STUN short-term authentication, and contain a username and password that were previously exchanged when transmitting the candidates inside the SDP offer/answer.

If the Binding Request reaches the other party on IP2:port2, then the other party will authenticate the request, and respond with a Binding Success Response, including the MAPPED-ADDRESS attribute, containing the source of the request.

If the Binding Success Response reaches the WebRTC client, then it will identify the request by looking into the transaction ID and mark the check as Successful, and so suitable for exchanging media.

If for any reason the Binding Success Response is not received, then the candidate pair will remain in a In Progress state for some time, and then move to Failed: that pair cannot be used to exchange media.

It’s important to note that this is symmetrical: the other party too can start the connectivity check from IP2:port2 towards IP1:port1. To avoid a conflict, the parties assume the role of "controlling" or "controlled" agent. The controlling agent will be the one deciding which candidate pair will be used for exchanging media.

Before media can flow through a TURN server, a client must create a Permission. This is important during ICE connectivity checks.

For ICE candidates of type 'relay', the connectivity check will be performed sending Binding Requests that traverse the TURN server and reach the other party on the relay side. The Binding Requests will be carried by a Send Indication, destined to the remote candidate as peer address. The TURN server will only accept it and relay it to the destination if a Permission has been granted for that peer.

(NAT has been omitted in this diagram for simplicity)

Of course a TURN allocation must exist for the CreatePermission request to succeed, but that’s already been created during the candidate gathering phase.

If the candidate pair selected for exchanging media will be one with a local 'relay' candidate, then typically the WebRTC client binds a TURN Channel to the other party, and starts exchanging media using ChannelData messages, instead of Send/Data Indications.

There are more details that can be discussed, like managing timeouts, role conflicts, ICE Lite, etc, which we will address in other articles. One additional aspect is important here: we mentioned three types of candidates, host, server reflexive and relayed, but there’s a fourth one, "peer reflexive".

Peer reflexive candidates are not provided directly by a party during candidate exchange, but are instead discovered dynamically during the connectivity checks. Getting back to the previous example with a candidate pair {IP1:port1, IP2:port2}, depending on NAT conditions, the response to the Bind Request from IP1:port1 can be received by another source, IP3:port3.

The WebRTC client can verify IP3:port3 is sending a valid response by checking the transaction ID, and if successful it will dynamically add a remote candidate of type peer reflexive. Sometimes the peer reflexive candidate is the only one suitable and will be used to exchange media.

Chrome’s chrome://webrtc-internals, Firefox’s about:webrtc and Safari's "WebRTC Logging" will show the list of candidates and the pair that was selected, so those tools are of great value when troubleshooting.

If you take a trace on the computer running the WebRTC client, you’ll be able to see the STUN Binding Requests and Responses, and CreatePermission, Send/Data Indications for connectivity checks if unencrypted TURN is used. Wireshark will filter those messages for you if you use the ‘stun’ filter, and will also be able to interpret the Binding Request/Response carried inside Send/Data Indications (and also the RTP streams, but that’s for another article).

If you're interested about troubleshooting TURN sessions, take a look at this other article, "Troubleshooting TURN".

Troubleshooting TURN

2022-05-16T17:29:00.002+02:00

WebRTC applications use the ICE negotiation to discovery the best way to communicate with a remote party. It dynamically finds a pair of candidates (IP address, port and transport, also known as “transport address”) suitable for exchanging media and data.

The most important aspect of this is “dynamically”: a local and a remote transport address are found based on the network conditions at the time of establishing a session. For example, a WebRTC client that normally uses a server reflexive transport address to communicate with an SFU. when running inside the home office, may use a relay transport address over TCP when running inside an office network which limits remote UDP targets. The same configuration (defined as “iceServers” when creating an RTCPeerConnection will work in both cases, producing different outcomes.

This means that a certain portion of WebRTC sessions happen over TURN, i.e. they are relayed through a TURN service, when the choice is left to the client. ‘host’, ‘server reflexive’ and ‘relay’ candidates are left to compete with each other, and the best will win, with the caveat that ‘host’ candidates have the highest priority, and ‘relay’ the lowest. This prioritization originates from the logical assumption that a relayed connection may be less performant than a direct one.

There are cases though when using a TURN service is not optional, but mandatory; an RTCPeerConfiguration setting, ‘iceTransportPolicy’ allows this.

In any case, when TURN is used, it’s important to be able to troubleshoot the session establishment, and this article aims to provide some important guidelines.

These are the key points:

Acquiring the TURN settings
Confirming the reachability of the TURN server
Creating a relay allocation on the TURN server
Setting permissions for using the created allocations
Exchanging ICE connectivity checks over TURN
Exchanging media and/or data over TURN

Acquiring the TURN settings

While STUN servers are typically used without the need for authentication, it’s unlikely that a TURN service can. The resources involved in a TURN service are expensive, in particular in the case of highly scalable and distributed systems, and for this reason are only allowed for authenticated customers.

The required TURN settings are:

A URL (in the form ‘turn:<FQDN or IP address>:port)
An username
A password (called ‘credential’)

These are provided inside the ‘iceServers’ configuration structure passed to the RTCPeerConnection at the moment of creation.

Troubleshooting points

It’s important to verify that the TURN settings are correctly configured; in Chrome, open Developer Tools and check in the JavaScript code that the ‘iceServers’ structure contains valid values.

Check also the ‘iceTransportPolicy’ (which default value is ‘all’).

Confirming the reachability of the TURN server

When the ICE candidates gathering phase begins, the ICE client verifies that the TURN URL defines a reachable service by sending a STUN Binding Request towards the IP and port resolved from the ‘iceServer’ settings.

This request originates from the IP address and port that will be used to access the TURN service, and so it will check that it’s suitable for it.

If the STUN Binding Request is received by the TURN server, then it will respond with a STUN Binding Success, carrying an attribute (XOR-MAPPED-ADDRESS) that tells what source IP and port was seen by the server.

If the STUN Binding Success response is received by the client, then there’s proof that the TURN server is reachable.

For example:

Now it’s possible to negotiate a relay allocation.

Troubleshooting points

In the host running the WebRTC client, take a network trace and verify that the STUN Binding Request is addressed to the expected destination (in particular if the TURN URL required a DNS resolution and so there are multiple IP addresses that could be used).

Verify in the trace that the STUN Binding Success Response is received.

Creating a relay allocation on the TURN server

This is the key element: the client asks the TURN server to become a relay on its behalf.

Here the TURN protocol is used, and the client issues an Allocate Request towards the TURN server. This request must be authenticated, for the reasons discussed earlier, and so it’s challenged with a 401 Unauthenticated response, carrying a realm and a nonce.

The client will use the provided credentials (username and credential), together with the given realm and nonce, to compute a MESSAGE-INTEGRITY attribute and send again the Allocate Request with this attribute.

If the credentials are correct (and also the user is allowed to access the service), then the TURN service will reserve a transport address for that allocation: this is the relay transport address. An Allocate Success Response is transmitted to the client, with a XOR-RELAYED-ADDRESS attribute.

At this point the client has gained a ‘relay’ candidate and transmits it to the remote party through the signalling system in use (this is service-specific and not standardized).

Here’s an example of a successful allocation:

Note that a client may create more than one allocation for the same session; each one will be identified by a different source port, so it will be easily identifiable. You can filter them out with something like ‘stun and udp.port==PORT`, where PORT is the client source port for a transaction you’re interested in.

Troubleshooting points

In the host running the WebRTC client, take a network trace and confirm that there’s an Allocate Success Response.

Wrong credentials

In case of wrong credentials, instead of an Allocate Success Response you’ll see another 401 Unauthenticated response. In this case you must check that the credentials are correct, and the user is authorized to access the service.

Other errors for Allocate Request

Any other error for the Allocate Request will have a detailed error code (in a similar fashion as HTTP or SIP have), so take a note on that and search for its root cause.

Setting permissions for using the created allocations

For security reasons, before media or data is exchanged through the relay, the client must set specific permissions for the remote party.

Once the client has a valid relay allocation, every time it receives an ICE candidate from the remote it must set a permission for the remote IP address.

This is accomplished with a TURN CreatePermission Request. The allocation the permission refers to is implicit from the client source IP address and port. The TURN server will respond with a CreatePermission Success if the request is accepted; note that often a client receives ICE candidates with private or reserved IP addresses: in that case the TURN server will most probably reject the request with a 403 Forbidden response.

Example:

Troubleshooting points

In the host running the WebRTC client, take a network trace and confirm that there’s a CreatePermission Success for at least one of the remote candidates.

If no CreatePermission requests are sent, or none of them is successfully accepted, then no relaying will be possible.

Exchanging ICE connectivity checks over TURN

Once the TURN server is reached, a relay allocation reserved and a permission created, there are the conditions for exchanging ICE connectivity checks over TURN.

These are performed by sending STUN Binding Requests with short term credentials; the peculiarity with TURN is that these Binding Requests are encapsulated inside a TURN Send Indication, addressed to the remote peer.

Wireshark will nicely solve this encapsulation for you, and instead of showing a Send Indication will show you its content, the Binding Request.

The TURN server will relay the Binding Request to the remote peer, performing the relay for the first time. The expected outcome is that the remote entity will respond with a Binding Success, which the TURN server will encapsulate inside a Data Indication and deliver to the client.

If that happens, then the client has learned that the remote candidate is indeed reachable via TURN and that’s a suitable candidate pair for exchanging media and data.

Troubleshooting points

In the host running the WebRTC client, take a network trace and confirm that there are Binding Requests carried over TURN that receive a Binding Success.

If Binding Success responses are not received, then something is preventing it and the best way to investigate is to take network traces on the TURN server host, if possible. Those traces will tell you whether the Binding Requests are correctly leaving the TURN server towards the remote party and whether the Binding Success responses are being received or not.

It’s possible that the remote endpoint is simply unreachable from the TURN service, and in this case the ICE candidates pair will be marked as unusable.

Exchanging media and/or data over TURN

The last fundamental step is the actual exchange of packets through the relay. The typical type of packets is RTP.

Once the connectivity checks will be successful, if the client has elected the relay candidate as the one to be used, then RTP can start flowing. You’ll be able to see the RTP packets flowing in both directions, typically with video and audio multiplexed.

There are two ways for transmitting data:

Indications
Channels

A Send Indication carries the data (RTP) and destination from the client to the TURN server. The TURN server, granted the allocation exists and the permission allows it, will extract the data and send it to the destination from the allocated relay transport address.

When the data arrives from the remote peer to the relay transport address, then the TURN server, after performing the above checks, will encapsulate the data inside a Data Indication and send it to the client.

There is a more efficient way though to exchange data: the client can define a Channel (through the ChannelBind request), which associates a channel ID to a remote party. From that moment both the client and the TURN server can exchange data via ChannelData messages carrying just the channel ID and data, omitting the remote transport address. This reduces the network and computing overhead and it is typically chosen against the use of Indications.

Troubleshooting points

In the host running the WebRTC client, take a network trace and confirm that data is being sent from the client with Send Indications or ChannelData messages, and to the client with Data Indications and ChannelData messages.

In case of monodirectional media, it’s advisable to take network traces on the TURN server host to clarify whether the media is being exchanged or not on the relay side with the remote peer.

Encrypted TURN

It’s possible to use TURN over TLS, with all the data exchanged encrypted. In this case using Wireshark as described won’t allow you to see the details of the requests and responses, and troubleshooting is harder.

One possible approach is to first of all ensure that all the operations described previously happen correctly when using unencrypted TURN (over UDP or TCP). It’s very likely that the TURN service you are using is accessible over unencrypted UDP (default behavior): before moving to TLS ensure UDP works fine.

Wireshark will show you anyway the TLS connections established with the server, so that will confirm whether the connection was successful, the TLS session established, and some application data exchanged.

Useful tools

Wireshark

Wireshark is available for a variety of platforms; it’s a fundamental tool to understand what’s happening between the local WebRTC client and the remote server.

It comes with filters that detect the type of packets. You can use `stun` to filter out STUN and TURN packets, and even select specific TURN transactions, like `stun.type.method==0x0003` to show Allocate Request and Responses.

Saving a trace into a pcap file and making it available to others helps enormously the ability to troubleshoot.

Wireshark can be used for both capturing and just displaying captures.

There are cases where the dissectors, i.e. the interpreters of the packets, don’t recognize a TURN transaction. For example this happens when they happen over a non-default port (3478 for UDP and TCP, 5349 for TLS). To “help” Wireshark, right click on a packet, select “Decode As…” and set ‘STUN’ as protocol: it will correctly interpret all the packets using that non-default port.

The same applies for RTP: when signalling is not available to Wireshark, then UDP packets containing RTP may not be correctly interpreted. Use the same “Decode As…” method.

tcpdump

On the server side, any tool for packet capture would do, with tcpdump being a common solution.

Save the trace into a pcap file with the `-w` option, e.g. `tcpdump -n -v -w trace_1.pcap`, copy it to your machine and use Wireshark to display the packets.

WebRTC samples, Trickle ICE

This open source tool allows you to verify the browser can correctly gather `relay` candidates with the given TURN server details (URL, username, credential).

Before troubleshooting a client implementation, ensure that this tool can correctly access the TURN resources you’re referring to.

turnutils_uclient

The popular open source implementation of a TURN server, coturn, comes with a tool that simulates a client. A plethora of options are available, allowing you to test specific aspects of the TURN operations, e.g. using Send Indications or using Channels, etc.

Use `turnutils_uclient` to ensure the TURN service you want to use is accessible correctly with the given TURN settings, You’ll also get information about the round trip time and jitter.

Chrome webrtc-internals

When using Chrome, the best way to understand what’s happening is to open a tab on chrome://webrtc-internals/. It will show you all the information related to each RTCPeerConnection being managed by the browser at that point, including the list of ICE candidates, the details of the TURN server being used (except the credential for obvious reasons), including the iceTransportPolicy (‘all’ or ‘relay’), the chosen ICE candidates pair, statistics on media transfer, etc.

Search for `relay` candidates and verify the client is able to retrieve them from the TURN service, and whether they are selected as the candidate pair or not.

Conclusions

This article should provide a good checklist for troubleshooting the connection to a TURN service. There is much more to say, in particular for what concerns browsers different than Chrome and server-side investigations: I plan to write about it in the future.

Differences between running and cycling

2021-03-11T11:01:00.001+01:00

I'm a passionate runner, and always considered cycling as something fun, e.g. mountain-biking, but difficult to practice regularly. There's a lot of overhead in cycling, like the preparation, bike maintenance, dealing with city traffic, etc.

Anyway about eight months ago I bought a road bike and felt in love with it. Soon after that I discovered Zwift and that gave an additional dimension to the sport: practice whenever you want from home, with accurate power measurements and a way to socialise with distant people. That was a game changer.

In five months I cycled 1600 virtual Km and climbed almost 17 virtual Km. Meanwhile my running performance, instead of degrading, improved, and that surprised me.

Anyway what I wanted to write about is a great article I read, "Physiological Differences Between Cycling and Running". It's a review of articles published in that area. Some conclusions are very interesting.

In general it seems sports medicine is still inconclusive for many aspects, and coaches may still have an advantage by following empirical/heuristic approaches in comparison with research-driven indications.

But more specifically, some notes from the conclusions:

- For the same person, VO2max depends on the speciality (i.e. runners achieve higher values on treadmill than cycle ergometer)

- There seems to be more physiological transfer from running to cycling than the other way around

- Pedalling cadence impacts the metabolic response during cycling, but also during a following run (at least in the short term)

- The Lactate Threshold is lower for athletes when not practicing their speciality, i.e. the Lactate Threshold depends on the training method

- Both female and male are impacted in the same way when comparing VO2max for running and cycling

- Triathletes have similar max Heart Rate when running and cycling, again pointing to the importance of the actual speciality used in training

- The position when cycling makes it harder to breathe

and probably other important elements that I wasn't able to fully grasp.

Notes on STUN protocol

2021-03-10T11:55:00.002+01:00

Since I needed to see a few details in the handling of attributes in STUN responses, I thought of going through the whole STUN protocol RFC again and take notes on the most important parts.

I put my notes in some slides, and I'm sharing then in Slideshare in case somebody else may find them useful too:

STUN protocol from Giacomo Vacca

Extracting RTP streams from network captures

2021-02-19T09:45:00.001+01:00

I needed an efficient way to programmatically extract RTP streams from a network capture.

In addition I wanted to:

save each stream into a separate pcap file.
extract SRTP-negotiated keys if present and available in the trace, associating them to the related RTP (or SRTP if the negotiation succeeded) stream.

Some caveats:

In normal conditions the negotiation of SRTP sessions happens via a secure transport, typically SIP over TLS, so the exchanged crypto information may not be available from a simple network capture.
There are ways to extract RTP streams using Wireshark or tcpdump; it’s not necessary to do it programmatically.

All this said I wrote a small tool (https://github.com/giavac/pcap_tool) that parses a network capture and tries to interpret each packet as either RTP/SRTP or SIP, and does two main things:

save each detected RTP/SRTP stream into a dedicated pcap file, which name contains the related SSRC.
print a summary of the crypto information exchanged, if available.

With those two elements, it’s then possible to decrypt an SRTP stream, depending on the availability of the exchanged crypto information, and also decode it into audio, depending on the codec.

Decryption and decoding is not part of my tool, but can be achieved easily with other tools, like pjsip’s pcaputil.

I might integrate that part into pcap_tool in the future. Again not because it’s strictly necessary, but to start getting more control on the parsing and manipulation. This may reveal to be useful in the future.

pcap_tool is available here for anybody interested in using it and may perhaps wish to change or extend some parts.

You can just clone it and build it as described in the README.

An example output:

./pcap_tool -d ../../trace_20210218_1.pcap

[…]

Extracted 1092 RTP frames

Detected RTP Stream: 0x7a2179fa Source port:22248 - Destination port:4000 - Packets: 544 (./stream-0x7a2179fa.pcap)

Detected RTP Stream: 0x772dc5d7 Source port:4000 - Destination port:22248 - Packets: 548 (./stream-0x772dc5d7.pcap)

source port: 22248 - tag: 3 - suite: AES_CM_128_HMAC_SHA1_80 - key: /1TI6DJWHk7fBJY1yBp7L51uEz1JJ2n6CcQAAsJM

-----

source port: 4000 - tag: 4 - suite: AES_CM_128_HMAC_SHA1_32 - key: mPytX24bRmyNgMaqQSxP8dMMqdkkmQeHgC2Ttb3v

source port: 4000 - tag: 3 - suite: AES_CM_128_HMAC_SHA1_80 - key: J1YS1owJDKAFdq5cRF+JtektYDf6IiowCAeijeal

source port: 4000 - tag: 2 - suite: AES_256_CM_HMAC_SHA1_32 - key: 5A9R8O8MCzbuGvJ08WWNJcNHsPaEcEp1ZDp5DunknZ+bZ2JQaVpZ2qmqraTmgQ==

source port: 4000 - tag: 1 - suite: AES_256_CM_HMAC_SHA1_80 - key: ZcZn1IY++2xsSIk/U1GsHSGp+OI/BYIocv/40ldJB28bcNeMmYzs4z4ozrNQ5Q==

-----

That network capture contained 2 SRTP streams, which have been saved separately into stream-0x7a2179fa.pcap and stream-0x772dc5d7.pcap files respectively.

For the negotiation it’s visible what the sender from port 22248 (owner of the 0x7a2179fa stream) used as crypto information, and looking at the same tag (3 in this case) it’s possible to see what crypto information was used by the sender of 0x772dc5d7 stream from port 4000.

With this it’s possible to decrypt (and decode since G.711 was used) with pjsip’s pcaputil with something like:

pcaputil -c AES_CM_128_HMAC_SHA1_80 -k /1TI6DJWHk7fBJY1yBp7L51uEz1JJ2n6CcQAAsJM stream-0x7a2179fa.pcap stream-0x7a2179fa.wav

and have the audio from that stream into a WAV file.

How to build pcaputil (in fact all pjsip’s applications) is widely documented but I also described it in the appendix of https://www.giacomovacca.com/2020/11/testing-sip-platforms-and-pjsip.html

The call in the example was generated in fact with pjsua.

SIP - Connection reuse vs Persistent connection

2020-12-04T09:52:00.000+01:00

It goes without saying that SIP solutions are impacted by NAT. So much that some scenarios required integration to RFC 3261, e.g. with RFC 3581, which defined the 'rport' attribute to be added in the Via header (integrating the 'received' attribute): with that information, responses could be routed to the source port of the related request, and not on the advertised port in the original Via.

That was called Symmetric Response, and applied to connection-less transports (UDP), while, as mentioned in RFC 6314, it's not necessary when using reliable transports (TCP in most cases): SIP responses can be sent back on the same connection on which the request arrived.

Also from RFC 3261, chapter 18, Transport layer, client behaviour:

"For reliable transports, the response is normally sent on the connection on which the request was received."

But the client needs to be prepared to receive the response on a new connection:

"[...] the transport layer MUST also be prepared to receive an incoming connection on the source IP address from which the request was sent and port number in the "sent-by" field."

That obviously would require the ability to create such connection from the server to the client.

Anyway when a reliable connection between two SIP entities is up, after a transaction is already concluded, there are two interesting opportunities:

- Use that same connection for more requests from the client

- Use that same connection for more requests from the server

where for "client" I refer to the entity that created the connection and sent the initial request, and "server" is the entity that accepted the connection and delivered the response(s).

The first case is mentioned in the same chapter on Transport layer:

"If a request is destined to an IP address, port, and transport to which an existing connection is open, it is RECOMMENDED that this connection be used to send the request, but another connection MAY be opened and used."

This is what's referred to as "Persistent connection", as mentioned in RFC 5923:

"The SIP protocol includes the notion of a persistent connection
   [...], which is a mechanisms to insure that
   responses to a request reuse the existing connection that is
   typically still available, as well as reusing the existing
   connections for other requests sent by the originator of the
   connection."

The second case (using the same connection from future requests from the server) is instead the subject of RFC 5923, and it is defined as "Connection reuse".

Once the connection is up, it seems a good opportunistic approach to reuse it, but an important limitation is mandated:

"Unlike TCP, TLS connections can be reused to send requests in the backwards direction since each end can be authenticated when the connection is initially set up."

In other words, only TLS connections formed by exchanging certificates can be reused, because the identities have been mutually verified.

The way the client can tell the server that connection reuse is desired is with a new parameter to be added in the Via header: 'alias'.

In general, RFC 5923 at chapter 5 clarifies:

"The act of reusing a connection needs
   the desired property that requests get delivered in the backwards
   direction only if they would have been delivered to the same
   destination had connection reuse not been employed."

Last important bit, not to be left implicit: persistent connections don't imply connection reuse, as RFC 5923 clarifies:

"[...] Persistent connections do not
imply connection reuse."

So this post is basically sharing my own notes on this topic, which maybe somebody else (including me) can find useful in the future.

Dissecting traces with DTMF tones

2020-11-24T09:54:00.003+01:00

I'm sure I belong to the large group of people who love to analyse network traces with tools like Wireshark. Being able to see the details of a packet or datagram down to the level of the bits is not only extremely useful, but also fascinating.

Time ago I wrote a dissector for Wireshark, using the Lua interface, and that was fun (I see it's still available here). The official recommendation is to use Lua only for prototyping and testing, but when performances are not key and there isn't the intent to add the dissector to the official distribution, it's fast and effective.

In order to parse network traces with audio and extract it into the payload first, and then decode it into a WAV file, C is a viable solution. I wrote about a program that does that here and since it attracted some attention and feedback I wrote an updated version later.

More recently I wanted to identify programmatically the presence (and value) of DTMF tones - as RTP Events, RFC 2833 - in network traces. This time rather than using C, I wanted to integrate it with python, and scapy seemed a good choice.

scapy is quite complete, but interestingly it doesn't have a parser for the RTP Event extension. So I thought of mapping the raw content in the RTP payload to a structure, with the help of C types.

This it the core of the program:

def process_pcap(file_name, sut_ip):

for (pkt_data, pkt_metadata,) in RawPcapReader(file_name):

ether_pkt = Ether(pkt_data)

# A little housekeeping to filter IPv4 UDP packets goes here
...

if ether_pkt.haslayer(UDP):
udp_pkt = ether_pkt[UDP]

# Get the raw UDP packet into an RTP structure
rtp_pkt = RTP(udp_pkt["Raw"].load)

ptype = rtp_pkt.payload_type

# Assume payload type 96 or 101 are used for RTP events
if (ptype == 96 or ptype == 101):
rtpevent_content = rtp_pkt.payload

# map the payload into an RTPEvent object
rtpevent_struct = RTPEvent.from_buffer_copy(rtpevent_content.load)

The RTPEvent class looks like this:

class RTPEvent(ctypes.BigEndianStructure):
_fields_ = [
('event_id', ctypes.c_uint8),
('end_of_event', ctypes.c_uint8, 1),
('reserved', ctypes.c_uint8, 1),
('volume', ctypes.c_uint8, 6),
('duration', ctypes.c_uint16)
]

so once mapped, the rtpevent_struct object will have its DTMF-specific details, in particular with the digit contained in rtpevent_struct.event_id, and the indication whether it's the marker of end of the event in the end_of_event bit.

All the other information (source/destination IP address and port, timestamp, SSRC) is obviously available in the UDP and RTP portion, so it's easy to adapt to your needs and filter out the DTMF tones for the streams you're interesting in.

Kubernetes role-based authorisation for controller applications

2020-11-23T10:07:00.002+01:00

There are many scenarios where an application running inside a Kubernetes environment may need to interact with its API.

For example, an application running inside a Pod may need to retrieve real time information about the availability of other applications' endpoints.

This may be a form of service discovery that integrates or extend the native Kubernetes internal service discovery. In most cases, DNS records are associated to a Service and provide the list of active Endpoints for that Service, with a proper TTL. There are situations though where those DNS records are not available, an application is not able to use them directly, or what's needed is more than the private IP addresses associated with the Endpoints.

If interacting with the Kubernetes API from inside an application is needed, then there are two main areas to consider: Authentication and Authorisation.

Every Pod has a Service Account associated to it, and applications running inside that Pod can use that Service Account. Without a specific configuration the Service Account will default to a generic namespace and generic authorisation.

It's possible instead to define a more specific Service Account, with fine grained permissions to access the API. This Service Account can then be linked to a Pod with a Role-based approach.

A reference: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/

You can get a list of available Service Accounts with an intuitive:

# kubectl get serviceaccounts

which is likely to show you a single 'default' service.

Service Accounts may have a namespace scope. You can check what Service Accounts are associated to a pod with a command like:

# kubectl -n NAMESPACE get pods/PODNAME -o yaml | grep serviceAccountName

Service Account authentication can use the token reachable from inside a Pod, in the /var/run/secrets/kubernetes.io/serviceaccount directory, under the namespace-specific directory, e.g. /var/run/secrets/kubernetes.io/serviceaccount/NAMESPACE/token.

Those tokens are also visible as Mounts in the related containers.

For internal requests, Kubernetes provides a local default HTTPS endpoint at https://kubernetes.default.svc - so a way to discover the details of a Service Account for a given namespace could be:

#!/bin/bash

# Point to the internal API server hostname
APISERVER=https://kubernetes.default.svc

# Path to ServiceAccount token
SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount

# Read this Pod's namespace
NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)

# Read the ServiceAccount bearer token
TOKEN=$(cat ${SERVICEACCOUNT}/token)

# Reference the internal certificate authority (CA)
CACERT=${SERVICEACCOUNT}/ca.crt

# Explore the API with TOKEN
curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api/v1/namespaces/${NAMESPACE}/pods

Accessing the internal API programmatically

With common python libraries as https://github.com/kubernetes-client/python (available on debian with the 'python3-kubernetes' package) it's extremely easy to automate the invocation of the internal APIs.

Most of the examples assume you're running your program as a user, and refer to the local kube config file, but when running inside a container it's possible to inherit the Service Account token associated with the hosting Pod.

For this, instead of using

config.load_kube_config()

use

config.load_incluster_config()

Then you can instantiate your API client object:

v1 = client.CoreV1Api()

and either do a single request, like getting a list of all pods inside any namespace:

ret = v1.list_pod_for_all_namespaces(watch=False)

or a list of pods belonging to a namespace and matching a specific application label, like:

ret = v1.list_namespaced_pod(namespace, label_selector=app_name, watch=False)

or you can "watch" some resources, which basically means subscribing to such resource updates and getting a notification at each change:

w = watch.Watch()
for event in w.stream(v1.list_namespace, _request_timeout=60):
...

Each event can be ADDED, DELETED and MODIFIED, and carries a rich set of information associated to the current status of the resource.

Defining your specific ServiceAccount

Before getting to that point, though, you need to define your non-default Service Account and assign specific permissions to it. To achieve this, the role-based approach can be used.

First of all define a ServiceAccount resource:

apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/component: mycomponent
name: mycomponent-serviceaccount

Then define a role associated to this resource:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: myrole
namespace: mynamespace
labels:
[...]
annotations:
[...]
rules:
[...]
- apiGroups:
- ""
resources:
- pods
- endpoints
verbs: ["get", "list", "watch"]

This example adds the permission to get, list, or watch the list of pods and endpoints in the given namespace.

Create a role binding:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: myrolebinding
namespace: mynamespace
labels:
[...]
annotations:
[...]
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: myrole
subjects:
[...]
- kind: ServiceAccount
name: mycomponent-serviceaccount
namespace: mynamespace

Whenever a Role cannot be restricted to a namespace, for example if it needs to access cluster-wide resources like Nodes, then the ClusterRole resource is available.

References and other sources

"Access clusters using the Kubernetes API", https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-api/

An interesting post about asynchronous watches with Python: https://medium.com/@sebgoa/kubernets-async-watches-b8fa8a7ebfd4

"Kubernetes Patterns", an ebook by Redhat: https://www.redhat.com/cms/managed-files/cm-oreilly-kubernetes-patterns-ebook-f19824-201910-en.pdf

Testing SIP platforms and pjsip

2020-11-20T13:27:00.014+01:00

There are various levels of testing, from unit to component, from integration to end-to-end, not to mention performance testing and fuzzing.

When developing or maintaining Real Time Communications (RTC or VoIP) systems, all these levels (with the exclusion maybe of unit testing) are made easier by applications explicitly designed for this, like sipp.

sipp has a deep focus on performance testing, or using a simpler term, load testing. Some of its features allow to fine tune properties like call rate, call duration, simulate packet loss, ramp up traffic, etc. In practical terms though once you have the flexibility to generate SIP signalling to negotiate sessions and RTP streams, you can use sipp for functional testing too.

sipp can act as an entity generating a call, or receiving a call, which makes it suitable to surround the system under test and simulate its interactions with the real world.

What sipp does can be generalised: we want to be able to simulate the real world that surrounds (or will surround) our system in Production. From this point of view sipp is not the only answer, and projects often use other tools, or a combination of other tools.

One simple and effective approach is re-using RTC applications and build the testing tool around them. When a system is built around an application, it's likely that the people working on it are familiar enough with the application to re-use it to mock the external world. This is often achieved with Asterisk or FreeSWITCH. They both expose an API for originating calls, and surely can play the role of called party (or "absorbers", or "parrots" depending on their main scope and terminology).

Kamailio also can be used to generate calls, even though its core focus on signalling makes it slightly more complex to use in generic cases.

Unless behavioural changes are put in place, such solutions imply compromising on the SIP stacks in use. Asterisk or FreeSWITCH won't make it too easy to generate an INVITE with a wrongly formatted SIP header, for example, while sipp is much more flexible, ad the SIP messages can be mocked down to the single character. What typically happens is that sipp is used to generate or receive calls when specific syntax requirements for the signalling are needed, while Asterisk and FreeSWITCH can be used in more permissive cases, where what's important is a generic session establishment.

When dealing with media (typically, RTP streams) is necessary, then sipp provides at least two methods: re-playing an RTP stream from a trace (pcap file), or encoding a WAV file into a stream. Recently sipp added the ability to play RTP Events separately (DTMF tones as in RFC 2833 - I think the first patch with this functionality was this). sipp is not able to transcode or generate non-PCM streams, but still it can play a non-PCM stream with just some limitations, which covers most of typical cases.

Less generic scenarios where RTC applications like Asterisk and FreeSWITCH can be useful are the ones requiring SRTP (encrypted RTP). Even though sipp can be used to negotiate SRTP, by adapting the SDP portion of the offer/answer, it doesn't provide a solution to generate SRTP streams.

In this case a very useful item to add to your toolbox is pjsip, which is a SIP stack library (used also by Asterisk and chan_pjsip being the current recommended SIP channel, as opposed to the older chan_sip) that exposes an API and also a command-line option (pjsua). pjsua can be used directly, with either command line arguments or a configuration file, or it's possible to use pjsip library to write programs with languages like python: this makes it very flexible and helps its integration in existing and new testing systems.

With pjsip, it's possible to generate calls that play audio and DTMF tones, in a similar way than sipp, but also encrypt RTP and establish SRTP streams.

pjsua

The easiest approach is to build the pjsip project and use the pjsua binary (you can see a procedure in the Appendix).

pjsua accepts command-line arguments, but can receive arguments from a configuration file, which makes it easier to read. For example you could just

# pjsua --config-file pjsua.cfg

where pjsua.cfg contains just the caller and callee:

sip:bob@example.com
--id=sip:alice@example.com

A more sophisticated configuration file contains instructions on codecs and encryption, e.g.

sip:bob@example.com
--id=sip:alice@example.com
--use-srtp=0
--srtp-secure=0
--realm=*
--log-level=6
--no-vad
--dis-codec GSM
--dis-codec H263
--dis-codec iLBC
--dis-codec G722
--dis-codec speex
--dis-codec pcmu
--dis-codec pcma
--dis-codec opus
--add-codec pcma
--null-audio
--auto-play
--play-file /some_audio.wav

Since I mentioned SRTP as a possible key element for using pjsip, let's look into the related options:

--use-srtp=0

--srtp-secure=0

'use-srtp' can be 0, 1 or 2, and means "disabled", "optional" and "mandatory", respectively.

With "optional" pjsua offers both plain and encrypted RTP at the same time, and the callee entity can decide. With "mandatory" it will only offer SRTP, and the callee will have to either accept or reject.

'srtp-secure' refers to the use of TLS, and can also be 0, 1 or 2, meaning "not required", use "tls", or use "sips" respectively. Needless to say, in normal scenarios you want to protect the SRTP crypto information carried in the SDP, so you want to encrypt signalling too. SIP over TLS is the typical solution. For testing purposes you may prefer making it easier to check the content of signalling, and use 'srtp-secure=0'.

'no-vad' formally should be used to disable silence detection; in practice you want this option when generating a call from a machine that doesn't have a sound card.

Similarly, 'null-audio' disables the requirement to play the audio, required when the calls are generated from a host with no sound interfaces.

'dis-codec' is used to disable a codec from the negotiation, and 'add-codec' instead selects a codec to be added to the offer. This adds flexibility, and it's also worth noting that video codecs are available too.

Using pjsip library with python

It's possible to use the pjsip library's API with high level programming languages like python. This makes test automation quite versatile, and I remember seeing this approach as early as 2012, where the project I was working on had the client applications built on top of pjsip: it was extremely valuable to simulate programmatically the clients from linux machines.

Being designed for interactive applications, pjsip comes with a nice event-based model, so in principle you need to trigger the desired actions and register callback functions that will be called at the proper moment.

A complete reference to the python library can be found here.

In general, after you import the library:

import pjsua as pj

then the library is imported in an object, the configuration objects are populated, and a call is triggered, e.g.:

lib = pj.Lib()
media_cfg = pj.MediaConfig()
media_cfg.no_vad = 0
lib.init(log_cfg = pj.LogConfig(level=3, callback=log_cb), media_cfg=media_cfg)
lib.set_null_snd_dev()

lib.set_codec_priority("GSM", 0)
lib.set_codec_priority("iLBC", 0)
lib.set_codec_priority("G722", 0)
lib.set_codec_priority("speex", 0)
lib.set_codec_priority("pcmu", 0)
lib.set_codec_priority("pcma", 1)

transport = lib.create_transport(pj.TransportType.UDP)
lib.start()
acc = lib.create_account_for_transport(transport)

call = acc.make_call(sys.argv[1], MyCallCallback(), hdr_list=custom_headers)

You can see that set_codec_priority to 0 is equivalent to the --dis-codec command line option.

MyCallCallback() is the callback function that will be invoked at each change of call state, with an event object passed as argument. You'll have something like:

class MyCallCallback(pj.CallCallback):
def __init__(self, call=None):
pj.CallCallback.__init__(self, call)

def on_state(self):
...
if self.call.info().state == pj.CallState.CONFIRMED:
# The call has been answered
# Here you can create a player to generate audio into an RTP stream, send DTMF, log information, etc
# You can even invoke other APIs to interact with more complex systems
...
def on_media_state(self):
global lib
if self.call.info().media_state == pj.MediaState.ACTIVE:
   ...
   # Media is now flowing, so you can connect it to the internal conference object
   # Connect the call to sound device
   call_slot = self.call.info().conf_slot
   lib.conf_connect(call_slot, 0)
   lib.conf_connect(0, call_slot)
   print "on_media_state - MediaState ACTIVE"

As it can be expected, exceptions can be caught and errors displayed:

except pj.Error, e:
print "Exception: " + str(e)
lib.destroy()
lib = None
sys.exit(1)

If you happen to need DTMF tones, pjsip offers the dial_dtmf() function, as part of the Call object, e.g.:

self.call.dial_dtmf("0")

Just remember that these calls are asynchronous, non-blocking: you need to add explicitly a delay to separate the beginning of a tone from other actions.

pjsip will generate proper RTP Event packets of the given duration, inside the existing RTP stream (and so they will have the same SSRC and proper timestamp reference).

I'll write about analysing pcap traces to extract information on RTP events in a separate article.

Wrap up

This article is somehow what I would have wanted to read on the topic some time ago, but I had to infer from various sources and after various experiments. I hope it will be useful to some of the readers.

Appendix - pjsua build and install

To build pjsua on debian you can do something like:

apt install python-dev gcc make gcc binutils build-essential libasound2-dev wget
wget https://github.com/pjsip/pjproject/archive/2.10.tar.gz
tar -xvf 2.10.tar.gz
cd pjproject-2.10
export CFLAGS="$CFLAGS -fPIC"
./configure && make dep && make

The binary will be available at ./pjsip-apps/bin/pjsua-x86_64-unknown-linux-gnu, which of course you can link to something easier to use, or copy to a directory in the PATH.

kamailio-tests, a testing framework for Kamailio developers

2019-11-22T09:26:00.000+01:00

Kamailio is an open source VoIP server, widely used in the VoIP industry for its performance and feature set.

kamailio-tests is a project that aims to provide a level of automated testing for developers.

The main idea is that simple things like loading a module or calling a core or module function can be tested without building an entire infrastructure around Kamailio.

Also, we want to be able to perform the tests against various versions of Kamailio, and on various OS distributions. In this way a backward incompatible function or a regression can be discovered by a developer even though their system uses one single version and OS combination.

This includes also third party libraries, e.g. you may think of testing the HTTP clients with different curl libraries, while Kamailio and OS stay at the same version.

kamailio-test doesn't perform tests at function level (but it requires the application to run) or "integration testing" (because in order to remain generic and compact it doesn't cover complex architectures), but aims to test Kamailio as an isolated application.

In order to achieve this, kamailio-tests relies on Docker to allow developers to experiment and perform their tests in perfect isolation. For example, once you've built the base Docker images you can add and modify tests while travelling on a plane.

Most VoIP platforms have extensive and complex test scenarios that involve setting up accounts, other components, specific logging systems, etc. We can't provide a compact, generic solution that would replace that, but instead we can focus on self-contained (pun intended), simple tests that give us key feedback on issues (and also the opportunity to debug them in that same infrastructure).

The internal structure of the project and how to use it is described in its README, but here I'd like to reiterate how tests are run and how new tests can be added.

Of course you'll need Docker, and it can be a Desktop or server version.

Currently CentOS 7, Debian 9 and Debian 10 are supported via dedicated Dockerfiles.

What version of Kamailio to test can be chosen by simply checking out the desired git branch or tag from Kamailio github repo.

The installation process is described in the README, but let's see comment it here too:

First create a directory where to store the resources and go to it, e.g.:

mkdir kamailio-testing
cd kamailio-testing

Clone the kamailio-tests git repository inside that work directory:

git clone https://github.com/kamailio/kamailio-tests

Clone the kamailio git repository inside that work directory:

git clone https://github.com/kamailio/kamailio

Here you can git checkout the branch or tag you want to test, e.g.

cd kamailio
git checkout 5.3.1
cd ../

Copy the desired Dockerfile from kamailio-tests in the current folder, based on the OS distribution you want, e.g. for Debian Stretch:

cp kamailio-tests/docker/Dockerfile.debian9 Dockerfile

Build the Docker image. Give it the name you want: you'll just have to refer to it when launching the tests. e.g.:

docker build -t kamailio-tests-deb9x .

Run the tests. If you're happy with the default behaviour you can just:

docker run kamailio-tests-deb9x

and all not explicitly excluded tests will be executed.

Some tests may be excluded from execution by listing them in the etc/excludeunits.txt.DISTRIBUTION file, e.g. etc/excludeunits.txt.centos7.

How to add a test?

Create a new folder in units/, with a name starting with 't', followed by a string indicating the module, like 'geoip', and four digits, e.g. 'tgeoip0001'.

Inside this folder the minimum amount of data is a shell script that the test framework will execute and a kamailio.cfg file. See for example this test for geoip module.

The script must start with:

#!/bin/bash
. ../../etc/config
. ../../libs/utils

to prepare configuration and utility commands.

Then you can launch Kamailio with the configuration required, e.g.:

${KAMBIN} -P ${KAMPID} -w ${KAMRUN} -Y ${KAMRUN} -f ./kamailio-tgeoip0001.cfg -a no -ddd -E 2>&1 | tee /tmp/kamailio-tgeoip0001.log &

You can see that logs are being sent to a local file; this is typically how the test result can be verified. Of course you can use a different approach.

Inside the container you have sipsak available, so you can launch a test call to trigger call processing in Kamailio with something like:

sipsak -s sip:alice@127.0.0.1

To help triggering the tests also tools like sipp and pjsua can be considered, but are not currently installed in the docker images.

After waiting for the necessary time, you can just kill Kamailio with

kill_pidfile ${KAMPID}

and check the test result by parsing the log file, e.g.:

grep "ip address is registered in US" /tmp/kamailio-tgeoip0001.log
ret=$?
if [ ! "$ret" -eq 0 ] ; then
exit 1
fi

'exit 1' will make the test fail.

End the test script with

exit 0

to make it pass.

Adding a README.md inside that same folder is highly recommended.

You can add one or more tests and run them before committing the changes.

Once happy please fork the kamailio-tests repo with your changes and raise a pull request.

Thanks for reading; feedback is welcome.

My notes on Kamailio Developer Meeting - November 2019

2019-11-17T19:42:00.001+01:00

The Kamailio Developers Meeting is a two-day event held in Dusseldorf, currently at the second edition.

As described in https://www.kamailio.org/w/developers-meeting/,

"The purpose of the event is to support the interaction between developers and to offer a great environment to work together on relevant topics related to the Kamailio project. It is intended for participants that want to write code for Kamailio and its tools or improve the documentation. There will be no formal presentations, only open discussions, coding or documentation writing sessions."

The sipgate offices offered a very welcoming environment. Noticeable to have a kitchen with chefs for breakfast and lunch, and a private pub, where the social event was held. I noticed the presence of art works and learned that those offices are also part of an art itinerary in Dusseldorf.

We started listing the topics that we wanted to tackle during the event, then discussed a plan to go through them.

The first important activity was the release of kamailio version 5.3.1 (minor bug fix). The release process includes running the tests in kamilio-tests repo (more on this later in a dedicated post), and an issue was found and resolved during the release.

Then it was the time to discuss RPM packages building: Sergey Safarov has made an incredible amount of work to ensure that RPM packages for kamailio are available, and now they are available at https://rpm.kamailio.org/. Work is in progress to move all the building phases to infrastructure belonging to the kamailio projects and avoiding personal accounts to access cloud services as much as possible.

A documentation review was carried out: it was reported that often the return values of module functions are not documented, and work is encouraged in that direction, including a review of documentation of function parameters. Modules may make available several pseudovariables, and not always they are documented. There's also a dedicated wiki page on pseudovariables that's useful as reference.

Daniel made a briefing on the Kemi framework, explaining how to structure the wrapper functions (two steps, one the parameters manipulation for the standard cfg file, and then encapsulate the code into a separate function. Also in this way executing a function from Kemi doesn't require executing all the typical wrapper manipulations required by cfg functions, impacting positively on performance). Developers need to ensure that when you develop a new function the body of the function is separate from the code that manipulates the parameters.

An open topic focused on the best process to handle "dialog failover". This may be necessary if a kamailio-based component disappears during the dialog lifetime, or if the architecture allows for in-dialog messages to be processed by different entities during a call, even in the typical case where record-routing applies. Currently it's possible to load dialog information on the fly from a DB, see dlg_db_load_callid(), but when an instance "takes over" the dialog information there's typically the need to manipulate the record-routing, which is complex and not exactly "standard" behaviour.

Modules like evapi and http_async_client allow for the management of asynchronous events by spawning dedicated workers which will execute portions of the routing logic when defined events happen. An open discussion is about the best method for managing asynchronous events and at the same time return execution to the main workers when an event is triggered.

Debian packaging; a PGP signed key was created, to benefit from the physical presence of various holders of PGP signatures. An open point is related to the ability to keep earlier versions of the kamailio packages every time a new version is released. This is a known behaviour that may limit the options in some environments, and I've experienced it directly. This is caused by reprepro and what seems to be the right way to go is moving from reprepro and use aptly instead. We'll update on this.

Use of TLSv1.3: the only limitation appears to be in libssl library. It's possible to allow the usage of version 1.3, but only by choosing tlsv1.2+, which admits using v1.2, and so doesn't enforce v1.3 only. When this will be allowed, kamailio administrators will just need to update the configuration and reload the tls module. Remember that only configurations in tls.cfg can be reloaded, while modparam declarations require a restart.

Finally, we know that a common pattern of usage of kamailio involves querying APIs and basing call processing on the API responses. A simple but powerful solution sees the use of http_async_client in collaboration with rtjson; see for example this article from wazo developers: https://wazo-platform.org/blog/kamailio-routing-with-rtjson-and-http-async-client We are discussing whether extending rtjson may be generally helpful for scenarios like this; in that case I'll post an update.

For me this event was a great experience, both from a friendship point of view and as a learning experience. I'm very happy about how the kamailio project is managed, decisions are taken and information is shared.

Here's my report; for obvious reasons this can only be a limited account of what happened during the event, and the information provided is based on my recollection and notes. I'm sure I'm omitting other important material, and I'll be happy to integrate with another post.

You need to slow down

2018-11-27T10:17:00.000+01:00

This blog has been historically focused on technical topics in areas related to Real Time Communications, but I'm taking the liberty to digress a little.

I've been reading recently about the dynamics of performance in running.

I'm just an amateur runner, and I always train (and race) alone, so I felt the need to make it more interesting than just reading training tables. I've been studying what is it that limits performance. "Train more", unfortunately, not only doesn't always work, but needs to take into account injuries, and overtraining in general.

One of the first concepts that struck me is that it's been proven that fatigue, and the consequential slow down, does not mean that the body is unable to continue with that effort. What's behind slowing down is a sort of protective mechanism in our nervous system, which wants to prevent the body to reach exhaustion.

Our nervous system is constantly getting feedback signals from the body, including the perception of adverse weather conditions, and computing for how longer the current effort can be sustained. Even knowing how long is left to run is a form of external feedback.

When this computation detects that the effort is too high, the nervous system takes control and doesn't fire as many muscle fibres as the athlete wants to. The athlete thinks the muscles can't continue to work at that level, but in reality they are entering a protective status. Without that, people could literally run until body exhaustion or even death.

I find this fascinating. Evolution has given us a sophisticated algorithm aimed to prevent body exhaustion by generating fatigue symptoms and consequently reduce the actual effort.

What's also fascinating is that it seems this system can be "tricked". One way of doing is by training. Training is a way of educating your nervous system that a certain effort is OK. "There's no need to shut me down, brain, I know what I'm doing. I did that thousands of times in my training sessions."

So while the body adapts to the stress of training, the nervous system too becomes more familiar with that stress, and gives up a little.

Another way of tricking that system is by providing false external feedback. It seems that if you get false information about elements like the air temperature, or even the remaining time in your training session or race, then the nervous system acts accordingly. If it believes it's not as hot as it is, it will not intervene to shut the muscle fibre activation.

Similarly, if the athlete thinks the race is close to the end, the nervous system will allow a prolonged effort. This is why elite marathoners, who clearly haven't underperformed for the first 40 km, can run the last 2 km even faster.

Of course, all these tricks have limits, and the improvements that can be tricked are in the order of small percentages. But still this shows the importance of external feedback.

This is properly explained by Steve Magness in his "Science of running" book. I then read another book from this author, "Peak performance". To be perfectly honest, I was expecting something strictly related to endurance sport, but in this second case the concept of performance was wider.

There seems though to be an analogy between the shutting down of muscle fibres from the nervous system during running, and something that happens behind a desk and is more widely known: mental burnout. From this point of view, mental burnout can be seen as a way of saying "You can't keep going at this (perceived) level of effort. You need to sleep, hydrate, rest, but you keep working. I'm going to take control and shut you down.".

As I'm making my own little experiments, in the future I'd like to write more about this, and in particular about the relationship between effort and rest.

Docker from scratch

2018-11-19T21:20:00.002+01:00

Some time ago I prepared an introductory seminar on Docker, which I called "Docker from scratch".

The audience was a local group of heterogeneous developers. As it typically happens, preparing that material was a great opportunity to understand better some of the aspects.

I then published the slides in Slideshare. I notice now that they got a decent number of views and downloads, so I'm linking those slides here as well for reference.

One big change in respect to 2016 is represented by the choice of abandoning Docker Toolbox in favour of running Docker directly on macOS. I have to say I liked the sandboxing that came with Docker Toolbox, where the docker engine ran inside VirtualBox, now only available for older versions.

On Kamailio World 2018, part II

2018-05-29T20:50:00.000+02:00

In the first part of my brain dump about this year's edition of Kamailio World I focused mainly on testing. Core developers and application designers want to be able to test the behaviour of Kamailio-based architectures with minimal effort and fast feedback.

A different dimension to testing, that I haven't mentioned in my previous post, was related to Fuzz testing. There were two presentations focused on this: Sandro Gauci's (The easiest way to understand who Sandro is: listen on port 5060 on the public Internet and wait a couple of minutes. You'll see a SIP request from a tool called sipvicious (aka friendly-scanner), a penetration testing tool Sandro wrote (and others misuse)) and Henning Westerholt, historical member of the Kamailio community.

Sandro's presentation focused around fuzzing approaches for RTC in general (slides), while Henning was more specifically focused on Kamailio.

Fuzzing is a sophisticated technique to verify the robustness of a software application, by sending input that can vary greatly from the typical or expected usage. The objective is to find weaknesses that can lead to crashes or other malfunctions, so that they can be fixed. Of course testing a server like Kamailio is even trickier than testing an application that can read from a file. It is a fascinating topic.

Kamailio proved to be very robust: Henning reported an average of about 1 message every 44 million required to make Kamailio misbehave. The video of Henning's presentation is here (by the way, Pascom have done a great work this year too, providing a flawless video streaming and recording. It feels like we are a little spoiled, because we give it for granted and barely notice all the work behind it).

In terms of learning opportunities for architects and administrators of Kamailio-based infrastructure, I found very valuable Daniel's presentations around high-level scripting (with KEMI) to build the routing logic (Video and slides).

Remember that Lua may not be the most popular - apparently - but it's the one estimated to give you performances closest to the native routing language.

Another valuable presentation was around the Least Cost Routing techniques that the Kamailio environment makes available. (Video, and slides). Some solutions use out of the box modules (like lcr, carrierroute, drouting), some are more indirect (pdt, mtree, dialplan, prefix_route), and others are a combination of them. Must-see if you're working in that area.

Another learning goldmine has been Lorenzo Miniero's (author of Janus, a WebRTC conferencing framework (this definition is mine)) lecture about Privacy, Security and Authentication for WebRTC. (Video and slides) Lorenzo does talk fast, but no word is spoken in vain. Worst case, you can watch the video at 0.5 speed (smile). Interesting the case of double encryption for media.

I guess there's enough for a part III in the near future! To be continued.

On Kamailio World 2018, part I

2018-05-24T10:20:00.000+02:00

This was my fifth time in a row attending Kamailio World in Berlin. The weather was warmer and sunnier than usual.

Apart from the obvious focus on Kamailio, as usual the RTC ecosystem was well represented (with Janus, Asterisk, FreeSWITCH, Homer, RTPEngine, and many others).

Attendance from the other side of the Atlantic Ocean gave stronger emphasis to the "World" term in the title.

My personal mission this year was to talk about a framework for testing Kamailio as a tool for developers and maintainers of the project: kamailio-tests. The main concept was that early tests that are not focused on a specific business logic (as we all have in our projects) and can be automated will be beneficial to Kamailio's reliability. We want to defer end-to-end testing to later stages, because they are expensive.

To provide a uniform infrastructure where to run the tests, without requiring permanent test environments, we use Docker for this. This is, of course, not the only possible approach, e.g. you could dynamically spawn VMs, AWS EC2s, etc. But Docker can run on your laptop as well as on a full-fledged CI environment, and this makes it easier to use for the developers.

Please take a look at the slides for more details. The feedback has been great so far, and this proved various points:

1. Conferences for developers are not paid holidays for IT guys, but opportunities for knowledge sharing and collaboration (I would say, in particular if Open Source is in the equation).

2. "Functional" or "component" testing is needed by many, but we haven't a mature solution yet.

3. Docker in RTC is less a fancy technology borrowed by other IT areas and more an everyday tool.

Some have already volunteered to help me improve kamailio-tests, and their point of view will be very useful. More on this project in the future.

Around the topic of testing, in this case not Kamailio itself but more the business logic built around it, there have been interesting insights from Sebastian Damm (sipgate) and Alex Sosic (evosip).

Sebastian presented an approach that benefits from moving the Kamailio routing logic from the native language to KEMI with Lua (https://github.com/sipgate/lua-kamailio). Alex presented a way to verify the routing logic is going through the expected paths, again with Docker, and sipp.

KEMI is an extension of Kamailio that allows developers to write the routing logic in high level languages, like Lua, Python, JS and others. Anedoctical experience made me think Lua was the most popular, while apparently Python is. For what concerns Lua in the RTC world, I wrote a few notes in February: http://www.giacomovacca.com/2018/02/the-interesting-case-of-lua-in-rtc-world.html

The advantages of working with a high level language are obvious: easier to read and maintain, it's easier to test the functions in isolation, and also easier to involve developers without specific knowledge in Kamailio's routing logic script. They will still need to understand how Kamailio works though, and the underlying protocols, so unless you're doing something extremely basic, it's not a complete abstraction from how Kamailio manages its role as "programmable SIP Proxy".

I have tons of notes from Kamailio World, but if I wait to go through all of them before writing something here, there will be the 2019 edition to talk about. So here's at least a part I.

The interesting case of Lua in RTC world

2018-02-05T21:44:00.000+01:00

An interesting pattern that caught my attention is the role that Lua is gaining in the RTC (Real-Time Communications) world.

Lua is a small-footprint programming language, powerful while keeping a simple syntax.

I’ve been using Lua to script dialplan actions for FreeSWITCH since about 2014. It has provided me with a way to define relatively complex logic and speed up the definition of FS’ behaviour.

Delegating this type of logic to a scripting language had several advantages, such as:

It’s easier to read and understand than native dialplans or native routing logic.
Makes unit testing of the dialplan possible/easier.
Allows changing some pieces of logic easily, in many cases preventing expensive reload of modules or restart of applications.

I’ve been using Lua for Kamailio as well. Kamailio is an open source programmable SIP Proxy. In a specific case, some bits of the routing logic required regex processing and was expecting to change often: an ideal case for an external script to do that work.

When the logic changes, it’s sufficient to instruct Kamailio to reload the script, and from that moment on the new requests being processed will use the new logic.

Recent versions of Kamailio though add a framework called KEMI. This opens up new possibilities, and also provides support for many other scripting languages, python being the most popular, JS, Squirrel. Still, Lua appears to provide the fastest implementations (with no observable performance degradation) while others have limitations. Python, as you can imagine, provides a rich set of functions and libraries, but it’s not as performant and the reload mechanism currently has some issues.

Wireshark, a tool to capture and analyse network activity, exposes a useful API for Lua. You can use the API to define your own Wireshark dissector (which you’ll need to install as a plugin). This has performance limitations in comparison with dissectors written in C - and so it’s recommended for prototyping only - but still can solve your problem perfectly. Out of need, I wrote a Wireshark dissector for HEP, a binary protocol used in the Homer environment. Homer is an open source framework for the monitoring and analysis of Real-Time Communications.

Last weekend a new interesting case was presented by Lorenzo Miniero at FOSDEM. The target application was Janus, an open source framework to build WebRTC gateways.

Janus allows to build applications by defining the transport and business logic as plugins, on top of the core that implements the WebRTC stack.

It’s written in C and so far users needed to write plugins with that language. The Janus developers have introduced the possibility to write plugins in Lua.

In his presentation Lorenzo explains also in detail what approach is best to use for a Real-Time application to interact with single threaded language like Lua in an asynchronous context.

Just a funny note: Lua uses a double dash for commenting out a line: '--'. Be careful when you watch diffs in a terminal because a removed comment will start with '- --‘ and may not the easiest thing to interpret (experiences may vary depending on the terminal, of course!).

Cache busting when building Docker images

2018-01-21T20:10:00.001+01:00

One of the handiest features of the docker build system is the caching system.

'docker build' tries to reuse the layers already built until something changes inside the Dockerfile. In this way, we can save several minutes when rebuilding an image if the changes happen further down the list in the Dockerfile.

Sometimes, though, we do want to invalidate the cache and ensure the next build won't use it.

To do this an option is to pass the '--no-cache' argument to 'docker build'.

When dealing with 'apt-get install' instructions though there are other tricks. I found this document on Dockerfile best practices very useful.

First of all an observation. If you have 'RUN apt-get update' as a single line of a Dockerfile, followed by the installation of a package, e.g.:

RUN apt-get update
RUN apt-get install -y nginx

then changing the list of packages and running again the build command won't trigger an 'apt-get update': that line hasn't changed so docker build reuses the cache. It might not be what you want.

To force cache invalidation for this specific case the recommendation is to use those commands in the form:

RUN apt-get update && apt-get install -y nginx

This will always install the latest version of the packages. It even has a name: "cache busting".

Another recommendation I like is to put each package on a single line, and have them in alphabetical order: this will ease visual inspection and prevent duplicates or other undesired conditions.

Of course, you can also specify exact versions for the packages as you would normally do with 'apt-get install'. That's "version pinning" and it invalidates the cache too.

You can find all this on the linked page on Dockerfile best practices; this is just my digested interpretation.

Just one more thing: a way to limit the size of a built image is to clean up the content of '/var/lib/apt/lists' in the same RUN command, e.g.:

RUN apt-get update && apt-get install -y \

aufs-tools \

automake \

build-essential \

&& rm -rf /var/lib/apt/lists/*

The command above will build an image layer that doesn't contain the apt cache.

If you had instead used this:

RUN apt-get update && apt-get install -y \

aufs-tools \

automake \

build-essential

RUN rm -rf /var/lib/apt/lists/*

you would have had not only a larger layer, containing the apt cache, but also an additional layer generated by the second RUN command.

SIP - ACK loose routing

2018-01-13T11:56:00.001+01:00

If you've ever worked with SIP, you must have stumbled upon a trace with 200 OK to INVITE being retransmitted for about 30" and then the call just being set up fail.

The ACK was never received.

Then comes the interesting part: discovering why.

Here are some notes about what should happen, in particular when there are multiple proxies along the path, and with a little additional complexity of one of the proxies with two network interfaces. All this assuming loose routing everywhere. The main reference here of course is RFC 3261.

Isn't an image worth a thousand words? Then here's a sequence diagram:

All Record-Route headers are assumed to carry loose routing URIs (they have the ;lr attribute).

B, C and D, working as proxies that want to stay in the path, record route themselves. For this reason E, the UAS and "callee", receives an INVITE with a list of Record-Route headers with B, C and D.
In particular for B there will be two Record-Route headers, since B is using two separate interfaces, one facing A and the other facing C.

In typical cases the two interfaces represent the interaction with the public Internet on a side and a private infrastructure on the other. But it's not important for this discussion.

Omitting provisional responses for simplicity, let's assume E responds immediately with a 200 OK. This response will have the same list of R-R headers, in the same order, as received by E.
E will also add its URI in the Contact header of the 200 OK.

In this loose routing context the IP address in E's Contact's URI will be relevant only for D in the future.

D, C and B don't modify the list of Record-Route headers, and A receives it as sent by E.

Apart from the operations related to the media session set up, A will send the ACK to the 200 OK.
This ACK will have a Request URI with E's Contact URI (stripped of anything that can't se inside a Request URI), and a Route header list which is basically the received Record-Route header list inverted (see images).

A is saying: "Route this ACK to E, routing it via this list of hops".

When B receives that ACK, it must recognise that the topmost Route headers are B itself, remove them from the Route list, and pick b2 as the interface to deliver the ACK to C.

C and D will have an easier task to remove a single Route header, the one representing them, and deliver the ACK to the next route.

For D, the next route will be in fact E, and the ACK will be routed using only the Request URI, as the Route headers have all been eliminated. This is the only step where the IP address that E has set in the Contact of its 200 OK response needs to be visible by another entity, namely D.

ACK routing as explained in RFC 3665

To further reiterate this concept, let's look at a somewhat simpler example in RFC 3665.

The ACK part is:

You can see there's no requirement for Proxy 1 to be able to reach the UAC contact (client.biloxi.example.com might be completely unreachable from Proxy 1).

It's Proxy 2's responsibility to route the ACK in the last hop towards Bob.

Proxy 1 must leave the R-URI as is (see below for more details on proxy behaviour), strip itself from the list of Routes and route the ACK to the new topmost Route (Proxy 2).

Proxy 2 will strip itself from the Route list, being the topmost Route, and forward the ACK to Bob. There are no more Route headers.

Only at the last hop Bob's contact reachability is relevant, and it is for Proxy 2 only.

More about the behaviour of the proxies to corroborate the ACK routing

From RFC 3261, 16.4:

“ If the first value in the Route header field indicates this proxy,
the proxy MUST remove that value from the request.”

From RFC 3261, 16.5:

“ A proxy can only change the Request-URI of a request during

forwarding if it is responsible for that URI.”

APPENDIX - Why is the ACK to 200 OK to INVITE a separate transaction?

From RFC 3261, ch. 17:

The reason for this separation is rooted in the importance of
delivering all 200 (OK) responses to an INVITE to the UAC. To
deliver them all to the UAC, the UAS alone takes responsibility
for retransmitting them (see Section 13.3.1.4), and the UAC alone
takes responsibility for acknowledging them with ACK (see Section
13.2.2.4). Since this ACK is retransmitted only by the UAC, it is
effectively considered its own transaction.

Copying a file from a Docker container to the host

2018-01-09T11:29:00.001+01:00

Often things are more convoluted than you expect. Sometimes they are easier than you fear.

Here's an example: you're generating files inside a Docker container (with no volumes configured) and you realise you need some of those files available in the host.

A simple solution is to use 'docker cp', with a format like

docker cp CONTAINER_ID:FILE_PATH HOST_FILE_PATH

Official documentation.
A Stackoverflow question with more discussions.

Around sipp with pcap and authentication support

2017-11-30T12:05:00.000+01:00

This is just a practical guide to have latest sipp on debian, to benefit from features that are not available in the stock package (sip-tester - version 3.2.1). For example support for playing pcap files or computing authentication hashes.

Build sipp with pcap support

apt-get install autoconf libncurses5-dev libpcap-dev g++
cd /usr/local/src/
git clone https://github.com/SIPp/sipp.git
cd sipp/
./build.sh --with-pcap

./sipp is the built binary. You can see the version and capabilities with './sipp -v', e.g.:

$ ./sipp -v
SIPp v3.6-dev-149-gb95f98f-PCAP-RTPSTREAM.
...

This version will be able to use actions like

exec rtp_stream="file.wav"

or

exec play_pcap_audio="pcap/g711a.pcap"

(see details in current documentation).

A little caveat for 'rtp_stream' and WAV files. As the documentation says sipp expects a WAV file encoded with PCAM ('A-law'). You can also loop that audio for as long as you need, by adding ',-1' after the file name (and within the double quotes).
But sipp will also send the WAV header in the RTP payload. If this is not acceptable for you, then you can strip the WAV header with something like:

sox -t WAV -r 8000 -c 1 -e a-law audio.wav audio.raw

(where sox is this tool).

play_pcap_audio is useful when you want full control of the RTP produced by sipp, e.g. the exact RTP SSRC, missing packets, Mark set bits, etc. Very powerful.

Add authentication support

Same dependencies as above, but add:

apt-get install libssl-dev

./build.sh --with-pcap --with-openssl

./sipp is the built binary. You can see the version and capabilities with './sipp -v', e.g.:

$ ./sipp -v
SIPp v3.6-dev-149-gb95f98f-TLS-PCAP-RTPSTREAM.
...

See the authentication features in current documentation.