SoundCloud Backstage Blog

Moving to Modern Streaming: Introducing AAC HLS on SoundCloud APIs [Deadline Extended]

Mon, 15 Sep 2025 00:00:00 GMT

At SoundCloud, we’re always looking for ways to improve the listening experience for our creators, partners, and listeners. One of the biggest improvements we can make is at the very core: the audio streams themselves. Over the years, our API has returned a set of streaming URLs based on MP3 and Opus formats. While these formats served us well, it’s time to modernize. To provide higher quality, more consistent playback, and better support across devices, we’re moving to AAC-based HLS streams.

🚀 What’s changing?

Starting today, the streams endpoint (/tracks/{track_urn}/streams) includes new HLS AAC transcodings:

hls_aac_160_url (standard quality, preferred format and bitrate)
hls_aac_96_url (alternative quality, depending on availability and can work as fallback to the 160k stream)

For a transition period, existing fields like http_mp3_128_url, hls_mp3_128_url, and hls_opus_64_url will remain available.

⏳ Deadline Extended: Deprecation Timeline

We plan to remove Progressive HTTP, HLS MP3 and HLS Opus transcodings from our APIs on December 31, 2025. The following fields will no longer be supported after that date:

http_mp3_128_url (Progressive HTTP MP3)
hls_mp3_128_url (HLS MP3)
hls_opus_64_url (HLS Opus)

preview_mp3_128_url will remain available for preview use cases.

✅ What you need to do

Migrate from MP3 / Opus to AAC
- Replace http_mp3_128_url, hls_mp3_128_url, and hls_opus_64_url with the new AAC HLS streams.
- Use hls_aac_160_url (preferred) or hls_aac_96_url where available.
Migrate from Progressive HTTP to HLS
- Progressive MP3 playback (http_mp3_128_url) will no longer be supported.

Test your integration with the new URLs to ensure a smooth migration.

🔍 Example Response as of now

{
"http_mp3_128_url": "https://cf-media.sndcdn.com/trackid.128.mp3?...",
"hls_mp3_128_url": "https://cf-hls-media.sndcdn.com/playlist/trackid.128.mp3/playlist.m3u8?...",
"hls_aac_160_url": "https://playback.media-streaming.soundcloud.cloud/trackid/aac_160k/uuid/playlist.m3u8?...",
"hls_opus_64_url": "https://cf-hls-opus-media.sndcdn.com/playlist/uuid.64.opus/playlist.m3u8?...",
"preview_mp3_128_url": "https://cf-preview-media.sndcdn.com/preview/0/30/trackid.128.mp3?..."
}

🔍 Example Response after Dec 31, 2025

{
"hls_aac_160_url": "https://playback.media-streaming.soundcloud.cloud/trackid/aac_160k/uuid/playlist.m3u8?...",
"hls_aac_96_url": "https://playback.media-streaming.soundcloud.cloud/trackid/aac_96k/uuid/playlist.m3u8?...",
"preview_mp3_128_url": "https://cf-preview-media.sndcdn.com/preview/0/30/trackid.128.mp3?..."
}

💬 Feedback

We’d love to hear how the transition works for you. If you run into issues or need additional support, please open a discussion on GitHub. This change sets the foundation for better audio quality and playback reliability across SoundCloud integrations. We encourage you to migrate early and share your feedback.

📣 Action required: Upcoming Update to SoundCloud API id fields

Tue, 29 Apr 2025 00:00:00 GMT

To all developers using SoundCloud’s public API — we’re introducing an important change that will affect how you reference resources like tracks, playlists, users and comments.

🧠 What’s Changing?

SoundCloud’s resource identifiers (id) have always been numeric. As the platform grows, some of these numbers will soon exceed the maximum value of a 32-bit integer. To ensure forward compatibility and prevent broken integrations, we’re deprecating the id field and introducing a new field: urn — a string-based unique identifier.

🚨 As a result, the id field in API responses is now deprecated. We strongly recommend switching to the new urn field — a string-based unique identifier.

🗓️ Timeline

Effective immediately: The id field is deprecated.

By June 30th, 2025: Clients must switch to using the urn field.

After June 30th: Services relying on the id field may experience degraded behavior:

Partial data loss (e.g., new resources may not appear)
Full API response failures in some cases

✅ What You Should Do

Update your integrations to rely on the urn field instead of the id. The urn is a stable, unique string that will not be affected by integer limitations.

💡 Example Changes

Here’s what the change looks like for different resource types:

🎵 Track

Before:

  {
    "id": 12345678,
    "title": "My Track"
  }

After:

  {
    "urn": "soundcloud:tracks:12345678",
    "title": "My Track"
  }

📁 Playlist

Before:

{
  "id": 98765432,
  "title": "My Playlist"
}

After:

{
  "urn": "soundcloud:playlists:98765432",
  "title": "My Playlist"
}

💬 Comment

Before:

  {
    "id": 87654321,
    "body": "Great track!"
  }

After:

  {
    "urn": "soundcloud:comments:87654321",
    "body": "Great track!"
  }

Accessing resources via URN

All the endpoints are ready to take either id or urn!

Before: https://api.soundcloud.com/tracks/12345678
After: https://api.soundcloud.com/tracks/soundcloud:tracks:12345678

We recommend to move always URN going forward. Accessing via id is deprecated immediately!

📦 Bonus Tip

To extract the numeric part of the URN for legacy support or internal mapping, you can parse the URN string — but we strongly discourage from doing, isntead treat the URN as the primary key moving forward.

💬 Need Help?

If you have questions or run into issues, please reach out through our developer support channels, open issues on GitHub or check the updated documentation here.

Let’s make this transition smooth — thanks for building with us ❤️

New artist name field on our API

Thu, 03 Apr 2025 00:00:00 GMT

A while ago we made an improvement to how track and artist information is displayed on SoundCloud, making it clearer and easier to understand.

Previously, it could be confusing when content wasn’t directly uploaded by the artist. One example might be tracks that are coming in as official label content. Another one would be artists collaborating.

This often lead to long track titles containing artist names, that were difficult to read on mobile devices. With our changes we cleaned those titles up. Now underneath the track we show the correct artist instead of the profile name.

For user uploaded content, we now display the information that users give us in the ‘artist’ metadata field during upload. If they leave this field blank, we will display their profile name.

Now this field is, finally, also availabe on our API.

Look out for metadata_artist

{
    "title": "Creepin'",
..
    "user": {
        "username": "Metro Boomin",
..
    },
..
    "metadata_artist": "Metro Boomin, The Weeknd, 21 Savage"
}

Upcoming changes to SoundCloud Access Tokens

Mon, 02 Dec 2024 00:00:00 GMT

tl;dr: If your app doesn’t store SoundCloud access tokens outside of the context of a user’s client (i.e. a database) the following should not affect you.

There will be upcoming changes to Access Tokens important for third parties integrated with SoundCloud. Access tokens were previously dispensed as opaque strings and going forward will be transitioning to a JWT format.

Ideally, clients should not rely on the shape/format of these tokens as this is transitive PII user data, but for example, embedded apps that are perhaps storing this information in a schema based environment, the size of the tokens will no longer fit in a basic 255 varchar. There will be no guarantees about size, as JWT’s are varying by nature and are dependent on the contents encoded inside of them.

Please update your app accordingly so that users do not incur any issues with your app’s integration.

Migration to Oauth 2.1

Tue, 23 Apr 2024 00:00:00 GMT

Earlier this year, we updated our API Guide to let developers know that SoundCloud authentication is now operating on the OAuth 2.1 protocol, a popular open standard used by many API providers. This change needed to be made to ensure that we are staying up-to-date in maintaining the highest level of security possible to protect our platform from potential bad actors.

As a result, we must ask that our developer community adhere to this new standard as well. We are giving developers who already implemented SoundCloud authorization using OAuth 2.0 a grace period to transition their integrations to OAuth 2.1 and ask that these updates be made by October 1, 2024, at the latest. We will deprecate the OAuth 2.0 protocol after October 1, so your integration will no longer function properly should you fail to meet this deadline.

For reference, you can see the differences between these standards in the RFC. The most notable change is that PKCE is now required in order to securely exchange the auth code.

Please understand that we will not be making exceptions to this requirement. We will be notifying all developers of this update via email, and will continue to send reminders over the next months. As always, also be sure to keep an eye on this blog and our Developer Twitter account for other updates to come.

Android Large Screen Optimization

Mon, 31 Jul 2023 00:00:00 GMT

Large Screen Devices - The New Frontier

Recently, the Android team at SoundCloud took on a project to optimize the Android app for large screen devices. With the increasing use of devices like tablets, foldables, and Chromebooks, we decided to provide better support for them.

The data showed us users were adopting these devices rapidly. What is more, both Google and Samsung had been vocal about the importance of optimizing for larger screens.

Google is even launching a new Playstore for large screen optimized apps only.

In this blog post, we’ll go through some of the challenges we faced, the UX and UI changes we made, and the code implementation. We hope our experiences will help you in your own large-screen adventures.

Why did we invest in Large Screens?

We saw a trend among our users - more and more of them were using SoundCloud on their tablets, foldable devices, and Chromebooks. And it wasn’t just happening in our user base. Google and Samsung have been going all-in on large screens, bringing them more into the mainstream.

When Google rolled out their Pixel Tablet, well-known tech reviewer Marques Brownlee pointed out that SoundCloud stood out from the crowd. He appreciated that we didn’t just stretch our phone app to fit the larger screen, but instead, we crafted an experience that fully utilized the extra space. If you’re interested, you can check out the video here.

Our Investigation

Where and how is music on large screens heard?

Fun fact: People listen to music on their Tablets mostly while cooking.

Imagine you’re in the middle of preparing a meal, your hands are busy, but you can still easily control your music - that’s convenience at its best.

But it’s not just about individual experiences. Tablets come into play during social events as well. The larger screen makes it simpler for everyone to engage with the music, whether it’s to see what’s currently playing or to add their favorite track to the queue.

Through understanding these usage scenarios, we can better tailor our services to enhance the music listening experiences of our users. Be it cooking dinner or hosting a gathering, SoundCloud is there to make those moments even better with great music.

Google’s and Samsung’s Large Screens Guide

Google’s and Samsung’s large screen guides are roadmaps for developers wanting to optimize their apps for large screens. It encourages using scalable images and responsive and dynamic layouts for a consistent user experience across different screen sizes.

Google also highlights using the extra space for features like split-screen views, and emphasizes touch target sizing, clear text, and intuitive navigation. Following these best practices ensures apps are user-friendly and competitive in the large screen devices market.

Google’s Large Screen Compatibility Checklist

Google’s Large Screen Checklist, available in their Android developer documentation, provides developers with a tier-based framework for ensuring high-quality app experiences on large screen devices. The checklist encompasses not only UI design but also considerations for navigation, keyboard/stylus input, and overall app usage.

The checklist is divided into tiers, offering developers a structured approach to prioritize and implement essential features and optimizations based on the level of support desired for large screen devices. The tiers range from basic support to more advanced features and functionalities that fully leverage the potential of large screens.

The SoundCloud App

When we reviewed Google’s checklist it was clear that the SoundCloud app fell short in certain areas, particularly in terms of adapting the screen layout. The app’s tendency to merely stretch the phone layout to fit larger screens resulted in a less than optimal user interface. This realization was our call to action.

In the upcoming sections, we’ll take you on a visual tour of these components, accompanied by code snippets and technical explanations to provide you with an inside look at our optimization process.

Bottom Navigation Bar

Before: ❌ Horizontally stretched phone layout bottom navigation bar

Traditionally, SoundCloud’s Android app utilized a bottom navigation bar in a phone layout, which was stretched horizontally to fit larger screen sizes. This approach was straightforward, but it proved ineffective for larger screens. The extended navigation bar made it difficult for users to navigate the app, as it required them to move their focus from one end of the screen to the other.

After: ✅ Nav bar rail optimized for large screens

To remedy this, we adopted a vertical navigation rail for large screen layouts, aligning it to the side of the screen. This is recommended by Google as it provides a more comfortable browsing experience. The menu options are now neatly stacked in a column, making them accessible without the need for extensive eye or cursor movement.

Code Implementation

Before: BottomNavigationView

Previously, the SoundCloud Android app used a BottomNavigationView to create the bottom navigation bar. This worked well on smaller devices but led to a stretched UI on larger screens.

After: NavigationRailView

To address this, we replaced the BottomNavigationView with NavigationRailView for larger screens. Both these classes implement the NavigationBarView interface. This allows for a smooth transition with minimal code changes.

To differentiate between smaller and larger screens, we created two separate layout files. One uses the BottomNavigationView, and the other uses the NavigationRailView. In Kotlin, we dynamically set the layout file based on the size of the device.

Identifying Large Devices

We identify whether a device is large or small by using a boolean variable, is_tablet, defined in three separate XML value files based on the screen width.

// res/values-sw600dp/booleans.xml
<bool name="is_tablet">true</bool>

// res/values-sw370dp/booleans.xml
<bool name="is_tablet">false</bool>

// res/values/booleans.xml
<bool name="is_tablet">false</bool>

In Kotlin, we check the is_tablet boolean value and set the layout file accordingly. This is done using the following code:

val isTablet = context.resources.getBoolean(R.bool.is_tablet)

if(isTablet) {
    setContentView(R.layout.tablet_layout)
} else {
    setContentView(R.layout.phone_layout)
}

Hardcoding resources is not the best solution. Google’s documentation provides some great insights on supporting different screen sizes, including the concept of Window Size Classes. Window Sizes allow the UI to adapt to various window configurations on foldable and large screen devices. Here’s the link for more details: Window Size Classes.

Unit Testing and A/B Testing

To enable testing, we added a wrapper class around the isTablet boolean. We also leveraged feature flags to conduct A/B testing of our new layout. We introduced a class NavRailExperiment to check both the device type and the feature flag state: Here’s a look at our implementation of this approach:

class NavRailExperiment(private val appFeatures: AppFeatures) {
    val isEnabled: Boolean
        get() = context.resources.getBoolean(R.bool.is_tablet) && appFeatures.isNavRailEnabled()
}

Mini Player

Transitioning from a Stretched Phone Layout Mini Player to a Right-Aligned Mini Player

Before: ❌ Horizontally stretched phone layout Mini Player

In the past, SoundCloud’s Android app featured a Mini Player stretched horizontally across the screen. While this design worked for smaller screens, it resulted in a suboptimal user experience on larger devices due to the extended reach required to interact with it.

After: ✅ Small right-aligned Mini Player optimized for large screens

To provide a more user-friendly interface we opted for a small Mini Player on the right side of the screen. This ensures controls are comfortably within thumb reach.

Code Implementation

The Player is a BottomSheet with a ViewPager, the Mini Player is a view inside each page of the ViewPager. We use a BottomSheetBehavior to support collapsed and expanded states. In the expanded state the player shows the full width and height of the ViewPager. In collapsed state it shows only the Mini Player view, and the height is collapsed to 50dp. The width was always match_parent, but since we want it to be small, we set the width of the BottomSheet to 200dp.

The class BottomSheetBehavior provides us with a method setMaxWidth which we use to set the width while animating the collapse and expansion of the BottomSheet Player. The other issue here is that we also have to set the width on the ViewPager and the ViewPager’s visible page. Additionally, we must move the player to the right side thus setting the left margin on layout params on each player transition.

TheBottomSheetBehavior also has a BottomSheetBehavior.BottomSheetCallback field. The BottomSheetBehavior.BottomSheetCallback has an onSlide method which allows us to set the width and left margin on the BottomSheet and the ViewPager’s visible page.

Below is the code:

bottomSheetBehavior.bottomSheetCallback = BottomSheetBehavior.BottomSheetCallback() {         
    val miniplayerWidth = screenWidth * offset
    val leftMargin = screenWidth - miniplayerWidth
        
    setLeftMargin(viewPager.views(), leftMargin)
        
    setWidth(viewPager.views(), bottomSheet, miniplayerWidth)
};

Comments & Playqueue on Player

The SoundCloud Android app hosts two main features on its player screen - the comments and the Playqueue. Before, when a user accessed either of these features, a new screen would open, occupying the entire display and obstructing interaction with the player.

Before: ❌ Horizontally stretched phone comments layout

Before: ❌ Horizontally stretched phone playqueue layout

We revamped this experience in the following ways: Instead of taking up the whole screen, the Comments and Playqueue sections now open on the right side of the player. This allows users to engage with these features and interact with the player simultaneously.

Additionally, the buttons to access the comments and Playqueue were originally at the bottom of the screen. To increase accessibility and free up space at the bottom, we moved them to the left side.

After: ✅ Right-aligned comments optimized for large screen

After: ✅ Right-aligned playqueue optimized for large screen

Repositioning the UI elements in this way contributes to a more ergonomic and intuitive user experience on larger screens, reinforcing our commitment to delivering the best listening experience across all devices.

Addressing User Experience (UX) Challenges

Our changes to the UI introduced some UX challenges, especially related to the comments feature. Two major questions emerged:

When a track changes, should the comments update to correspond to the new track, or should it continue displaying comments from the previous track?
What should happen to a comment a user is currently writing if the track changes?

To tackle the first issue, we opted to update the comments to align with the new track. We believe this adjustment is more in line with user expectations, as it helps keep the entire screen contextually relevant.

For the second scenario, we implemented a solution where the comments section doesn’t update if the text field is currently in focus. This prevents any in-progress comments being lost, thereby providing a more user-friendly commenting experience.

Code Implementation

To integrate the comments and Playqueue screens over the player, we introduced a new FrameLayout named player_side_fragment_holder. This frame holds the additional features and ensures they are displayed without disrupting the main player.

This side fragment holder is positioned above the player and is assigned a transparent background along with a width of 400dp. Consequently, when the user taps on the comments or Playqueue button, the respective screen appears over the player. This is made possible by a secondary FragmentManager which adds either the CommentsFragment or the PlayqueueFragment to the FrameLayout.

Here’s how the layout is structured:

<FrameLayout
    android:layout_width="match_parent"
    android:layout_height="match_parent">

    <Player
        android:layout_width="match_parent"
        android:layout_height="match_parent" />

    <FrameLayout
        android:id="@+id/player_side_fragment_holder"
        android:layout_width="400dp"
        android:layout_height="match_parent"
        android:layout_gravity="right" />
</FrameLayout>

In this layout configuration, the player_side_fragment_holder is specified to align on the right side, allowing it to overlay the player when comments or the Playqueue are invoked. This approach ensures that the main player remains interactive while users are interacting with these features.

Search

In our drive to optimize the large screen experience, we revamped the search screen to offer both search suggestions and top results side by side. The updated design splits the screen into two sections when users begin a search. One side displays real-time search suggestions, while the other showcases the top results. This allows users to compare suggestions and top results instantly, leading to a more seamless and enriched search experience.

Before: ❌ Horizontally stretched search suggestions

After: ✅ Split search suggestions and search results

Code Implementation

Code Implementation for Split Search Screen

In order to implement our split-screen search feature, we use ConstraintLayout. This offers the ability to create layouts based on percentages, which we use to divide the screen between search suggestions and top results.

To split the screen, we defined a width constraint of 40% for the search suggestions, leaving the remaining 60% for the top results:

app:layout_constraintWidth_default="percent"
app:layout_constraintWidth_percent=".4"

Here is the full code for the side-by-side layout:

<androidx.constraintlayout.widget.ConstraintLayout
    android:layout_width="match_parent"
    android:layout_height="match_parent">

    <FrameLayout
        android:id="@+id/search_container"
        android:layout_width="0dp"
        android:layout_height="match_parent"
        app:layout_constraintWidth_default="percent"
        app:layout_constraintWidth_percent=".4"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintStart_toStartOf="parent"
        app:layout_constraintTop_toTopOf="parent">

        <androidx.fragment.app.FragmentContainerView
            android:id="@+id/search_suggestions"
            android:layout_width="match_parent"
            android:layout_height="match_parent" />
    </FrameLayout>

    <androidx.recyclerview.widget.RecyclerView
        android:id="@+id/section_results_top_items"
        android:layout_width="0dp"
        android:layout_height="wrap_content"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintEnd_toEndOf="parent"
        app:layout_constraintStart_toEndOf="@id/search_container"
        app:layout_constraintTop_toTopOf="parent" />
</androidx.constraintlayout.widget.ConstraintLayout>

This layout allocates 40% of the screen width to a FrameLayout which houses the search suggestions, while the remaining space is assigned to a RecyclerView for displaying top results. The constraints ensure the proper positioning and responsiveness of these elements, providing a dynamic, user-friendly search interface.

Artist

Our previous design for the Artist screen wasn’t well-optimized for larger screens. The result was a horizontally stretched layout with excessive padding and white space around the artist details. As a consequence, users had to scroll vertically to access tracks or albums from the artist.

Before: ❌ Stretched artist profile

To improve this, we took advantage of the space available on larger screens. We moved the Artist details to the left side of the screen and displayed them more compactly. This makes all the necessary information immediately visible and reduces unnecessary scrolling.

Simultaneously, the right side of the screen is dedicated to displaying the artist’s tracks and albums. This clear separation of content means that users can quickly view and interact with the artist’s music. Through this design, we’ve been able to optimize space and present a well-structured layout that enhances the navigation and interaction for our users on large screen devices.

After: ✅ Compact artist profile

Keyboard Tab Navigation

We needed to support keyboard tab navigation for devices that have keyboards attached. You can easily test your apps by connecting a bluetooth keyboard.

If you are using AppBarLayout you might end up not navigating through the toolbar icons. We had to set touchscreenBlocksFocus = false to our AppBarLayout in order to allow navigating through the toolbar icons.

At first, we had problems navigating the player using a keyboard. The default navigation did not work correctly so we had to force set the navigation flow. To do this we use the xml attribute android:nextFocusForward.

For better understanding, here’s the specified order for tab key navigation in the player:

First, the focus lands on the artist’s name.
Then, it moves to the track name.
Next, it goes to the option to minimize the player.
After that, it directs to the ‘Follow artist’ option.
It subsequently moves through options like ‘Like’, ‘Comments’, ‘Share’, ‘Playqueue’, and ‘More’ buttons.
Lastly, it settles on the ‘Play/Pause’ button.

Space bar toggles playback - Only for media playback apps

One of the requirements from Google is to support playback toggling through the space bar key. This feature allows users to play or pause music just by hitting the space bar on their keyboards, irrespective of their current position in the app.

The implementation of this functionality can be achieved by overriding the onKeyUp method. On detection of a key press, it verifies if the pressed key corresponds to the space bar, and if so it toggles the playback and returns true.

In all the Activity classes where the play/pause button is present, the onKeyUp method is overridden as shown below:

override fun onKeyUp(keyCode: Int, event: KeyEvent): Boolean {
    takeKeyEvents(true)
    return if(keyCode == KeyEvent.KEYCODE_SPACE) {
        true
    } else {
        super.onKeyUp(keyCode, event)
    }
}

It’s worth noting that we used the takeKeyEvents(true) method to ensure that the current activity can handle key events.

Finally, to make sure the play/pause button is the default focus, we set android:focusedByDefault="true" on the button. This is done to prevent other views from gaining focus by default, which could intercept the space bar key press event.

Testing on Emulators and Devices

When it comes to testing on large screens, we found the Desktop emulator to be the most suitable option. The key benefit is its ability to swiftly resize the app window, mirroring the actions performed on a typical device. This allows us to test responsive behavior on different devices like phones, tablets, and desktops.

Google also offers support for emulators designed for foldable devices. These emulators can be useful for preliminary testing. However, we mostly relied on a physical device, the Samsung Fold, to conduct tests for foldable devices, as it proved to be more accurate.

Lastly, to ensure our app supports hover states and keyboard operations efficiently, we used the Samsung S7+ equipped with an S Pen and Keyboard Bundle. This allowed us to simulate an environment similar to a laptop or a tablet with a keyboard.

Conclusion

Our journey to optimizing for large screens was filled with exciting challenges and insightful discoveries. We realized that providing a seamless experience across a variety of devices - from smartphones to tablets, desktops, TVs, foldables and more - was more than just stretching our existing user interface. It was about reimagining and reconfiguring every element and interaction to deliver a superior music streaming experience.

In addressing the various UI and UX challenges, we found that each element of our service had the potential to be adapted and improved. We reworked our Player and Search screen, redesigned the Artist screen layout, and made sure that keyboard navigation and hover states were smoothly implemented. This process of continuous improvement reinforced to us how important it is to stay flexible in the face of evolving trends.

As technology continues to change and expand, we are committed to staying ahead, making sure that SoundCloud remains the leading choice for music streaming, no matter the device.

An update on our API: Changes to access for inactive apps

Fri, 02 Jun 2023 00:00:00 GMT

We’re working on improving our API offering. In order to do that, a few changes are coming:

We need to revoke access for some of you with inactive apps.
We’ve reached out directly to those who will be affected via email and we’re working to mitigate impact where we can.

We need to do this for two reasons:

Recently, we haven’t been able to provide developers with the level of support that you deserve. That includes being able to help you when things go wrong and supporting you when you build amazing things.
More pressingly, we haven’t been able to sustain the level of oversight needed to make sure our bad actors aren’t abusing our API. After analyzing trends in abusive API calls, we’ve determined that change is necessary in order to protect the artists and fans that use SoundCloud’s services.

Developers have been and will continue to be an important part of the SoundCloud ecosystem. Rest assured, this isn’t the final state of our API access – just a temporary reset so that we can implement additional security mechanisms. We’ll also be using this opportunity to redesign our API program so that there’s more internal support and better optimization for individual developers.

In the meantime, we’re working hard to reopen the doors as soon as possible for developers that want to use our API. Be sure to keep an eye on this blog and the SoundCloud Developer Twitter Account for updates to come.

What I Learned in One Year as an SRE Trainee

Fri, 06 Jan 2023 00:00:00 GMT

I recently celebrated my one year anniversary as a Site Reliability Engineering (SRE) trainee at SoundCloud. Looking back, I had very little idea of what I was getting into. I studied politics, and had done various teaching and translating jobs over the years before deciding to learn programming. Now I had a coding bootcamp under my belt, but no SRE experience. That said, I was drawn to the idea of troubleshooting complex systems and automating away repetitive tasks. I was also incredibly excited to work at SoundCloud, the world’s largest music and audio platform that lets people discover the greatest selection of music from the most diverse creator community on earth.

SRE is an approach that uses software engineering concepts to solve operations problems. Its principles work to align the goals of development and operations to create reliable and scalable systems.

My team, Production Engineering, is responsible for ensuring the availability of SoundCloud. We make sure that systems are resilient and running smoothly, whether by taking care of it ourselves or enabling other teams to own what they build.

In this post, I want to share my experiences as a new SRE. I’ll reflect on the lessons I’ve learned and challenges I’ve faced over the past year. I hope to give you an insight into the role of an SRE, and perhaps even inspire others to consider a career in this field.

The scope is huge

The first thing I noticed was the scope of the work that we do. Everything from Infrastructure as Code to Monitoring and Incident Response training potentially falls under the SRE umbrella. This has been overwhelming at times. However I remind myself that I’m not going to learn everything in one day, and take it one step at a time.

On the other hand, this huge scope is also why I love it. There’s definitely no shortage of things to learn. Some things I’ve worked on so far include upgrading Kubernetes and decommissioning part of our infrastructure. I’m currently in the process of migrating a service from our data center to Google Cloud. This is part of the exploratory phase of a bigger migration, and what I learn from it will benefit other teams when the time comes for them to do the same.

I’ve also taken on the role of First Responder. Supporting other teams is a big part of our job, and the First Responder is the dedicated person for this. It means it’s clear who to ask for help, and the rest of our team can work without interruptions. Part of the role is also looking into the alerts our team gets. This can be daunting because it involves getting to know the nitty gritty details of each specific system. However it’s no coincidence this is also when I learn the most.

On the cultural side of things, I lead the SRE Collective. This is a space where we get together to talk about SRE topics and share knowledge. The format is flexible, and leading means anything from facilitating a discussion to inviting someone to give a presentation.

If this year has been about getting a high level overview, I’m looking forward to diving deeper into some topics in the year to come.

Systems fail all the time

Another thing that stood out to me is the way we view failure at SoundCloud. Failure is inherent in complex systems, what matters is how we deal with it. At SoundCloud we practice blame aware postmortems. This means we carry out incident reviews with a focus on learning rather than pointing fingers. When failure is viewed as something normal, not as something to be swept under the rug, then we can examine what happened, learn from it, and make our systems stronger.

This outlook has also challenged how I think about cause and effect. Often in life we look for the root cause of something. However there is rarely (if ever) one root cause, but rather a series of contributing factors. One of the most interesting yet challenging aspects of my traineeship has been understanding how these factors interact to result in the functioning or failure of a system.

Kubernetes is not so scary

Perhaps the most intimidating technology I’ve had to learn this year is Kubernetes. I’m not sure if it was its reputation for being complex, or the fact that it comes with a whole new vocabulary (pods, replicaSets, etc.), but I put off learning it for a while before finally diving in. And it’s true - it is complex and it took some time to get my head around it. However the process of learning it taught me something that can be applied to any technology.

Basically, it’s essential to understand why that technology exists in the first place. What problem does it solve? In the case of Kubernetes, it solves the problem of having a running application tied to physical infrastructure. This is a problem because if a server crashes then the application goes down with it. While it’s true there’s more that can be said about Kubernetes, understanding this made it click for me and from that point on it became a lot less confusing. I still have a lot more to learn, but knowing its basic purpose makes new concepts fall into place easily.

It’s all about asking questions

There have been so many new concepts to grasp and technologies to learn, and asking questions has been key to improving my understanding. It’s true that reading and googling have their part, but asking questions and chatting with my teammates is when it really comes together.

I’m also lucky enough to be working with some incredibly smart and talented people who are more than happy to share their knowledge with me. It’s an opportunity that’s not to be missed!

As a team, we decided to set aside a dedicated time for questions, and we now have a ‘Coffee and Q&A’ session twice a week. It started with me as a trainee in mind, but quickly turned into an informal knowledge sharing space for the whole team. It’s a time where anyone can bring questions that have come up during the day, or ask things that are outside the scope of their daily tasks. It’s also a great team bonding moment.

The journey continues

When I look back to when I started it’s hard for me to believe how far I’ve come. I’ve improved my technical skills, worked on some very exciting projects, and learned a lot about SRE culture along the way. However all this was only possible because of the support I got from my colleagues. Whether as pair programming sessions or informal chats, the time they have given me and the kindness and enthusiasm they have shown me has been astounding.

I’m also happy to share that I’ve accepted a permanent position on the team, and I’m excited to continue my journey as an SRE at SoundCloud.

Look out for my next post, where I’ll share my experiences on migrating to the cloud.

What's New with SoundCloud, November 2022

Fri, 11 Nov 2022 00:00:00 GMT

We recently made updates to our SoundCloud iOS and Android apps to address key user experience (UX) fixes according to user feedback.

We’ll be updating this blog regularly as we actively work to improve the SoundCloud experience for our users, and we’re so excited to share these improvements with everyone.

Here are our most recent updates across iOS and Android as of 11 November 2022.

New Creator Subscription Names (Android, iOS)

With the rollout of our Next, Next Pro, and Next Plus subscription names as part of our SoundCloud for Artists evolution, we’ve replaced old creator tiers on profiles to align with these new plan names, resulting in a more streamlined experience. This will be available to all users.

Sheet for Playlist/Album Descriptions (iOS)

We know that what artists and curators have to share about their music is important — so important that we’ve given playlist and album descriptions their own dedicated space with room for 4,000 characters.

Previously, when expanded, descriptions would take up the entire screen. To accommodate both artist expression and ease of platform use, we separated these two elements.

Reduced Carousel Sizes (iOS)

Following our initial reduction of track, playlist, and album cell sizes, we’ve now also reduced the size of carousels across the app to fit even more music and content you know and love.

Combined Share and Additional Options Icon (iOS)

To streamline and simplify users’ experiences, reduce clutter, and prioritize key actions, we’ve added sharing to the “more” icon.

Bug Fix: Creator Profiles in Recently Played (iOS)

We fixed a bug that caused creator profiles to be displayed as playlists on Home and Library views.

Changed Background of Comments (iOS)

In an effort to create added consistency across our app, we removed the blurred overlay in the comment section.

Reduced Subtitles on Home (Android, iOS)

We’re continuing to simplify screens by removing redundancies on our Home screen, including only introducing subtitles where necessary. For US users who previously had subtitles attached to almost every module, this change results in a more organized experience and faster scanning that clearly conveys the content they’re seeing.

More to Come

We’re excited to unveil future updates to SoundCloud, and we couldn’t do it without your support and feedback — stay tuned for what’s next!

Bar Raiser Interview

Tue, 25 Oct 2022 00:00:00 GMT

In an earlier blog post, we talked about how we changed the interview process during remote working as a way of improving our recruiting efforts. In this post, we’ll talk about the training process we defined for the Bar Raiser interview.

The Bar Raiser interview is a critical part of the final round of engineering interviews at SoundCloud, and it’s focused on increasing the quality of hiring, which ultimately has an impact on the products we develop for our users.

Bar Raiser interviews are well-known in the industry. Amazon popularized the concept of the Bar Raiser role, which is a unique part of its culture, by focusing on interviewing candidates and overseeing the hiring process so that every new hire is “raising the bar.”

What Is a Bar Raiser at SoundCloud?

A Bar Raiser is an interviewer at SoundCloud who’s brought into the hiring process to contribute an outside perspective to the hiring panel. They represent the technical excellency, culture, and mindset of SoundCloud. Bar Raisers have a history of hiring effective engineers, experience running interviews, and technical depth in multiple areas. With each new hire, we seek to improve and raise our standard.

Who Are Bar Raisers?

Bar Raisers are experienced interviewers who have usually worked in more than one area at SoundCloud. They carry the mindset of continuous improvement and empowering one’s self and others. They focus on value for our users and have built and operated systems at scale. They value and promote workplace diversity and have seen the long-term effects of hiring decisions.

To join the group of Bar Raisers, a candidate should show a desire to exhibit cross-engineering awareness, the ability to identify the best engineers in the industry, and a willingness to put SoundCloud’s hiring needs above a single team’s hiring needs. If they see themselves as a good candidate for this role, they go through a training process to observe how these interviews are conducted and further practice themselves.

What Is the Goal of the Bar Raiser Interview and How Do Bar Raisers Contribute to the Hiring Process?

The goal of the Bar Raiser interview is to determine the attitude and instincts of a candidate to understand whether any given team at SoundCloud would culturally and technically benefit when working with a specific person, and if the person would in turn thrive in any team they’re assigned to at SoundCloud. By bringing these critical signals to the hiring process, the Bar Raiser helps assess to what extent a potential hire could raise the effectiveness of every engineering team.

What Is the Training Process to Become a Bar Raiser?

To become a Bar Raiser at SoundCloud, we’ve defined a training process that comprises multiple steps. This includes shadowing established trainer Bar Raisers for multiple interviews over the course of several months. Prior to this, the candidate Bar Raiser must attend other training sessions, including one on removing unconscious bias.

Trainer Bar Raisers assess candidate Bar Raisers through a series of real Bar Raiser interviews. Each candidate Bar Raiser first shadows a trainer conducting one Bar Raiser interview, and then is shadowed by the trainer for several interviews. After each shadowing interview session, the candidate and trainer debrief about the methods of the interviewer but not the content. Each individually writes an interview scorecard, and together they compare these in a follow-up debrief. During the debrief, they discuss how the interview process went and what observations and conclusions were covered or missed.

After these sets of interviews, the trainer Bar Raiser decides whether the candidate Bar Raiser is ready to conduct these interviews on their own, if they require more shadowing, or if they should step back from the Bar Raiser pool at this time.

What “Shadowing” an Interview Means in Practice

During shadowing, the primary Bar Raiser interviewer leads the interview. The shadow Bar Raiser has a passive role and isn’t expected to actively contribute to the session. Instead, they observe and take notes. After the interview, the primary Bar Raiser fills out a scorecard. At this point, the shadow and primary interviewers can reflect on how the interview itself went without discussing the scorecards. During the hiring sync, the shadow Bar Raiser will share their feedback for the candidate.

How Do We Level Up Bar Raisers?

We have a session every few months with all Bar Raisers to share experiences, learnings, and interesting cases using anonymized data to see how we could further improve the process by iterating on it. Moreover, we continue to do cross-pairing among all Bar Raiser interviewers, which involves shadowing each other to learn from each other’s techniques.

Conclusion

We strongly believe that finding the right candidate is crucial for the future impact of our business and what that means for the products we’re creating for our users. Furthermore, we believe that Bar Raiser interviews play a significant role in this.

If you’re interested in finding out more about current roles at SoundCould, please visit SoundCloud jobs.

What's New with SoundCloud, October 2022

Mon, 17 Oct 2022 00:00:00 GMT

We recently made updates to our SoundCloud iOS and Android apps to address key user experience (UX) fixes according to user feedback.

We’ll be updating this blog regularly as we actively work to improve the SoundCloud experience for our users, and we’re so excited to share these improvements with everyone.

Here are our most recent updates across iOS and Android as of 4 October 2022.

Track Cell Size Reduction (iOS)

We made the size of tracks, playlists, artwork, and creators smaller so you can access more music with less scrolling.

Labels in the Navigation Tab Bar (iOS)

We updated the navigation and included labels, making it easier to understand the purpose of each menu item, in turn streamlining your experience.

Stream Is Now Feed (iOS)

Based on user feedback, we renamed Stream to Feed to make it more widely understood. Nothing in your stream has been removed or lost — you still have access to all the music discovery you know and love.

Smaller Page Titles (Android, iOS)

Thanks to labels in the navigation bar, we made page titles smaller to make the best use of space and ensure users they’re not missing out on any music. This increases content density in the first scroll of pages like Home.

New Playlist Header (Android, iOS)

Our updated and reduced-sized playlist header puts less space between you and your music, allowing you to access your playlist tracks — plus shuffle and play buttons — faster and easier.

More to Come

We’re excited to unveil future updates to SoundCloud, and we couldn’t do it without your support and feedback — stay tuned for what’s next!

Android Image Loading at SoundCloud

Thu, 13 Oct 2022 00:00:00 GMT

The SoundCloud Android app recently got a significant makeover: a “new year, new me” type of thing. Our remarkable design team and outstanding engineers worked together to revamp the way we display all kinds of images in the app, and I’m about to take you behind the scenes on this “glow-up” journey.

A Peek at the UI

First, here’s a look at the user interface (UI).

We went from the old design:

To the new design:

Below are some images you’ll come across while using the redesigned SoundCloud app.

Simple Track

This is a basic single image for tracks.

Playlist

This image is for a playlist. It has a background stack with colors picked from the playlist artwork.

Special Playlist

This image is for special playlists. It includes additional indicators, a blurred image stack, and a circular artist image.

These images look great, but as you might guess, they’re all backed by code, and the implementation wasn’t exactly straightforward.

“With great design comes great code!” —Someone’s uncle

From Design to Code Implementation

Want to know how we did it? This section will cover how we went from the design ideas to implementing them in code.

First off, all images now have rounded corners. This is thanks to ShapeableImageView, which supports rounded images.

Playlists show the main artwork and two images behind the main artwork, and each image has rounded corners. We accomplished this by creating an XML file containing a ConstraintLayout with three children of ShapeableImageView:

<ConstraintLayout>
    <ShapeableImageView id="stack2" />
    <ShapeableImageView id="stack1" />
    <ShapeableImageView id="playlistArtwork" />
<ConstraintLayout/>

We have some special types of playlists that can have additional overlays/blur or more images on top of the main artwork. For special playlists (see the Special Playlist section), we added another ShapeableImageView as an overlay to the ConstraintLayout with visibility="gone". If a playlist is “special,” the ShapeableImageViews visibility is changed to visible:

<ConstraintLayout>
    <ShapeableImageView id="stack2" />
    <ShapeableImageView id="stack1" />
    <ShapeableImageView id="playlistArtwork" />
    <ShapeableImageView id="playlistTypeIndicator" visibility="gone" />
<ConstraintLayout/>

For playlists, the primary colors of the main artwork determine the image stack colors. We used the [Android palette library][] to make this happen. It has a simple API that picks the primary colors from images, which we can use as background colors for the stack.

Additionally, images fade in when loaded. We use the PicassoDrawable, which crossfades the newly loaded bitmap with whatever image was previously loaded. To highlight the fade-in animation, the duration of the animation was set to 200 milliseconds.

Placeholders are displayed while the actual images are downloading, and com.squareup.picasso.Target provides a callback function:

fun onPrepareLoad(placeHolderDrawable: Drawable?)

onPrepareLoad enables us to add a gray color to the stacks and a default image while waiting for the main artwork to load.

Loading and Displaying Images with Picasso

We use Picasso to load images. Picasso also offers a simple API, picasso.load(url).into(target), to download and cache images.

The image cache is used throughout the whole app so that an image in the cache will never download twice. Picasso can download or load an image from the local disk (cache).

Picasso is extendable for custom image loading operations. For example, if image loading fails, we want to display an error image. There are a few different ways to do that with Picasso. At SoundCloud, we have a custom ImageView that implements Picasso.Target.

Every time we load an image into the ImageView, we animate (fade in) the new image bitmap:

class ArtworkImageView : Target {
    override fun onBitmapLoaded(bitmap: Bitmap, from: Picasso.LoadedFrom) {
        animateBitmap(bitmap)
    }

    override fun onBitmapFailed(e: Exception, errorDrawable: Drawable?) {
        animateFailureBitmap()
    }

    override fun onPrepareLoad(placeHolderDrawable: Drawable?) {
        showPlaceholderBitmap()
    }
}

After implementing the redesign, the images looked fantastic, the design team approved, and the product managers were happy. But it came with a catch: Images were loading slower than before. The slow performance was visible while scrolling, and images flickered when moving between screens. The redesign had impacted image performance.

The following section will cover how we fixed the image loading performance.

Slow Image Performance Investigation and Fixes

We started debugging and discovered not a single culprit, but rather a bunch of issues that were contributing to the performance problem. We also investigated the app’s GPU overdrawing, which I’ll elaborate on later. The next sections outline the specific issues we encountered and how we fixed them.

Issue #1 — Palette LruCache

To generate the image stack colors, we use the Android palette library. To save time on regenerating the palette, we used LruCache<Bitmap, Palette> to store the bitmap and palette key-value pair for faster on-demand loading.

So every time we loaded an image, the code below executed:

if (isInCache(bitmap)) {
    return getTheCachedPaletteFor(bitmap)
} else {
    val palette = generatePaletteFrom(bitmap)
    storePalette(bitmap, palette)

    return palette
}

Using a Bitmap as a key, we retrieved the palette from LruCache. However, using bitmaps as keys to the cache caused a performance issue. Given that bitmaps are large objects, it’s best to never use them as keys to a Cache, Map, or any other data structure. This is because comparing bitmaps is time-consuming.

Solution #1

The fix ended up being simple. We switched from LruCache<Bitmap, Palette> to LruCache<String, Palette>. The String is the URL of the loaded bitmap, and comparing strings is quick and consumes less memory.

The code is below:

if (isInCache(url)) {
    return getTheCachedPalette(url)
} else {
    val palette = generatePaletteFrom(bitmap)
    storePalette(url, palette)

    return palette
}

Issue #2 — Palette Color Generation

To generate the palette colors from bitmaps, we used the code below:

Palette.from(bitmap).generate()

The code is simple: It provides the bitmap and generates the primary colors for that bitmap. But palette generation takes time. It works by checking each pixel in a bitmap to find the most used colors. For example, if an image contains a lot of red, then the palette will contain the red color.

The bitmaps we use to generate the palette have a size of 500×500, which is large and slows the palette generation.

Solution #2

A faster way to generate the palette is to first scale down the bitmap and then generate the palette.

The palette has a function, resizeBitmapArea(size), that scales the bitmap down, so we went from 500×500 to 15×15. We also use maximumColorCount(colors) to find fewer colors, as in our implementation, we only needed the dominant color. By default, the palette finds 16 different colors.

The code below generates palettes faster, but the tradeoff is primary color accuracy, which is ultimately fine; shades of red are still red:

Palette.from(bitmap)
    .resizeBitmapArea(225) // Scales down the bitmap area to a 15×15.
    .maximumColorCount(6)
    .generate()

Issue #3 — Animating from the Placeholder to the Real Image

When loading images the first time, we displayed a placeholder and then displayed the actual downloaded image with a fade-in animation. Once the image was cached and the user scrolled through the RecyclerView, we still saw the fading in and out from the placeholder to the actual image, even though the image was cached.

Solution #3

To address this, we dropped animations for cached images. Luckily, Picasso offers the from enum, Picasso.LoadedFrom, together with the loaded bitmap. Picasso.LoadedFrom tells us where the bitmap was loaded from:

enum LoadedFrom {
    MEMORY
    DISK
    NETWORK
}

If the image was loaded from NETWORK, we animate the new image; otherwise, we display the image without an animation:

ArtworkImageView : TrackArtwork {

	override fun onBitmapLoaded(bitmap: Bitmap, from: Picasso.LoadedFrom) {
            if(from == Picasso.LoadedFrom.NETWORK) {
                animateBitmap(bitmap) // Animate from placeholder to real image.
            } else {
                showBitmapWithoutAnimation(bitmap) // Directly display image from cache.
            }
        }
}

Another minor tweak was that we made the fade-in animation faster — from 200 ms to 60 ms. 200 ms felt like we were showing off our fancy animations rather than benefitting the user by displaying the image quickly.

Issue #4 — Image Preloading in RecyclerView

Many screens in the SoundCloud app consist of lists of tracks or playlists, which means the user will scroll through the list to play a track. While the user scrolls, the new images are loaded and cached. Downloading new images takes time, so we thought it’d make sense to download and cache a few of the upcoming images in a list before the images of tracks/playlists are even displayed on the screen.

Solution #4

We preloaded the three subsequent images in RecyclerViews. If you use Glide, you’ll know it supports preloading images out of the box.

Since we use Picasso for image loading, we implemented image preloading based on Glide’s example.

When preloading items, we needed to consider the following:

How many not-yet visible items do we want to preload? In other words, how many items did we want to preload? These items are always not visible until the user scrolls down to them.
How many items are left to preload? We needed to check if we were about to reach the end of the RecyclerView; for example, if we were always preloading the next five items and reaching the last bottom four items, we needed to stop preloading toward the end of the RecyclerView.
Make sure we don’t preload items twice.
Only load items that aren’t yet visible on the screen. If we hardcode the preload size to 5 items but show 10 items, we end up preloading items that were already loaded.

Below is the ItemPreLoader class, which handles all the points mentioned above:

const val DEFAULT_PRELOAD_ITEMS_SIZE = 3

/***
 * The `ItemPreLoader` calculates which items should be preloaded.
 *
 * Items can only be preloaded when the conditions below are met:
 * 1. `preloadEnabled` is `true`, meaning that `PreloadScrollListener` has enabled preloading.
 * 2. `lastVisiblePosition` is larger than `0`. The first few items are already preloaded by the adapter, and if we try to load more items immediately, it could overwhelm the app's memory.
 * 3. `lastVisiblePosition` is smaller than `items.size`. We shouldn’t continue preloading elements outside the bounds of the list (`IndexOutOfBoundsException`).
 * 4. `startIndex` is larger than `lastPreloadedPosition`. Already preloaded items must not be loaded again.
 * 5. The list isn’t exhausted; there are more items. Same as 2.
 * 6. The last visible item’s `ViewType` must be the same as the `ViewType` of the item we’re currently preloading. Otherwise, we'll cause a `ClassCastException` if the `ViewTypes` are different. `getItemViewType(lastVisiblePosition) == getItemViewType(realIndex)` ensures we don't cause a `ClassCastException`.
 *
 * If the above conditions are met, `ItemPreLoader` slices the items list from the `adapterPosition`(`startIndex`) to the `lastIndex`(`startIndex + preloadItemsSize`) and calls `viewHolder.preloadNextItem` on those items.
 */
class ItemPreLoader(
    val preloadItemsSize: Int = DEFAULT_PRELOAD_ITEMS_SIZE
) {
    var preloadEnabled = false

    internal var adapterPosition = 0
    private var lastPreloadedPosition = -1

    fun <T> preLoadItems(
        viewHolder: ScViewHolder<T>,
        items: List<T>,
        position: Int,
        getItemViewType: (Int) -> Int
    ) {
        val lastVisiblePosition = max(adapterPosition, position)

        /*
            - `preloadEnabled` must be `true` to continue preloading items.
            - `lastVisiblePosition` can be set to `-1` if there are no items in the adapter. In this case, we don't want to preload items.
            - `lastVisiblePosition` can be set to `0`. We don't want to start preloading immediately when creating an adapter,
                and the adapter already loads the next item(s) to be shown.
         */
        if (!preloadEnabled || lastVisiblePosition <= 0) {
            return
        }

        val startIndex = if (preloadItemsSize < lastVisiblePosition) {
            lastVisiblePosition + 1
        } else {
            preloadItemsSize
        }

        val lastPreloadIndex = (startIndex + preloadItemsSize - 1).coerceAtMost(items.lastIndex)

        if (startIndex in lastPreloadedPosition until lastPreloadIndex) {
            lastPreloadedPosition = lastPreloadIndex
            var realIndex = startIndex

            items.slice(startIndex..lastPreloadIndex).forEach {
                if (getItemViewType(lastVisiblePosition) == getItemViewType(realIndex)) {
                    viewHolder.preloadNextItem(it)
                }

                realIndex++
            }
        }
    }
}

We also needed to set a custom OnScrollListener to the RecyclerView. The OnScrollListener:

Enables item preloading when the user scrolls down.
Disables item preloading when the list is idle or the user scrolls up.
Finds the last visible item position in RecyclerView.

class PreloadScrollListener(val itemPreLoader: ItemPreLoader) : RecyclerView.OnScrollListener() {

    /**
     * The scroll state can only have the following three values:
     * 1. `SCROLL_STATE_IDLE` is when the user isn't scrolling, and it enables image preloading.
     * 2. `SCROLL_STATE_DRAGGING` is when the user is scrolling with their finger on the screen, and it enables image preloading.
     * 3. `SCROLL_STATE_SETTLING` is when the user fling scrolls, and it disables image preloading.
     */
    override fun onScrollStateChanged(recyclerView: RecyclerView, newState: Int) {
        super.onScrollStateChanged(recyclerView, newState)

        when (newState) {
            SCROLL_STATE_IDLE, SCROLL_STATE_DRAGGING -> itemPreLoader.preloadEnabled = true
            SCROLL_STATE_SETTLING -> itemPreLoader.preloadEnabled = false
        }
    }

    /***
     * There's a special case where the user scrolls upward. In this case, image preloading is disabled.
     *
     * The `PreloadScrollListener` also sets the `ItemPreLoader.adapterPosition`.
     * The `LastVisibleAdapterPositionFinder` class grabs the `LayoutManager` from the `RecyclerView`, and from the `LayoutManager`, we find the last visible item position.
     */
    override fun onScrolled(recyclerView: RecyclerView, dx: Int, dy: Int) {
        super.onScrolled(recyclerView, dx, dy)

        if (dy < 0) {
            itemPreLoader.preloadEnabled = false
        }

        val position = findLastVisiblePosition(recyclerView)
        itemPreLoader.adapterPosition = position
    }
}

Finally, here’s the code to find the last visible item in the RecyclerView:

fun findLastVisiblePosition(recyclerView: RecyclerView): Int {
    return when (recyclerView.layoutManager) {
        is LinearLayoutManager -> getLastVisiblePositionFrom(recyclerView.layoutManager as LinearLayoutManager)
        else -> -1
    }
}

fun getLastVisiblePositionFrom(manager: LinearLayoutManager): Int {
    return manager.findLastVisibleItemPosition()
}

GPU Overdrawing

GPU overdrawing is when an app draws the same pixel more than once within the same frame. As a result, GPU overdraw visualization shows where an app might be doing more rendering work than necessary.

We caused a high overdraw severity because we were rendering/drawing three ImageViews on top of each other in the stacked ImageViews when displaying playlists. This is something we shouldn’t do.

The Issue

We draw three full images on top of each other, even though only 10 percent of the stacked images are visible. With stacks, 90 percent of an image isn’t visible, but it’s still drawn, which affects the frame rate.

To fix overdrawing, we would’ve needed to add a lot of custom complex drawing logic, which would take time to build, debug, and maintain. So the tradeoff here was unlikely to be worth the upfront effort and maintenance difficulties.

We did build the custom drawing logic, but it was too complex to maintain, and there was no performance benefit.

The main ideas were:

Flatten the stacked ImageViews into one ImageView.
Draw only the visible areas of bitmaps.
Faster image drawing.

The code below draws three bitmaps on a custom view. Only the bitmaps’ visible areas are drawn, so as to avoid GPU overdrawing. Since only a subset (10 percent) of the bitmaps is drawn, drawing is now faster:

StackedImageView: View() {

    val visibleBitmapOffset = 40f

    override fun draw(canvas: Canvas) {
	// Show the bitmap area calculations.
	val bounds = RectF(0, 0, width, height)

	val paint = Paint()
       	paint.xfermode = PorterDuffXfermode(PorterDuff.Mode.DST_ATOP)

        canvas.drawBitmap(topStackBitmap, null, topArea(bounds), paint)
        canvas.drawBitmap(middleStackBitmap, null, middleArea(bounds), paint)
        canvas.drawBitmap(bottomStackBitmap, null, bottomArea(bounds), paint)
    }
}

private fun topArea(bounds: RectF) = bounds.copy().apply {
        right -= visibleBitmapOffset * 2
        bottom -= visibleBitmapOffset * 2
    }

    private fun middleArea(bounds: RectF) = bounds.copy().apply {
        top += visibleBitmapOffset
        right -= visibleBitmapOffset
        left += visibleBitmapOffset
        bottom -= visibleBitmapOffset
    }

    private fun bottomArea(bounds: RectF) = bounds.copy().apply {
        top += visibleBitmapOffset * 2
        left += visibleBitmapOffset * 2
    }

The code above calculates the bounds (area) of each bitmap. The bounds are the area into which the bitmap will be drawn. The visibleBitmapOffset defines how many pixels the bounds will be moved/offset.

In the top stack bitmap area, the bitmap is displaced by visibleBitmapOffset*2 pixels toward the top-left corner.
In the middle stack bitmap area, all sides are moved by visibleBitmapOffset/2 pixels, placing the bitmap in the middle of the view.
In the bottom stack bitmap area, the bitmap is displaced by visibleBitmapOffset*2 pixels toward the bottom-right corner.

To draw only the visible bitmap areas and avoid overdrawing, we use paint.xfermode = PorterDuffXfermode(PorterDuff.Mode.DST_ATOP).

PorterDuff.Mode.DST_ATOP draws only the outside area of the following image. So if there is already something drawn in the view, then DST_ATOP only draws the outer part of what we’re drawing now. Since we already drew the top bitmap, only the middle bitmap’s outer part is drawn. It’s the same with the bottom bitmap; only the outer section is drawn.

The section above outlines the basics of how to draw three images without overdrawing. Complications arise when the images have round corners; the stacks can be solid colors; and there can be more images on top of the stack, fading in each stack of bitmaps and drawing rounded borders. Implementing all of that led to a great deal of drawing code, which was hard to maintain. Thus, it wasn’t included in the project.

If you want to learn more about computer graphics drawing, Fundamentals of Computer Graphics is a great book.

Conclusion

Redesigning an entire app takes a lot of time — from product managers coming up with the product specifications, whiteboarding sessions, and creating wireframes, to UI and UX designers drawing the final design. Finally, the engineers have to implement the new design and rewrite UI components, and usually, all this work is done under pressure with a set deadline.

We all like to go fast and get a redesign done, but as a result, we end up rushing and forgetting the important stuff like properly testing the new UI to see if it works, and checking to ensure the performance and frame rate are acceptable. Testing and measuring performance is the responsibility of the engineers, so it’s of the utmost importance to always test and measure the performance of your new UI.

With proper performance measuring and testing, we were able to identify and fix image UI issues we encountered. And although it took longer, SoundCloud ended up with a more polished product.

SoundCloud Echo: Next-Level Humane Registry with Backstage

Mon, 19 Sep 2022 00:00:00 GMT

We’ve seen considerable adoption of our internal humane registry, Services Directory, since we sanitized ownership definition and introduced an easier-to-use user interface (UI) to our development infrastructure. Namely, managing backend feature toggles (or as we call them, rollouts) and visualizing interservice dependencies has been helpful in bringing engineering managers, product managers, and engineers closer — not only in their day-to-day work, but also when communicating with other teams in the organization.

With higher engagement, we also got feedback and tons of ideas for extending Services Directory and making it even more useful.

When talking to engineers at other tech companies, and also internally to our recent hires, we recurringly heard “organizing the software inventory” to be a repeated problem companies solved in different shapes and forms, just like we did. At approximately the same time we were investing in Services Directory, the industry saw the emergence of open source alternatives for building developer portals. Our team analyzed options, weighed the cost of maintaining our own Services Directory, and finally decided to migrate (and contribute) to externally developed software.

In this post, we’ll describe how we took Services Directory to the next level and migrated our internal system to the open source Backstage developer portal technology, and how we’re now providing a lot more capabilities and empowering our engineering team, in turn speeding up software delivery and engineering effectiveness.

Adoption Path

Like most pieces of work that influence other teams’ workflows and critical-path tooling, we wrote an RFC document to collect thoughts and establish a migration path out of Services Directory and into Backstage.

The key idea and deliverable was to provide a narrow but high-value use case, and only later expand to new features and start evangelizing the platform. As mentioned in our last post on the topic, we wanted Backstage to replace Services Directory in its core value proposition:

The registry must enable finding out which systems a team owns and who the owners of given systems are.

Naming Is Hard

As we set out with a plan to build a prototype for what Backstage could look like at SoundCloud, we also had to make a naming decision. Although Backstage is a compelling name, SoundCloud — being an audio platform — already had at least two other things with that name (including this very blog). We didn’t want to make things even more confusing, so after long deliberation, we settled on SoundCloud Echo for the codename of our Backstage instance. Not only does “echo” have a relation to sound in general, but it’s also one of the very first commands engineers write on their path to software development:

$ echo 'Hello World!'

Bringing In Some Glue

We went through a few steps to get to what we considered our adoption milestone.

Getting started with Backstage meant making our platform team acquainted with how the platform looks and works, along with becoming familiar with its community. With access to the maintainers and other project contributors, we also identified opportunities to learn the long-term project roadmap and unblock ourselves if we ran into questions. We were quickly able to identify which bits would smoothly connect to our existing infrastructure and which bits would require some “glue.”

Database

The first challenge was a fundamental one: the database.

As described in a previous post, the golden paths at SoundCloud evolved around MySQL as our choice for relational database management systems (RDBMS). We have experts within the company that have plenty of knowledge on how to provision, query, and maintain MySQL clusters. On top of that, we’ve built plenty of supporting infrastructure for this technology.

What we found, though, is that Backstage can work out of the box with PostgreSQL. Even though there’s interest from the open source community in providing Backstage with support for running on MySQL, it remains unavailable. We ran an internal proof of concept to estimate the work it would take for us to implement that support, but we deemed going forward too risky, especially given that we would’ve had to play catch with the upstream project.

The alternative was to rely on Cloud resources. Instead of provisioning a PostgreSQL cluster ourselves, we set up a Postgres database node in Google’s Cloud SQL, through a proxy deployed as a sidecar container in its Kubernetes pod.

The Cloud SQL Auth proxy works by having a local client running in the local environment. Our instance communicates with the proxy through the standard database protocol. The Cloud SQL Auth proxy uses a secure tunnel to communicate with its companion process running on the server.

This way, our app can use a PostgreSQL database, and we get the benefits of managed resources in the Cloud — for example, automatic backups, data replication, telemetry, etc. Most importantly, engineers operating the project have a less steep learning curve to the PostgreSQL setup. When, in the future, Backstage supports MySQL as a database engine, we can then decide to move over.

Catalog Descriptors

So we have a storage for the catalog, but how do we get the data in?

As we saw in our previous post, Services Directory already relied on a background worker job indexing “manifest files,” which were essentially JSON files with a predefined schema. We were positioned especially well for the migration to Backstage because we had done the work to standardize and sanitize all these files across SoundCloud repositories, which meant we could make some assumptions about their shape when ingesting them into our new PostgreSQL database.

Backstage’s catalog feature is built around the concept of metadata YAML files, so our job here was to write a parser that could translate our JSON manifests into Backstage’s entity metadata objects by hooking into the entity processing loop.

It was quite pleasing to see how quickly we got to a working prototype of Backstage that indexes most of SoundCloud’s software components.

This was also an enabler for our platform team: With real-life data coming in, we could speed up our exploration and experimentation and identify special-case scenarios even before we rolled Backstage out to the rest of our engineering organization. Just like with human relationships, with developer tooling, sometimes the first impression is the only impression, and we aimed to push that bar very high to drive adoption among teams.

Months later, when we had already established Backstage as a replacement for Services Directory, we ended up substituting our manifest files with Backstage catalog descriptor files. In addition to making the custom-made parser obsolete, using the format from the open source project meant we got access to plenty of new features while still reducing our internal maintenance burden: No custom internal format means no learning curve and no work to keep all files following the schemas.

That change was transparent to our engineers — our platform team ran a bulk change automating the translation from JSON to YAML and pushing these files to all repositories in the organization. That’s yet another learning for successful adoption of a tool on the engineering toolbelt: Users love new features, especially when they have to do no work to get them. By hooking in the entity processing pipeline, we were capable of stitching in several quality-of-life improvements for catalog users. Soon enough, we had most of our APIs cataloged, we had public repositories, and we could also support indexing of monorepos.

Backend Integrations

With the catalog up and running, we moved our attention to external integrations. Services Directory catalog’s main supporting feature was to provide a friendlier UI for engineers to interact with our backend feature toggles.

We took that as an opportunity to get acquainted with the different system interactions within: how we could instruct Echo’s backend to communicate with an already-existing API, and how easy it would be to design and code a frontend plugin for that integration.

The fact that Backstage uses a well-thought-out and documented design system sped up our efforts considerably. On top of that, having a React codebase meant we could lean on both the existing plethora of open source UI libraries and components, and the extensive knowledge of our internal team of web engineers.

For the first time, “word was out,” and we started seeing excitement within the organization for what potential Echo/Backstage would unlock.

Maintenance, Ownership, Updates

Naturally, at SoundCloud, each team is able to operate on its own and make the best decisions for themselves. Our platform teams provide guardrails and support common use cases that we consider our golden paths, but we don’t want to mandate technology choices for everyone else. Backstage fits perfectly into that philosophy, and its plugin mechanism serves as a way for our engineers to build value together.

On top of that, we recognize how some teams are way better suited than our platform team to drive the vision for their own software ownership. In that sense, the plugin for interacting with our build system is provided by the team that owns the build systems, the plugin that shows active or past A/B tests is owned by the “experimentations” team, etc.

Our platform team is constrained in people, time, and resources, so the co-ownership model works very well for us. With more team autonomy, we’ve seen fast adoption, growth, and also interest in contributing to Echo across the engineering organization. Meanwhile, our platform team continues to be responsible for the infrastructure of, running upgrades on, and keeping a vision of the longer-term plans for Backstage and Echo at SoundCloud — along with its limitations and opportunities.

Powering Up Engineers

Echo gets more powerful every time we deploy an upgrade to the underlying Backstage infrastructure, but over time, we’ve collected and implemented several plugins of our own that had a positive impact on our engineering team’s day to day — whether that’s by bridging communication or helping service operations. Here are a few examples of “power-ups” we’ve built to aid our engineering organization.

Entity Augmentations

Different to most entities in the catalog, Groups and Users are sourced from external “entity providers.” In our current implementation, that means GitHub.

The entity provider modeling can differ from the Backstage modeling, so we allow entity augmentation by merging the source-of-truth data (GitHub) with manually curated fields from the YAML entity envelope. In most cases, these will be annotations:

apiVersion: soundcloud.com/v1alpha1
kind: GroupAugmentation
metadata:
  name: beep-team
  annotations:
    soundcloud.com/team-readme: confluence:139924197
    soundcloud.com/team-alerts-email: team-email-for-alerts@sc.com

During the next catalog-refresh cycle, Echo will fetch available augmentations and merge them with existing entries in its catalog. The final shapes of the merged entities become available through Echo’s UI and its API.

This added feature allows further integration of Echo’s entities to reference into external systems.

In the example above, we point a page in our Confluence instance to render a “team README” with basic information about the team, contacts, ways of working, etc. The interesting aspect is that the content doesn’t need to be moved or replicated within Echo; the data source continues to be Confluence, where our non-technical audience is comfortable, but Echo, home of engineering, can read, display, and include information in its search index.

Another existing integration is with Prometheus’ Alertmanager. Since Echo exposes its catalog through a queryable API, annotations in the entity augmentation can inform our Alertmanager deployment about notification receivers — these can be email (as exemplified above), Slack, PagerDuty, etc.

The benefit of using entity augmentations is that of data centralization: When, unavoidably, a company reorganization happens and a team gets created or renamed, the team manager updates a single file and our automation kicks in for the dependent systems. In the Alertmanager use case, this means no manual interaction to configure receivers. Similar workflows can be established for automating the creation of pull requests in selected repositories, or even infrastructure provisioning.

Daily Ritual Support

As the engineering home page, Echo also supports recurring tasks individual contributors run through. Within the teams, that can be checking open pull requests across all of their ownership domain, or even selecting a team member at random to take notes in a retrospective meeting. These capabilities can be especially useful when we consider the interactions that happen across teams as well.

As an example project workflow, an engineer can visit action item lists from previous incident review meetings and check whether they’re assigned to anyone, write and publish an RFC outlining a possible architectural change as a solution, and finally announce the system migration to the rest of the engineering organization so dependent teams can prepare accordingly.

All of these are supported by Echo in one way or another, and the long-term vision is to continue to identify intra- and inter-team interactions to standardize solutions to speed up the engineering practice end to end.

Domain-Specific Technical Health

Especially important as we consolidate our golden paths and internal tech radar, Echo also helps increase visibility of our team’s ongoing operations. The data set is an aggregation of sources like Prometheus, Zoekt, GitHub, CodeScene, and others. Through the modeling of the catalog, we then further slice our software components across our business domains and establish a maturity grade as to what we consider best practices or technical debt. We’ve also been exploring building a “migration tracker” in a similar fashion as to what Spotify describes in this post.

Put in a service perspective, these insights can be useful for individual contributors. Now, take that to the organizational level: Through historical data and Echo’s visualization-aggregating services, teams, and domains, its dataset is also informational to engineering managers preparing for their next planning meeting, and to leaders looking to build an ROI argument to select a domain to work on accumulated technical debt.

Down the Line

We’ve seen more engagement within teams and toward our platform team as more engineers got acquainted with Echo and brought it into their daily workflows. Often, conversations bubble around feature requests and plugin ideas.

These proposals feed back into our own internal roadmap and help us define KPIs to measure the success of the platform. From a product perspective, we’ve been tracking page views and DAU/MAU to establish where to invest or what to defund.

For the future, we envision being able to bring usage information back to the teams themselves: Imagine, for example, being able to inform teams of opportunities to provide more extensive documentation, or even when knowledge siloing exists.

Other opportunities lie on getting Echo closer to where our users are: For example, a lot of daily work happens on Slack. Individuals asking questions on how services work, checking in on dependent teams, incident management and response — all of that, and more, happens in Slack. Instead of engineers coming to Echo, why not make Echo conversational?

Similar potential exists in making information contextual by other means: GitHub comments, Jira tickets, Confluence pages — you name it. The fact information is available is the important part, and where it lives shouldn’t matter. Part of our platform’s team mission is to make that information accessible, discoverable, and easy to digest, whether that’s for a human reader or to build automation upon.

Learning as We Go

It’s been two years, and we’re still gaining confidence and working on bringing traction to Backstage at SoundCloud. We’ve gone through some operational hoops to move from Services Directory into SoundCloud Echo, and we’ve objectively provided a better development experience to our engineers, while also opening doors to plenty of potential the platform provides.

Advocacy and Education

As envisioned, relying on an open source solution brought us a lot of value “for free,” but we still had to invest a considerable amount of work to make it match our team’s needs and wishes. Developer productivity is a neverending road, and although we see lots of aptitude for the future, SoundCloud (and every Backstage adopter) makes an operational funding decision when picking Backstage to power its developer portal. Nowadays, Echo is a central piece of our infrastructure, and its maintenance means responsibility for the platform team.

Onboarding the engineering group to Backstage is more than telling engineers about it. The investment from the business is quite demanding, and to make it “the homepage of engineering” also means cultivating it as part of the culture for engineering stakeholders (EMs, PMs, Directors). Training and advocating is a work-heavy task and demands organizational efforts, especially when it’s cross-team and cross-discipline and involves non-technical folks. A big part of the work relies on “vision selling,” and that might include reshaping some of the existing culture, introducing practices like docs-as-code, or asking teams to change their ways of working, for instance, to keep their ownership listing clear and sanitized, or to consult some Backstage-provided dashboard during standup meetings.

As exemplified earlier, in many cases we choose not to change workflows, but rather to build system integrations and provide new/alternative UIs, so individuals continue writing and consuming documents where they’re comfortable. This can be counterproductive: Tenured folks stick to “the old ways,” while new team members adopt Backstage’s way, and due to information fragmentation, the platform team starts seeing a wave of “what option should I use?” questions. It’s the responsibility of the platform team to work with the rest of engineering to decide the best moment to decommission features — even if sometimes that means deciding to roll back on a Backstage feature.

Similarly, pushing the Backstage catalog model onto teams, albeit simple, still demands educational efforts. Pair that with our previously existing catalog model, and some bridging work had to happen. Components, systems, domains; split and combined monorepos; first-party and third-party dependency listing — all of these are catalog-modeling questions whose answer is “it depends,” and for the most part, whose intricacies users shouldn’t care about.

After two years, we’re still in the early days of the full Backstage model adoption, and even then, we might decide not to pursue it completely or to introduce our own constructs. Likewise, we haven’t yet adopted the full spectrum of Backstage core features: Software Templates, although super powerful, are still something we haven’t invested in. Given the maintenance cost, we’re not yet ready to support them, nor are we prepared for the cognitive load it might introduce to the rest of our engineering team.

Contributors

As we evaluated, experimented, learned, and adopted Backstage internally, we also found opportunities to play a part in the wider community. That means engaging with the maintainers and adopters at the Discord chatroom, presenting at the monthly Backstage meetups, and contributing code.

The plugins SoundCloud open sourced revolve around areas like continuous delivery, observability, and technical health assessment, and there are more to come. Summed up, we see more than 1,000 downloads per week.

WINE Workshop

Tue, 02 Aug 2022 00:00:00 GMT

SoundCloud has a variety of resource groups, and one such group is called Women in Engineering, or WINE. The mission statement of the group — which is open to people of all genders — is to work “on initiatives to attract and ensure an environment for women to thrive within Engineering at SoundCloud and in the industry.”

This is accomplished in a variety of ways — from hosting monthly events like meetings and lunches, to scheduling regular internal knowledge-sharing sessions. And in 2021, one of the group’s initiatives was to support woman and non-binary engineers at the company in writing and publishing posts on our engineering blog.

Making a Plan

To get started on this initiative, I met with Aleksandra Gavrilovska, one of our backend engineers and former iOS Engineering Manager, to talk about what we wanted to accomplish and how we’d do it. But before we could proceed, it was important to first figure out the barriers engineers were facing. So we created a short survey and asked members of WINE to take it.

We ended up with answers from 20 individuals, or 37 percent of the members of WINE. Of those 20 people, 7 of them said they would likely publish a post in the following year.

When we looked at the reasons people weren’t writing posts, the top answers were:

I don’t know what to write about.
I don’t have enough time.
I don’t think my topic is interesting enough.

We also asked what could motivate people to write more, and the most common answer was having someone help them choose a topic.

Next, Aleksandra synced with Tiffany Conroy, SoundCloud’s former VP of Engineering, who agreed to host a workshop for brainstorming ideas.

Hosting a Workshop

The workshop took place in March of 2021, and there were 23 participants. Although the goal was to increase the number of blog posts, the workshop was also geared toward WINE members who were interested in doing demos — which is a bi-weekly event on Friday afternoons when engineers at SoundCloud share what they are or were working on — or presenting their work in other public ways, like at conferences or meetups.

In total, the workshop was approximately 90 minutes, and it included interactive breakout sessions where participants talked with one another, formulated their ideas, and expanded on them. At the end of the workshop, everyone had at least one topic they could confidently write a blog post about, along with an outline of what it should entail.

Looking at the Results

We looked at blog posts in the time frame of 2016 to 2020, and saw that there were 64 total posts published. Of those 64, only 6 of them were written by women — less than 10 percent. However, after this workshop, we published 6 posts by women.

Here’s a breakdown of the statistics for 2021:

Total Posts — 14 Total Authors — 15 Women Authors — 5 Posts by Women Authors — 6

That means that 43 percent of the posts published in 2021 were written or co-written by a member of WINE. Ideally, we want that number to be at least 50 percent, but it’s still a huge improvement over the 10 percent from the five-year period before.

What’s Next

It’s safe to say we were successful in what we set out to do, but our next challenge to tackle is how to keep this momentum. As new people are always joining SoundCloud’s engineering organization, repeating this workshop would be a good start, as it was clearly a successful first step at getting more diverse voices to contribute to our blog.

PR Templates for Effective Pull Requests

Tue, 12 Jul 2022 00:00:00 GMT

The first step to any code review is understanding what the code does and why. Although the author is by far the most qualified person to answer these questions, all too often, the reviewer is left to answer them on their own. With no pull request (PR) description, reviewers need to assess changes with only the title and, at best, some kind of tracker ticket for context — and let’s be real, the ticket is probably empty too. The review process essentially becomes a big game of telephone. The truth is, empty pull request descriptions slow down teams.

When I first joined SoundCloud, that’s what PRs in many of our greener projects looked like. Every time I reviewed a PR, I had to meet with the author in real time to get answers about what their code did or why they made certain decisions. Often, after talking to them, I’d realize that all the assumptions I’d made on my own were totally incorrect. In turn, I found I was starting to avoid PR reviews because there was so much overhead in getting to the review.

Before adding PR templates

To combat this, I recommended a PR template as a way to make it easier to orient myself when reviewing. Not wanting to introduce too many changes too quickly, we initially added the template to just one repository. There was also no rule against empty PRs; we merely added the template and left it up to the individual to decide whether or not to fill it out. Lo and behold, with just this simple suggestion, suddenly all of our PRs were full of context, explanation, and discussion!

After adding PR templates

By reducing the PR writing process to a series of fill-in-the-blank steps, the PR transformed from a throwaway document to a valuable asset for knowledge sharing across teams that will outlive its authors. PR review instantly became easier, and the feedback on PRs became more relevant. Our experiment in templates was so successful that other teams began asking for our template to use in their repositories. I even had friends outside of work curious what kind of template we were using, so today I’m sharing the template that got us there.

Anatomy of a Good PR Template

## Scope

<!-- Brief description of WHAT you’re doing and WHY. -->

[closes TICKET-###](https://link-to-your-ticket)

The scope is the big picture of what and why. If your PR were an essay, the scope would be the introduction. Scope eases reviewers into your PR, so they spend less time orienting themselves to the ticket and more time reviewing effectively.

## Implementation

With @pair

<!--

Some description of HOW you achieved it. Perhaps give a high level description of the program flow. Did you need to refactor something? What tradeoffs did you take? Are there things in here which you’d particularly like people to pay close attention to?

-->

This section gives a human-readable breakdown of the changes in the PR. It might not be immediately obvious to reviewers what every change in the git diff does, so a summary of the technical implementation gives reviewers a context to place those individual changes into.

I like to leave a spot to call out pairing in this section. It positions pairing as a core value for the team, and it comes in handy when the PR surfaces in a git blame; the more names on the PR, the higher the chance that one of them has an answer to questions.

## Screenshots

|         | before | after |
| ------- | ------ | ----- |
| desktop |        |       |
| mobile  |        |       |

A picture is worth a thousand words, so definitely include a screenshot section if your team is responsible for any kind of UI component. It’s by far the most direct way to show the changes introduced in your PR. It also provides the fringe benefit of forcing developers to QA their own work. Including a section for web and mobile screenshots ensures that responsive designs are maintained.

Screenshots might make less sense for backend APIs, but including class or flow diagrams is a great alternative to visually demonstrate your changes.

## How to Test

<!--

A straightforward scenario of how to test your changes could help colleagues that are not familiar with the part of the code that you are changing but want to see it in action. This section can include a description or step-by-step instructions of how to get to the state of v2 that your change affects.

A "How To Test" section can look something like this:

- Sign in with a user with tracks
- Activate `show_awesome_cat_gifs` feature (add `?feature.show_awesome_cat_gifs=1` to your URL)
- You should see a GIF with cats dancing

-->

There’s no better way to judge the success of a PR than actually running the code and seeing it work. A straightforward scenario of how to test your changes will help your reviewers see your change in action.

## Emoji Guide

I like to include an emoji guide for reviewers in my PR templates, because it enforces code review standards on every PR. Color coding review comments with emojis helps keep feedback organized, disambiguates non-blocking and blocking feedback, and encourages positive feedback as a crucial part of the review process. 😃

The Template

GitHub makes installing templates extremely convenient. Just copy the template into pull_request_template.md (or any of the files listed here) and commit it to your main branch. From then on, you’ll see your PR template populate your new PRs automatically.

<!--

Please use the content below as a template for your pull request.
Feel free to remove sections which do not make sense.

-->

## Scope

<!-- Brief description of WHAT you’re doing and WHY. -->

[closes TICKET-###](https://link-to-your-ticket)

## Implementation

With @pair

<!--

Some description of HOW you achieved it. Perhaps give a high level description of the program flow. Did you need to refactor something? What tradeoffs did you take? Are there things in here which you’d particularly like people to pay close attention to?

-->

## Screenshots

|         | before | after |
| ------- | ------ | ----- |
| desktop |        |       |
| mobile  |        |       |

## How to Test

<!--

A straightforward scenario of how to test your changes could help colleagues that are not familiar with the part of the code that you are changing but want to see it in action. This section can include a description or step-by-step instructions of how to get to the state of v2 that your change affects.

A "How To Test" section can look something like this:

- Sign in with a user with tracks
- Activate `show_awesome_cat_gifs` feature (add `?feature.show_awesome_cat_gifs=1` to your URL)
- You should see a GIF with cats dancing

-->

## Emoji Guide

**For reviewers: Emojis can be added to comments to call out blocking versus non-blocking feedback.**

E.g: Praise, minor suggestions, or clarifying questions that don’t block merging the PR.

> 🟢 Nice refactor!

> 🟡 Why was the default value removed?

E.g: Blocking feedback must be addressed before merging.

> 🔴 This change will break something important

| | | |
| --- | --- | --- |
| Blocking | 🔴 ❌ 🚨 | RED |
| Non-blocking | 🟡 💡 🤔 💭 | Yellow, thinking, etc |
| Praise | 🟢 💚 😍 👍 🙌 | Green, hearts, positive emojis, etc |

Conclusion

Adding a PR template is a quick and easy way to improve PR hygiene on your teams. As a reviewer, I enjoy a lower barrier of entry in my reviews. As a code author, the reviews I receive are also more useful and productive as a result of including this template. I strongly recommend using a PR template, and I’d definitely consider adding a PR template for any new codebases in the future.

Learning Scala at SoundCloud

Tue, 19 Apr 2022 00:00:00 GMT

I’m a backend developer who worked extensively writing code in Golang before joining SoundCloud. SoundCloud mostly uses Scala for its microservices, so when I joined, it was important to learn it. As I was new to the language, I faced some challenges going from knowing nothing to becoming skilled at writing code in Scala.

This blog post will cover how I kickstarted the learning process, what resources I made use of, how the Scala community at SoundCloud helped me learn and grow, how I was able to keep going when things got challenging, and how I set achievable objectives to monitor my progress.

Learning a New Language

Learning a new language of any kind requires rewiring one’s way of thinking and seeing things. Personally, I had anxiety about not being able to do simple things, like fixing a compilation error that felt straightforward, converting one type to another, or debugging the existing code that had imports that provide implicit functions. However, I found that both being consistent in studying and using a language daily and getting help from people are key to learning a new language.

To give you more insight into this, I’ll share how I started learning with the help of Scala developers at SoundCloud. More specifically, I’ll talk about how I:

Enrolled in and completed a Coursera program
Ordered a book, Scala for the Impatient
Mixed up my learning methods
Eagerly dove into the work and sought help along the way
Took advantage of learning and development
Set personal objectives and key results

Without further ado, keep reading!

Coursera

Coursera is an online learning platform for taking courses on a variety of topics via its partnership with universities and companies around the world. It also provides certification for these courses. I began by enrolling in Functional Programming Principles in Scala, which is taught by Martin Odersky, one of the designers of Scala. I followed the course to first set up my environment, and then I started with my first “Hello, World!” program. I then began learning the concepts, and I worked on a few exercises.

It wasn’t as easy as I thought it would be, and a lot of that had to do with coming from Golang. More specifically, here are a few things I found different:

Go and Scala have different approaches to concurrency and parallelism. Go has goroutines and channels, while Scala has Futures and Promises. As a result, concurrent code written in Go and Scala looks quite different. (Here are examples of my code for running a task concurrently in Go and Scala.)
Scala takes more time to compile compared to Go. The compile time for the code linked above was 3 seconds in Scala and .16 seconds in Go. The slowness of the compiler is thoroughly described here by Odersky.
Go is a procedural and imperative compiled language, and its syntax is similar to that of the C family of languages. Meanwhile, Scala is a hybrid language that combines object-oriented and functional approaches.
Unlike Go, Scala is very concise and requires less boilerplate code.
Each language has a different approach to tooling. Common tasks — like compiling code, running tests, and building an application — are built into Go. Scala, on the other hand, has a separate tool to do these things, sbt.

To understand why applications should use Scala, it was especially helpful to watch videos and read blogs. They demonstrated both why Scala supports both functional and object-oriented programming, and its uses in different applications. There’s a suggested video on Coursera called Working Hard to Keep It Simple (also from Martin Odersky), and it was a great introduction to understanding Scala in a nutshell. I also recommend the blog post, Strategic Scala Style: Principle of Least Power, which aims to provide style guidelines at the “strategic” level. Scala provides various ways to solve a problem, and that post outlines how to choose one from the range of possible solutions.

Scala for the Impatient

I started reading Scala for the Impatient, which was ideal for a Scala beginner like myself. It doesn’t just introduce concepts; it also compares them to other languages like Java and C++, which was one of the things I really liked, because it helped me better understand and made me curious about how things are done in Scala. It also helped me visualize the comparison and provided a sense of familiarity when learning a new language. It has simple examples for learning Scala and an extensive list of exercises at the end of each chapter. The book began with the basics — like types and arrays — and it progressed to concepts like inheritance, objects, operator overloading, tuples, and lists.

One fun fact I learned is that usually, other languages (such as C, Java, or Golang) use [] to access an element. Scala, however, uses () to access an element. For example, to create an array and to access an element, respectively, in Golang, you’d do the following:

var z [Size]String
var i=z[0]

Meanwhile, to create an array and to access an element, respectively, in Scala, you’d do this:

val z = new Array[String] (Size)
val i= z(0)

As you can see, both the above statements create an array and access an element; however, Scala uses parentheses to access an element, while other languages — such as Go, C, C++, and Java — use square brackets to access an element. Apart from this being unusual in other languages, using parentheses means that collections are considered functions, and this can be useful to include functionalities like map andfilter, which expect their arguments to behave syntactically as functions, and not just semantically. To understand more of the history of using square brackets and the importance of switching to parentheses, please refer to the Syntax for Indexing blog post.

Switching Patterns

All that said, most of the exercises weren’t easy, in the sense that I needed to focus on the problem to solve, and not just the language to express the solution. This is something I felt could’ve been different in the course. Giving problems like Huffman coding, Anagrams, LinkedList, and sorting to Scala beginners is like asking a person who just started learning how to skateboard to do it for 30 kilometers: At this point, the person isn’t really great at skateboarding, much less for that distance.

Since it takes time to learn and solve the problems, I learned not to rush or feel like I had to finish the course quickly. And to keep things interesting and to reinforce my learning, I regularly changed how I learned by switching between watching videos and reading.

I had started reading Scala for the Impatient while still taking the Coursera course, as I found the exercises from Coursera were difficult, and I hoped the book would provide more context. Once I finished reading the basic concepts of Scala and doing exercises from the book, I wanted to try a different learning pattern, so I resumed the Coursera course. This time, I felt like it was easier to listen to and digest the video lectures, as I had just learned these same concepts from the book and the videos served to reinforce those learnings. This is because the simple exercises that I tried from the book provided me with more practical and hands-on exposure to functions and libraries in Scala, which in turn helped me complete Coursera exercises that I wasn’t able to finish before.

Diving In and Seeking Help

Even though exercises and courses helped me code in Scala, the one thing as a beginner that really helped accelerate my learning was to actually work on the projects. I started working on a pull request approximately two weeks after I started learning Scala to update a Scala version in a SoundCloud repository. It used a library called Scalaz, but it started failing after the update. I could understand the compilation error, but I had no idea how to fix it. As a note, this repository was written years ago, and most of the methods or concepts had been deprecated or replaced in the new version of Scala. These things can be learned only by working on them.

When I felt like it was really difficult for me to understand and to fix these errors, I asked questions and sought help from the #scala channel on the SoundCloud Slack, and I also did pair programming with colleagues. In doing so, I found that one of the more effective ways to learn a language, beyond just diving in and doing it, is to actually be part of the community and have discussions. It made me realize that it’s OK to make mistakes and ask questions, as this knowledge sharing helps us learn from each other.

Taking Advantage of L&D

SoundCloud has great employee policies for learning and development (L&D). It provides self-allocated time (SAT) for employees, a budget for learning that allows us to buy books and take courses, and extensive documentation for resources that help us learn and grow as developers. Additionally, SoundCloud has great internal documentation for Scala that covers the best practices, test styles, patterns, and libraries used here to provide guidelines for developers to write idiomatic “SoundCloudy” Scala. This provides sufficient resources for beginners.

Setting Objectives

Learning is a process, and especially considering that Scala is both an object-oriented and functional language — meaning it has many ways of expressing things — it might take a long time for people to master this language. As such, it’s helpful to set objectives to track learning and stay on course, which is where Objectives and Key Results (OKRs) come into play. This is a goal-setting methodology used to set goals with measurable results.

In my opinion, learning is a neverending process, but being able to track things on an individual and company level is crucial to know that the learning is indeed happening. OKRs helped with this. Together with my manager, I came up with my own OKRs using a performance management toolkit at SoundCloud called Small Improvements. These OKRs were to:

Finish the Scala course from Coursera
Create a small personal project using Scala
Get my hands dirty by implementing as many pull requests as possible
Perform code reviews on my colleagues’ pull requests
Record this process in a blog post to help others who are new to Scala

Key Results

After setting and working toward the objectives outlined above, here are my key results:

I am certified in the Coursera Functional Programming in Scala course.
I wrote a web service to create/retrieve a playlist that uses the following libraries:
- Http4s — for the HTTP layer, this library uses cats-effect to manage I/O
- Flyway for database migrations
- PostgreSQL for the database server
- quill for expressing SQL as a Scala class
- circe for JSON encoding and decoding

Doing this helped me start a project from scratch, incorporate concepts I had learned, and explore various libraries in Scala. You can see the end result, playlists, on GitHub.

Additionally:

I implemented features by raising 64 pull requests and code reviewed 44 pull requests in Scala at SoundCloud in the past few months.
I finished writing this blog post.

Takeaway

In addition to all I did and learned, I had a few additional takeaways from this process:

One of Scala’s goals is to make programmers focus on high-level application logic, so it provides extensive built-in methods. This means I can do many manipulations of any type with less effort and time than in Go. One of the best examples of this is the .map method. With the help of this method, I don’t have to worry about writing code to iterate an array or list. Rather, I can spend my efforts on writing the logic of what manipulations need to be done on these elements.
As a developer, I think one of the essential tasks of my job is to write effective test code that’s both readable and easy to write in order to cover simple and complex test cases. One of the simple and yet more efficient testing frameworks that I used in the project is MUnit, in combination with the integration library munit-cats-effect. This made it easy to test the stream response for HTTP testing. Meanwhile, there are other simple and more important testing frameworks, like ScalaTest, with different testing styles.
Because I was a Golang developer for almost four years, I nearly forgot about object-oriented programming concepts like inheritance, encapsulation, and polymorphism. The whole process of learning Scala made me recall and refresh these concepts.

Conclusion

Learning is a continuous process, and the longer I’m here at SoundCloud, the more I’m learning. It’s a great opportunity to be able to grow my skills and learn a new language, and if you’re considering doing the same, I hope this post can help you kickstart your learning.

Additional References

The End of the Public API Strangler

Mon, 14 Mar 2022 00:00:00 GMT

This is the story of how we used the Strangler pattern to migrate our public API from a monolithic codebase to a fully fledged BFF over the course of eight years. It also discusses some of the trials and tribulations we encountered along the way.

History

SoundCloud started as a single Ruby on Rails application more than 14 years ago. Back then, this single application served the website and the public API. While going through multiple growth phases — both in terms of user traffic and the size of the engineering team — we made the choice to more formally adopt a type of service architecture commonly referred to as microservices. This move promised to unblock engineering teams and widen technology choices, in turn allowing us to pick appropriate tools for handling our scaling challenges.

After the Cambrian explosion of languages, frameworks, and approaches, the engineering organization started to consolidate, and Scala became the default language for delivering our microservice architecture. That’s because it’s underpinned by Twitter’s Finagle as the RPC framework for interservice communication, and we knew we wanted to use Finagle.

Motivation

In the new reality of a microservices architecture, where some new features now existed outside of the Rails application, and some services supplemented the existing features of the Rails application, we needed to decide how to maintain the public API going forward. It was crucial to ensure that important features — like serving content — continued to work for existing integrations, even though their implementations had changed and now spanned multiple services.

We tried various approaches and learned that our Rails application didn’t perform well when interacting with multiple microservices to serve user traffic. As a result, in 2014, we made the decision to not integrate it with other services to serve the public API, but to instead build a Scala service using Finagle that would internally proxy requests to the existing public API. This new service would intercept and augment the public API responses by calling additional services when necessary, (somewhat loosely) following the Strangler pattern.

Typically, the goal of the Strangler pattern is to incrementally replace the functionality of one system with the functionality of a new, more desirable system or systems, one piece at a time. It’s commonly used when migrating from a monolithic codebase to a microservice architecture. Although this end goal informed the original decision to adopt the Strangler pattern, our choice to use it was more motivated by an immediate need rather than planning for a future free of the public API monolith.

As a result, the Strangler was left, along with the monolith, largely unmaintained while feature development on our internal APIs continued at pace. To facilitate the continued development of our internal APIs, it became necessary to duplicate code paths for accessing core entities, e.g. tracks, playlists, users, etc. This meant one code path for internal clients and one for the public API. In addition to the obvious downside of this duplication, inconsistencies between the two APIs also emerged. A lack of maintenance also meant knowledge loss, security issues from exposing the monolith with deprecated Rails versions via transparent proxying from the Strangler, and scope creep due to feature teams often needing to touch the Strangler and/or the monolith without much prior knowledge.

As the business matured further and new investments in the public API were planned, the case to address the current situation became compelling, and the work was scoped to complete the migration of the Strangler to a fully-fledged BFF. In January 2020, after a six-year period in which modest progress was made, the work began in earnest.

Porting from the Public API to the BFF

Importantly, two preliminary steps helped us reduce the scope of the work.

Some of the official apps were still making some direct calls to the public API. These were migrated to use the official BFFs, which enabled us to shrink the API surface, and hence the scope, significantly.
Much of the common functionality in the BFFs was consolidated in Value-Added Services, further reducing the scope of the work.

Without these preliminary measures, completing this project may have been unrealistic.

Endpoints could be ported with varying degrees of difficulty. In some cases, there were existing reference implementations in other BFFs, e.g. for web or mobile, that could be used as a guide. In others, a complete rewrite was needed, and often it was necessary to add missing functionality to downstream microservices.

Challenges

Due to the lack of experience within the company with the public API codebase, it was actually necessary to first investigate and document the full list of endpoints that the public API exposed. This involved:

Adding telemetry to understand which endpoints were still in use.
Explicitly declaring all known public API routes in the Strangler codebase.
Adding a fallback to call the public API for any undeclared routes.
Removing the fallback once we were confident we had identified all routes.
Removing routes that weren’t in use and weren’t documented on the developer portal.
Creating a JIRA ticket for each endpoint to be ported, i.e. reimplemented in the Strangler, by calling existing microservices instead of the public API.

Also, some things that come for free (or “magic”) in Ruby were things we needed to implement ourselves — for example, multipart request parameter parsing. Furthermore, Rails doesn’t need to be explicit about all route and Content-Type combinations it supports. This sometimes led to unpleasant surprises during porting as it became clear that entire chunks of functionality remained to be implemented.

Response Comparisons

To build up confidence in ported endpoints, the process typically goes like this:

The ported implementation gets deployed alongside the old code that proxies to the public API.
Incoming requests execute both codepaths — the old (using the proxy) and the new code.
The response of the proxy’s call to the public API gets returned to the caller.
At the same time, the responses of the proxy and the new code are compared for consistency.
If the responses of the old and new code don’t match, a telemetry event is triggered and the difference is logged for inspection by the developer.
The developer may then need to make some changes to the ported implementation until they’re confident that the new code matches the original in terms of functionality.
At this point, the proxy can be removed and the ported response gets returned.

Of course, this is only really possible for non-mutating methods, i.e. GET or HEAD requests. Otherwise, the code might end up creating two entries in the database for a single request. To deal with mutating methods, i.e. PUT or POST requests, it was sometimes necessary to perform extensive manual regression testing. Not everything went as smoothly as we would’ve liked, and the use of rollout flags also proved useful for quickly disabling the ported code where problems did occur.

Learnings

Decisions need to be made based on data, and telemetry was key in informing such decisions. For example, undocumented endpoints that receive minimal usage can sometimes be deprecated and later removed. The less code to port and maintain, the better.

Adopting the Strangler pattern comes with significant risks. As mentioned, there was initially a long fallow period where not much work was done to port the public API endpoints to the Strangler. During this period, some knowledge about the project was lost and the Strangler in fact added some complexity for teams developing new features. If you decide to adopt the Strangler pattern, make sure to have a plan to complete the migration before the knowledge is lost and it becomes a daunting task with increased risk.

As with any kind of rewrite, even an incremental one, bugs can occur. The nature of the public API — being accessible to anyone who wishes to use it — meant that sometimes things broke, and sometimes for seemingly innocuous changes. In some instances, the tenet of Hyrum’s Law came into play, where breakages occurred due to third-party integrations relying on undocumented aspects of the APIs.

It’s worth considering whether such disruptions to your business are worth the ultimate benefits of the work.

Conclusion

The work wasn’t without its challenges, and anyone embarking on such a project should be aware of the risks and benefits involved. However, now that all endpoints have finally been ported, there are some notable benefits: the Strangler is now a fully fledged BFF; the entire codebase of the public API has been deleted; and we have a codebase that most engineers can contribute to (Scala service), that doesn’t negatively impact project scope, that fits with our microservice architecture, and that helps ensure data consistency and security.

What Is New with Periskop in 2022

Tue, 01 Feb 2022 00:00:00 GMT

In a previous blog post, we explained how we built an internal pull-based exception monitoring service called Periskop — which is heavily influenced by Prometheus — which we open sourced.

The adoption of this tool at SoundCloud has proven successful during the past year. Currently, it’s a Golden Path tool in our observability stack, and it’s particularly key when an incident happens. Many teams started using Periskop as part of their debugging workflow — even those that still have to maintain legacy systems that don’t use SoundCloud’s default technical stack.

This increased use encouraged us to improve Periskop by adding more functionality and releasing more open source projects for the Periskop ecosystem.

In this post, we’ll discuss recent updates we made to Periskop and its ecosystem. Many of the new features explained here were mentioned in the previous blog post as part of future work.

Official Website and New Organization

We’re happy to announce that we have our static site pointing to http://periskop.io. We also moved all related Periskop repositories to a new GitHub organization called periskop-dev. The goal of this is to change the governance model to one that’s more community-centric and not exclusive to SoundCloud. This lines up with our mission of making Periskop the go-to tool for exception reporting in the cloud native ecosystem.

Client Libraries

Since we opened Periskop to the community, we wanted users of different languages to also use Periskop for their programs. Initially, we developed a client library for Scala, which is the technology we use the most internally. Later, we started adding more client libraries, and there are currently four:

The development of periskop-go was especially interesting, as Periskop uses it to scrape itself to capture any error that occurs during its execution.

Persistence

We built Periskop with simplicity in mind: deploy and run. Software that’s similar to Periskop — like Sentry — requires provisioning a database if you want to deploy the service internally to store all the exceptions.

With Periskop, you can simply deploy using in-memory storage for the exceptions. This has the downside that if you redeploy your Periskop instance, the captured errors won’t persist. However, this usually isn’t a problem, since most of the time, if the scraped service had the errors reported in memory (via a client library), the errors would be scraped again.

To solve this problem, we decided to extend Periskop, providing the ability to store exceptions in a persistent storage. Currently, Periskop can be configured to use persistent database systems such SQLite, MySQL, and PostgreSQL. Documentation on how to achieve this can be found in part of this README.

Push Gateway

We also wanted to provide push capabilities to Periskop. This is especially useful for shortlived processes like batch jobs or fork-based application servers. In the Prometheus documentation, you can find plenty of good examples and a detailed explanation as to why a push gateway is useful.

The following graph shows the architecture of the push gateway.

Basically, it’s a simple Go service that you can deploy as a sidecar container. Client libraries can use the push_to_gateway method pointing to the address and port where the service is deployed, pushing any exported metrics to the gateway.

We use it in production for our monolith written in Rails with periskop-ruby. There’s Rack middleware, which handles exceptions that may happen during the execution of a request, and it pushes them to the attached push gateway of the deployed service.

Service Discovery

As we’re following the principle of Prometheus, but for exceptions, we also wanted to have the same service discovery mechanism that Prometheus has in place. This pull request shows how this was achieved and how we can configure Periskop to use Prometheus service discovery.

User Experience Improvements

We introduced several improvements to the UI that make Periskop more user friendly — especially when searching or filtering for a specific error during an outage. These are the main improvements:

Search for errors — We had an initial search functionality that was only searching for the error name. We extended the search functionality to search any text within the reported errors. This is particularly useful when searching for a specific endpoint or for an error message.
Filters and sorting — We added the ability to filter per severity type and sort errors per number of occurrences or the date of last occurrence. Thanks to Tiago Taquelim for the contributions!
Mark errors as resolved — We also added the option to mark errors as resolved. This comes in handy if you fix a bug, you deploy it, and you don’t want it to be in the UI anymore.

Prometheus Metrics Support

All reported errors are instrumented using Prometheus metrics. In the documentation, there’s a basic example of an alerting definition for Alertmanager.

What’s Coming Next

Some of the ideas we have in mind for continued improvement and added features include:

Built-in federation (hierarchical collection)
Time series visualization
More integrations (Backstage, Grafana)
Support for more language and frameworks
Labeling of errors

Periskop is open source, and we’re happy to accept external contributions! If you find this project useful, we’d love to hear from you. Please drop us a line at our gitter chat.

Obvious Ownership: A Sensible Humane Registry

Thu, 06 Jan 2022 00:00:00 GMT

Imagine yourself as an engineer who just joined SoundCloud. Besides meeting your colleagues and getting your new laptop, badge, and that cool branded hoodie, the first weeks at work will be about exploration. Onboarding involves a lot of “looking around”: finding mentorship, getting to know the company’s culture, and exploring codebases — and most likely, you’ll want to contribute code too!

The first question you might ask is “What do I even work on?”. With time, you’ll identify a few sibling teams you interact with often — because you either consume their data or provide them with yours. So now, it’s important to know not only what you work on, but also which folks depend on you, and vice versa.

As SoundCloud has grown rapidly — both in the number of people and in software — we’ve identified “Who owns X?” as a recurring question coming primarily from new joiners, but also from tenured folks. You can imagine how this information can also be useful to an EM advocating for more people on their team, or to an incident-response team when diagnosing an outage scenario. We needed an obvious way to consult system ownership, so we decided to build a humane registry.

Humane Registries

If you’ve been around service-oriented architecture (SOA) before, you may have heard of service directories. The WS-* standard for them is called Universal Description, Discovery and Integration (UDDI), and it’s a complicated mix of service discovery and human readable information. A snippet from a classic book describes UDDI:

“At its core, UDDI consists of two parts (…) a technical specification for building a distributed directory of businesses and web services [that] enables anyone to search [its] data. It also enables any company to register itself and its services.

The data captured within UDDI is divided into three main categories:

White pages: This includes general information about a specific company—for example, business name, business description, contact information, address and phone numbers.

Yellow pages: This includes general classification data for either the company or the service offered. For example, this data may include industry, product, or geographic codes based on standard taxonomies.

Green pages: This category contains technical information about a web service. Generally, this includes a pointer to an external specification and an address for invoking the web service (…) UDDI can be used to describe any service, from a single web page or email address all the way up to SOAP, CORBA, and Java RMI services.”

Humane registries play a similar role, but they focus on human-readable information. These systems are pretty much what OpenHub does, i.e. automatically derive as much information as possible from a project based on its code and metadata, and add a layer of prose via wiki-like tools.

The whole point is that a humane registry acknowledges that while automation and APIs are crucial for an SOA architecture, people talking to each other is still the most important and effective collaboration we have. The registry helps start effective conversations; it doesn’t try to eliminate them.

Services Directory

When planning our implementation, we had a single requirement: The registry must enable finding out which systems a team owns and who the owners of given systems are. As a stretch goal, we were also interested in exposing dependencies across systems and teams.

To deliver, we introduced the concept of “manifest files,” which would serve as entity metadata descriptors: Each code repository should have at least one manifest file. Naturally, one of the fields in the manifest file would be the owner team for that specific software piece. These files would then be ingested by the registry on a recurring basis, in order to form an up-to-date software-catalog graph that could then be rendered into the registry’s UI:

{
  "_schema": "soundcloud/applicationmanifest/v1",
  "name": "periskop",
  "description": "Language agnostic exception aggregator for microservice environments.",
  "owner": "engineering-productivity-team",
  "type": "tool",
  "status": "production"
}

Our humane registry was named Services Directory. Of course, later on we learned we were interested in cataloguing more than just services.

More Automation: A Schema Definition

By introducing manifest files and ingestion through a services directory, we attained our goal of having a central index for our software pieces. What we observed, though, is that maintaining correctness of the data over time became a challenge: teams get renamed, systems are handed over, etc. In addition, small things like typos, or different ways of referring to teams — as exemplified by the screenshot above with user-growth and User Growth team names — prevented our aggregation logic from providing good results.

Well, there are existing technologies built specifically for the purpose of validating documents. So we began to define a vocabulary of properties our ingestion system expected, and we encoded them in a schema:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "SoundCloud's Application Manifests Schema",
  "type": "object",
  "required": [
    "owner",
    …
  ],
  "properties": {
    "owner": {
      "type": "string"
    },
    …
}

The next step was to provide a validation engine based on the predefined schema, running it against the manifest files. In the spirit of continuous delivery, we could hook that up in our CD pipelines as part of the linting stage: Our manifests would be validated against each individual change to these repositories, assuring they never drift from the schema.

As a bonus, by relying on JSON Schema as a schema-description standard, we were also able to provide ahead-of-commit support (developer experience FTW) by leaning on autocompletion of popular code editors.

On top of executing single-pass manifest validation, having an automated way to check for schema compliance opens the door to a more holistic view of the company’s software catalog. Services Directory ingests manifest files, checks syntactically for unwanted properties, and has the ability to report back semantically on values.

It was easy to set up a validation step to compare owner values with a knowingly good list of existing teams, sourced from our LDAP server. This means a software piece with an invalid team name gets rejected in a CI environment. It also means we could provide teams with a comprehensive view of their ownership. A background job exports Prometheus metrics on the manifest validation engine, which powers a dashboard, like shown below.

What Else?

Ensuring code ownership visibility is just one aspect of Services Directory, our humane registry. Including manifest files as entity descriptors became ingrained in SoundCloud’s engineering culture, and as schema providers, we have a way to ensure standardization across our vast code landscape.

Since introducing validation logic, we’ve seen higher engagement with Services Directory and identified it as a central piece in our developer experience tooling offering. Various teams and individuals have reached out with feature requests and integration ideas, which we’ve been partnering on to provide — we’ll share more about that in the future.

Tiny Letter from Kafka

Wed, 13 Oct 2021 00:00:00 GMT

This article discusses the powerful design choice of Apache Kafka, “an open-source distributed event streaming platform,” and gives a sneak peek at how SoundCloud teams consume event data from it.

The Story of the Name

Why is Apache Kafka named after the author Franz Kafka? Frankly, this was my first thought when I began learning about Apache Kafka. After researching, I found out there’s a little story behind the name.

In a LinkedIn post from 22 November 2019, Langfei He shared the following story, which has been told and retold by many people over the years.

Franz Kafka met a young girl crying in a Berlin park after she lost her favorite doll. Kafka tried to find the doll for her, but he wasn’t successful. He asked her to meet the next day to continue looking, but when they still hadn’t found the doll, Kafka gave the girl a letter he claimed was from the doll.

It read: “Please do not cry. I have gone on a trip to see the world. I’m going to write to you about my adventures.”

And so, whenever Kafka would come back from his travels, he’d meet with the girl and read her the letters. After a few years of this, he came back with a new doll and gave it to the girl.

Upon receiving this doll, the girl said, “This doll does not look like my doll at all.”

But Kafka gave the girl a tiny letter, saying it was from the doll. The letter said, “My trips have changed me.” She was convinced and took the doll with her.

The following year, Kafka died.

Many years later, the girl found a letter inside a hidden crevice in the doll. The tiny letter said, “Everything you love is very likely to be lost, but in the end, love will return in a different way.”

I found this story amusing, not just because of how sentimental it is, but also because of how it can be used to explain the core flow of Apache Kafka.

How is that exactly?

If you want to understand why Apache Kafka is a brilliant choice of a distributed messaging system and why we should discuss it more, keep reading.

Publish–Subscribe Messaging Pattern

To understand Apache Kafka, you first must understand the publish–subscribe messaging pattern.

The publish–subscribe messaging pattern has a publisher and a subscriber — the publisher is a sender, and the subscriber is a receiver. The publisher doesn’t send messages directly to the subscribers; instead, it classifies each message under a certain topic without knowing if there are any subscribers to its messages. The same goes for receivers: They request a certain type of message without knowing if there are publishers.

The publish–subscribe system usually has a broker where all messages are published.

Image 1-1

Image 1-1 illustrates how Kafka uses the publish–subscribe pattern. Senders don’t send messages directly to receivers. Instead, senders publish messages inside the broker, which is where all messages are published, just like a bulletin board. Subscribers who are interested in a certain topic will pull the messages from the brokers. Imagine a library: Authors don’t deliver their books to readers directly. Instead, readers check out the books they’re interested in.

Topic? Log? Kafka as a Database?

In Kafka, a topic is a category name that represents a stream of “related” events. For example, we can set topics like user_created, password_changed, new_track_release, user_email_confirmed, etc., and consumer applications can collect the messages from each topic to do useful things that fit our business needs.

Let’s say there’s a topic called new_track_release. Under this topic, the following events can be saved as distributed logs: “Billie Eilish’s track ‘Not My Responsibility’ was released on 26.05.2020”; “Sébastien Tellier’s track ‘Stuck in a Summer Love’ was released on 29.05.2020”; “Kehlani’s track ‘Can I’ was released on 30.07.2020.”

There’s a timestamp that corresponds to each event, and the latest record will be appended at the end of the log, and the position of the log and the contents of the logs are immutable.

There’s a saying from the Dalai Lama: “Time passes unhindered. When we make mistakes, we cannot turn the clock back and try again.” The same rule applies to Apache Kafka: Nobody can turn back time or manipulate what has happened.

Another fascinating fact about Kafka is that Kafka can function as a hybrid system between a database and a messaging system.

Kafka topics are persisted on disk, not memory, for a certain period. During this period, Kafka provides durable storage of the messages. This is called a retention policy, and the retention period can be configured by engineers as per our needs, depending on the topic. For instance, if the topic is metrics, we’d set a short retention period, as we usually don’t need the logs of the metrics for longer than a couple of weeks. On the other hand, a topic like subscription_purchase_history will need to be set to a much longer retention period because we need a user’s purchase history stored in the system for years.

In a Kafka cluster, even if the node that stores messages restarts or dies, the data won’t be lost, since data is stored across multiple servers in the cluster. This means there are multiple copies of the same data in a cluster in case one of the brokers goes down or becomes unavailable.

Image 1-2

In Apache Kafka, producers and consumers are the applications you as an engineer develop, as you can see in Image 1-2 above. Producer applications write data into the Kafka cluster. Consumer applications then read the data out of the Kafka cluster and do things like generating reports; creating dashboards; sending notifications; and performing business logic, which includes making campaigns or creating another pipeline to send data to other applications.

To summarize, the architecture of Kafka is like this:

Producers act as publishers and consumers act as subscribers.
Thanks to the decoupling, slow consumers don’t affect producers.
Engineers can freely add more consumers without affecting producers.
The failure of consumers doesn’t affect the system.

Image 1-3

At SoundCloud — especially on my team, Marketing Tech — we own multiple consumer applications, which do interesting things by consuming topics from the Kafka cluster. As Image 1-3 illustrates, one of the consumer applications we own is called Scoop. Scoop listens to the events for topics from the Kafka cluster, such as pageviews, sound_played, user_primary_email_changed, user_registered, user_reaped, and user_email_confirmed. These events are initially emitted to the Kafka cluster from our legendary Mothership. Then, Scoop pulls the messages from the Kafka cluster and sends them to another system called Bob the Braze Builder, which acts as a proxy to sync data to our CRM tool, Braze. Based on the events persisted in Braze, our Growth Marketing Team can create personalized campaigns for each user.

Powerful Design Choice of Kafka Consumer Groups

By definition, a consumer group is a group of consumer instances that has the same group_id. A consumer group exists to read Kafka topics in parallel as a system scales up. And in this section, I’d like to demonstrate why Kafka consumer groups are such a great design choice and how we benefit from them.

The simple fact that messages don’t get destroyed after being read and committed is key in enabling different consumer groups to work independently.

How the Kafka consumer group is designed is as follows.

A consumer instance always belongs to a consumer group; even if there’s only one consumer instance, a consumer instance belongs to a default consumer group automatically. Certainly, it scales to balance to consume more partitions as the system scales up, which means in one consumer group, multiple consumer instances usually exist.

Another interesting fact is that a partition is read by only one consumer instance within the same consumer group. For example, if Partition 0 is read and committed by Consumer 1, this partition won’t be read by any other consumer instances from the same consumer group. However, Partition 0 can be read by any other consumer instances from different consumer groups. In other words, multiple reads on the same partition can be happening simultaneously or at a different time, as long as the consumer instances reading the same partition are from different consumer groups.

Remember what I said about Kafka being a database for a certain amount of time? In Kafka, messages are retained for a certain period, as long as their retention period isn’t over. Therefore, multiple consumer groups can benefit from reading the same messages during the retention period. This is one big difference compared to other messaging systems like RabbitMQ, where once a partition is read and committed, the message is removed from the queue.

Image 1-4

Image 1-4 demonstrates that a partition gets read by one consumer instance exclusively within the same consumer group. When there’s capacity, one consumer instance can read more than one partition.

As you can see, Partition 0 isn’t read by other consumer instances from the same consumer group. However, it is read by other consumer instances from different consumer groups.

Scaling consumer instances in a consumer group is automatic. When a consumer instance dies, another instance comes up, or a different consumer instance starts to consume from the partitions that are assigned with the instance that has just died.

Moreover, Kafka stores the offsets where the consumer group has last read. Consider Image 1-5 below. When a consumer processes data, it commits the offsets. Even if a consumer process dies or something happens, it can read back from where it left off, thanks to the consumer offsets. When different consumers that belong to different consumer groups are reading the same partition, those consumers will probably have different offsets, because the pace of reading likely varies.

Image 1-5

The offset numbers are generated when a message is produced and sent to a Kafka cluster, just like a page number is already created when an author publishes a book.

The nice thing here is that the ordering is always guaranteed in the same partition; it isn’t possible for consumers to read Offset120 before reading Offset119 inside Partition 0.

Let’s take a closer look at Image 1-4 one more time. You see that whenever there’s a new_track_release by Billie Eilish, the message gets written into Partition 0, Justin Bieber’s message gets written into Partition 1, and Drake’s message gets written into Partition 2. This isn’t magic. The reason behind this is that we as engineers most likely set the same key for the same artist so that the messages that belong to the same artist can be written into the same partition. This is something we can do by setting up the same key when building a producer application (a subject we won’t cover in this blog post). We’d do it this way because the ordering is guaranteed in the same partition, and it’s convenient for consumers to consume the messages in the order of events.

For instance, let’s say there are two songs released by Billie Eilish with two different timestamps.

The message, “‘Therefore I Am’ has been released on 12.11.2020” is written to the Kafka cluster before the message, “‘NDA’ has been released on 09.07.2021.” This means the message, “‘Therefore I Am’ has been released” will always have a smaller offset number than the message “‘NDA’ has been released,” as long as these messages are stored in the same partition. The consumer applications will always read the messages from the smaller offsets.

To summarize, in Apache Kafka:

Offsets are created when a message gets written into Kafka and it corresponds to a specific message in a specific partition.
Consumers read messages by subscribing to one or more topics and reading messages in the order in which they were produced in every partition. A consumer keeps track of its position in the stream of data by remembering what offset was already consumed.
It’s a consumer’s responsibility to remember which position they’re at in the partition by storing offsets. By doing so, consumer applications can stop and restart without losing their position in the stream of data.

Some argue that Apache Kafka has a limit of not being able to guarantee the order across partitions. However, as long as you as an engineer set up the same partition key for the related messages, it’ll serve your needs. Usually, having an order guaranteed within a partition is enough.

Image 1-6

Image 1-6 shows how the same partition can be read by different consumer groups and do useful things independently at SoundCloud. In this diagram, Scoop listens to a topic, pageview, which fires when a user views a subscription page. For instance, if user#1 with a Go subscription checks out the subscription page, the pageview event gets fired. When this event fires, Scoop will consume the event and sync it to our CRM tool, Braze. Then, our Growth Marketing Team can provide personalized campaigns or discount suggestions on a Go+ subscription to user#1. At the same time, our Data Science Team’s data analysis consumer group can consume the same event from the same partition and create a report to analyze user#1’s behavior. This demonstrates how Kafka messages can be consumed by multiple downstream systems to do interesting and useful things.

In the End, a Message Will Return in a Different Way

Returning to the story with Franz Kafka, remember how the little girl found the tiny letter hidden in the doll many years after Franz Kafka passed away? Now you might understand that the doll acts as a broker, and Franz Kafka himself acts as a producer, and the little girl is like a consumer.

Of course, this story is oversimplified. In reality, our engineering ecosystem mostly lives in a distributed system. It requires Kafka to be scalable, reliable, and fault tolerant.

Kafka’s choice of a publish–subscribe system and the architecture of consumer groups are quite valuable. Ultimately, Apache Kafka:

Is a great tool for ETL and big data ingestion
Has disk-based — not memory-based – retention
Has a high throughput
Is highly scalable
Is fault tolerant
Has fairly low latency
Is highly configurable
Tackles integration complexity
Allows multiple producers and consumers

As an engineer, it’s your decision to design system architecture, data flow, and which messaging system to use for your business needs. Apache Kafka can be a useful choice if you want your event data to be retained and consumed by many other downstream systems, and if you want to fight the ever-growing complexity of integration in the future!

Service Architecture at SoundCloud — Part 3: Domain Gateways

Fri, 17 Sep 2021 00:00:00 GMT

This article is the last part in a series of posts aiming to cast some light onto how service architecture has evolved at SoundCloud over the past few years, and how we’re attempting to solve some of the most difficult challenges we encountered along the way.

In the second part of this series, we discussed how we evolved the use of the BFF pattern in SoundCloud by moving existing duplicated logic to a more centralized and elegant solution called Value-Added Services (VAS). We covered how we benefit from this architecture pattern, as we have all the authorization, content policies, and fetching logic of tracks and playlists in a single service.

In this blog post, we’ll cover how we evolved the concept of Value-Added Services to Domain Gateways, which allow us to extend those services to have read and write operations in a single and centralized service for each business domain.

Growing Aggregates

As we described in the previous installment, the core responsibility of a VAS is to serve our core aggregates, such as Track and Playlist. To do this, a VAS fetches states for associated entities and value objects from corresponding Foundation¹ services, and then it applies business authorization rules. For example, the Tracks VAS will filter out all tracks that are geoblocked in certain territories.

Roughly speaking, one can imagine a VAS as a big fanout together with authorization logic. One of the first challenges that we faced while extending our Value-Added Services was the size of this fanout. As we were adding features to the platform, our aggregates — and hence the amount of network calls — were growing as well.

On the other side, our BFFs often have different needs dictated by their applications. For example, one track feature might only be available on mobile, which makes fetching the entire track aggregate from the Web API unnecessary. Moreover, even within a single BFF, we sometimes support multiple aggregate representations that can be built without fetching all dependencies.

How can we provide centralized endpoints for serving aggregates that can be customized to the specific needs of BFFs? Luckily, this problem has a pretty straightforward solution — partial responses. This pattern allows API consumers to tell the producer which part of the response they’re going to consume by specifying a FieldMask in the request. Field masks support protobuf and JSON representations that make them essentially protocol agnostic.²

In our particular case, we use Twinagle — a protobuf IDL based on the Twirp protocol. Protobuf definitions provide type-safe construction and validation out of the box via FieldMaskUtils that we’ve ported to the ScalaPB library.

One disadvantage of field masks for partial responses is a tighter coupling between microservice topology and aggregate schemas (IDLs). Field masks can be defined according to service dependencies and network calls to reduce the number of requests necessary to produce a BFF representation. At SoundCloud, our focus is more on the reduction of complexity in the edge layer (specifically in BFFs). While field masks can optimize network calls as well, it isn’t necessary to have a 1:1 mapping between field masks and network calls.

Commands

While we were extending the scope of the VAS to serve aggregates of our entities, we identified that we could also extend our VAS to those actions that mutate the state of the core entity (i.e. write operations) but at the same time would require authorization logic. To centralize even more core entities, we extended our VAS with commands. Some examples of these command operations in the Tracks domain include “download a track,” “like a track,” and “repost a track.”

Since it’s an operation that lives in the VAS, it also has the benefit that we reduce complex logic in BFFs (in case such logic was duplicated there) and improve reliability in terms of access logic of those operations that require grant access to a given track.

We can illustrate the case of liking a track in the Track VAS:

As we can see in the graph, BFFs would send a request to the Tracks service to perform a track operation. The service that usually registers “like” operations lives in the Likes service. This service isn’t aware of track authorizations; it only creates/deletes links between tracks/playlists and users. That’s why we need to check first if the user who wants to like a track has access to it. The best place to achieve such logic in a centralized place is the Tracks VAS.

Separation of Queries and Commands

To summarize, the VAS interface consists of two parts: an endpoint to serve its aggregate according to BFF needs, which we call queries; and endpoints that expose core entity operations, which we call commands. This separation is the core idea behind the CQRS pattern and provides some practical benefits, as it’s possible to provide separate upstream services or stores for read and write operations. For example, the foundational service that provides operations to add or remove a follow/er/s relationship between two users (a write) is different from the service that serves follower counts. This relationship between foundational services is now abstracted away from users of the Tracks VAS, which improves consistency and reduces complexity in the BFFs.

Beyond Core Entities: Domain Gateways

As our VAS grew in scope, we identified that a single core entity (like a track) can be used in different domains, for different purposes, and with different access patterns and authorization requirements. For example, SoundCloud not only provides a consumer application to a music catalog; it also provides tools for creators to upload and distribute their music. Consumer and Creator are different domains, owned by different teams — all of them referencing and using tracks for different purposes within their specific domains.

A possible approach in this case is to implement everything that can possibly be done with a track (in all the different domains) — including related queries and commands — in a single VAS. This can work well for some time, but eventually there’s a risk of creating large amounts of coupling and complexity, causing friction and decreasing productivity.

A more scalable approach is to identify the different business domains that need to make use of a given entity and create a Domain Gateway for each of them. In essence, a Domain Gateway is an implementation of a VAS tied to a specific business domain. Each one can be maintained by different teams and represent different views on a given entity, relying on the same foundational layer of services. This façade can provide stability and act as an anti-corruption layer for each of the domains.

The Domain Gateway approach involves a certain level of duplication in exchange for autonomy and increased scalability, and it makes sense to apply in cases in which the different domains have very different access patterns and highly disjoint feature sets, or when communication and collaboration between teams is difficult (for example, due to geographic location of teams and non-overlapping time zones).

Summary

As we discussed in the previous blog post, the evolution of SoundCloud’s architecture into a three-tier architecture with Value-Added Services as authoritative entry points for accessing aggregates has proven successful — even more after evolving them into the concept of Domain Gateways. This is a pattern that we’ll continue adopting and applying in the future.

We plan to move other operations that are duplicated in our codebases to their respective gateways. This will provide more flexibility to evolve our system as soon as we want to add new functionality without the hassle of duplication in each of the BFFs.

In parallel, we’ll continue encouraging feature teams to evolve their microservices architecture around their core domain. This will create a more solid landscape where business logic is centralized and more easily accessible from other dependent systems.

Finally, we’re still exploring the possibilities enabled by Domain Gateways — including improved team autonomy and reduced cycle times for our development process.

1: For a review of SoundCloud’s architectural layers, refer to our previous blog post.
2: GraphQL is an alternative approach to provide an API that can be customized to consumer needs. Although it provides more flexibility, we decided that its benefits won’t offset the cost of migration from our standard Twinagle stack.

Service Architecture at SoundCloud — Part 2: Value-Added Services

Fri, 20 Aug 2021 00:00:00 GMT

This article is part of a series of posts aiming to cast some light onto how service architecture has evolved at SoundCloud over the past few years, and how we’re attempting to solve some of the most difficult challenges we encountered along the way.

In the first installment, we covered the use of the BFF pattern within SoundCloud, detailing its pros and — more significantly — its cons. While the BFF architecture comes with many benefits, such as optimizing backends suited for different clients and a higher level of resilience than a shared single backend, its implementation at SoundCloud became problematic over time. Unnecessary complexity and duplicate code developed. Even worse, we had business and divergent authorization logic living in each of the BFFs, which is a dangerous pattern, as the maintenance and synchronicity of this code is of paramount importance. It became clear that we needed a different approach: Enter Value-Added Services (VAS).

Value-Added Services

First, let’s cover the different service layers at SoundCloud that make up this architecture.

Edge: This layer provides API gateway capabilities and is where our BFFs live. The BFFs are published and maintained dedicated APIs that are tailored to specific client needs.
Value Added: Services in this layer consume data from other services and process them in some way to build rich experiences for users.
Foundation: This is a low-level service that provides the building blocks around a domain.

It’s also important to understand the building blocks that come together in Value-Added Services. These are all well-known domain-driven design concepts, which you can read more about in this article.

Domain: A user or business concern that can be used to draw boundaries/scope around service integrations.
Entity: An entity is an object that has an independent identifier and a lifecycle.
Value Objects: Value objects contain metadata related to a given entity; they’re also tied to the lifecycle of the given entity.
Aggregate: An aggregate is a collection of one or more related entities. An aggregate has a root entity called the aggregate root. Aggregates can also contain references to other entities, but not the referenced entity metadata. It’s then up to the consuming services to call other services to synthesize the entity references.

Value-Added Services are business services responsible for returning an entity and its associated value objects (in other words, an aggregate) to the caller. It’s important to note that a VAS is not responsible for synthesizing metadata for any associated entities. This allows for a nice separation of concerns — along with a centralized point where metadata and authorization rules for a given entity can be defined. A VAS can then orchestrate calls to these services to synthesize and authorize aggregates to then return to the BFF.

Let’s apply these concepts to real-life examples at SoundCloud. An example entity is a track, which has associated value objects like metadata, transcodings, and authorization policies to determine visibility. A track is also connected to an owning user, but since this is another entity, it only contains the user ID as a reference. If a consuming service has a track ID it wants to resolve, it’ll then call the Tracks VAS, which takes care of ensuring that the track is visible, and then it returns the according track aggregate.

Previously, if an end user wanted to fetch a track, the request would be sent to the BFF. It would then be up to the BFF to determine whether the session user was authorized to view this track and, if so, to synthesize the external track representation to return back to the user. This would involve calls to various Foundation services that are individually responsible for returning both authorization information and track metadata.

However, once the Tracks VAS was introduced, this pattern changed. All the duplicate logic surrounding calls to Foundation services in the BFFs was moved to the Tracks service, which now took care of synthesizing track aggregates for the BFFs, in addition to handling context-specific track visibility and authorization. The BFF was then responsible for mapping those internal track aggregates to external representations for the clients to consume.

Of course, nuances in how the BFFs behave in fetching tracks remains, but all shared code now lives in a singular codebase. Integrations requiring track aggregates are now as simple as querying endpoints exposed from the tracks VAS, removing the need to reorchestrate calls to Foundation services, and maintaining a guarantee that authorization is properly taken care of.

Evolution of VAS at SoundCloud

Now that we’ve explained the basis of VAS and how we integrated it into SoundCloud for tracks, we’d like to share how we adapted the same architectural pattern for the case of playlists. We’ll also outline the challenges we encountered during the process of evolving our architecture toward a VAS landscape.

As we already discussed, in 2019, we started the development of a Tracks service using the concept of a VAS. That was the first implementation of such a concept in the organization, and it helped us validate a model to apply to the rest of our entities. In 2020, we began a major refactoring of our Public API. The codebase of the Public API was divided between the Public API component of the Mothership monolith and the Public API BFF, which is a facade of the Public API. The refactor involved migrating all endpoints from Mothership and rewriting them in the Public API BFF.

Rewriting all track-related endpoints (all the endpoints that were returning the representation of tracks) was an easy task, as we already had a Tracks VAS up and running, so we just needed to connect the Public API with the Tracks service.

However, we also needed to rewrite all the playlist-related endpoints, but we didn’t have such a VAS. So we had to decide whether to duplicate existing authorization and fetching logic from other BFFs and move it to the Public API, or to create a new Playlists VAS to be the central service to handle fetching playlists logic and to make the Public API BFF dependent on it. The latter solution was, for us, the most attractive, as it would require refactoring and cleaning up the rest of the BFFs to also use the Playlists VAS to handle the many different playlists-related endpoints.

The following graph illustrates the process of migration:

As you can see, the main SoundCloud APIs were calling directly to the Mothership (the original SoundCloud app which is the main interface to our databases) and to Playlist Authorization (a system that authorizes users according to business rules of tracks in a playlist). This summarizes the two main problems of this approach: duplication of logic in the BFFs, and fragile authorization.

The following graph shows our solution:

With this architecture, all the logic is centralized in the Playlist VAS.

This project was divided into the following steps:

Extracting the logic from the BFFs and creating a new Playlist VAS. We investigated and documented all the different implementations of playlists logic in our main services to come up with the base logic for playlist-fetching logic. Once we documented all the information needed and collected feedback from the rest of the organization, we implemented the service.
Automatic tests to ensure that centralized logic matches refactored services. We added unit tests to ensure that we covered all the possible scenarios of fetching and authorization logic for playlists. We also added integration tests to ensure that response formats with clients weren’t broken, and that integrations with dependent services were working properly.
Migrating the BFFs to use the Playlists VAS. This might seem like an easy step, but it actually took us longer than implementing the service itself. We had to migrate more than 50 playlist-related endpoints from the main BFFs. We did this endpoint by endpoint, and by carefully comparing responses from both implementations, as we didn’t want to break anything from the clients of the BFFs.

What We Learned

VAS comes with many benefits, and it was a logical solution for the problems seen with playlist resolution at SoundCloud. The first and perhaps most obvious mitigation was that a Playlists VAS reduces duplicated playlists code in the BFFs. Each BFF contained the same logic for orchestrating various calls to dependent services for resolving playlist metadata, but with a VAS, we could congregate this logic in one service. Having a centralized service containing business logic means refactoring and optimizations are easier and faster. This also helps simplify cross-platform feature development, as value objects — in this case, playlists — are only exposed in a central place. So it becomes trivial to the rest of the clients to access playlists instead of reimplementing that functionality in each BFF.

One of the most critical parts of fetching playlists was authorization: Leaking private or unreleased tracks to the general public is a worst-case scenario we want to avoid at all costs. Authorization logic was spread out over multiple services, increasing the risk of inconsistencies (and therefore vulnerabilities) sneaking in over time. The Playlists VAS means having one central codebase for authorization logic, which can be audited easily.

It’s worth noting here that VAS-to-VAS communication can occur in some circumstances. Let’s say, for example, that a request is sent over to the Playlists VAS to add a track to a given playlist. Before we continue with this write command, we must first check that the track is visible to the session user. Authorization logic for entities is centralized in the VAS, so we’d have to make a request to the Tracks service to determine whether the requested track is authorized to be added. In this case, we ensured that the previous logic for tracks that has depended on playlists was decoupled, meaning we can have VAS-to-VAS communication without circular dependencies.

Considerations

The migration of entities to their own services has as many benefits, as outlined above. However, implementing this solution has downsides as well, including:

Adding a new service to our systems. Creating a new VAS comes with maintenance costs and infrastructure costs. We need to provision new nodes, deploy the monitoring service, maintain a new codebase, add a new service to our on-call rotation, etc.
Network latencies. In previous implementations, when BFFs wanted to fetch playlist metadata, they just needed to do a round trip to Mothership and Playlist Authorization. With this new implementation in place, they need to do an extra roundtrip to the Playlists VAS.

In addition to considering downsides, we also considered the alternate solution of having the Playlists VAS as an external library that would be linked during the app build process. This is an approach we used in the past — for instance, with our JVMKit library (you can read more about that here). However, this solution comes with extra complexity and risk, in that different versions of the library could be used in each service, potentially causing a state with diamond dependencies. In our case, the benefits of introducing a VAS far outweigh the tradeoffs.

It’s also worth mentioning that the approach of using a VAS means we can expose new integrations using Twinagle, which is an in-house implementation of the Twirp protocol for Finagle. You can read more about the motivation and benefits behind Twinagle in our related blog post, but one bonus is the added ease of integration and maintainability. Twinagle uses an interface description language (IDL) to generate server stubs and clients, meaning that integrating services have a clearly defined API to work with. Making changes to the VAS therefore becomes safer, given that the API contract stays consistent and will clearly expose any (backward compatible) changes made, with all implementation details being encapsulated behind the API.

Summary

The adoption of Value-Added Services at SoundCloud has been well received. It allows for a cleaner architecture and a better separation of concerns. We’re definitely going to move more entities to their own VAS and extend existing ones to have more operations, all of which allows us to have a clear roadmap for the architecture of our microservices.

In the next blog post — and the last one in this series on the evolution of Service Architecture at SoundCloud — we’ll talk about the next iteration of Value-Added Services and how they evolved into Domain Gateways.

Service Architecture at SoundCloud — Part 1: Backends for Frontends

Thu, 29 Jul 2021 00:00:00 GMT

Backends for Frontends

SoundCloud pioneered the Backends for Frontends (BFF) architectural pattern back in 2013 while moving away from an exhausted model of an eat-your-own-dog-food approach. The exhausted model involved using a single API (the Public API) both for official applications and third-party integrations. But the need to scale operationally and organizationally led to a migration from a monolith-based architecture to a microservices architecture. The proliferation of new microservices, paired with the introduction of a private/internal API for the monolith (effectively turning the monolith into yet another microservice), opened the door for new and innovative dedicated APIs to power our frontends. Thus, BFF was born, and it was really exciting, as it enabled autonomy for teams — along with many other advantages that will be discussed shortly.

In a nutshell, BFF is an architectural pattern that involves creating multiple, dedicated API gateways for each device or interface type, with the goal of optimizing each API for its particular use case.

Plenty has been written about BFF, the theoretical advantages that it provides, and how it compares to other approaches and technologies like centralized API gateways and GraphQL. However, little can be found on real-life experiences, risks, and tradeoffs encountered along the way, so we decided to write this series to shed some light on these topics.

BFFs at SoundCloud in 2021

SoundCloud operates dozens of BFFs, each powering a dedicated API. BFFs provide API gateway responsibilities, including rate limiting, authentication, header sanitization, and cache control. All external traffic entering our data centers is processed by one of our BFFs. Combined, they handle hundreds of millions of requests per hour.

BFFs make use of an internal library providing edge capabilities, as well as extension points for custom behavior. New library releases are semi-automatically rolled out to all BFFs within hours.

Some examples of BFF include our Mobile API (powering Android and iOS clients), our Web API (powering our web frontends and widget), and our Public and Partner APIs.

BFFs are maintained using an inner source model, in which individual teams contribute changes, and a Core team reviews and approves changes based on principles discussed in a collective. The Collective, organized by a Platform Lead, meets regularly to discuss issues and share knowledge.

The Good

One of the key advantages BFFs provide is autonomy. By having separate APIs per client type, we can optimize our APIs for whatever is convenient for each client type without the need for synchronization points and difficult compromises. For example, our mobile clients tend to prefer larger responses with a high number of embedded entities as a way to minimize the number of requests and to leverage internal caches, while our web frontend prefers finer-grained responses and dynamic augmentation of representations.

Another advantage of BFFs is resilience. A bad deploy might bring down an entire BFF in an availability zone, but it shouldn’t bring down the entire platform. This is in addition to many other resilience mechanisms in place.

Additionally, high autonomy and lower risk lead to a high pace of development. Our main BFFs are deployed multiple times per day and receive contributions from all over the engineering organization.

Finally, BFFs enable the implementation of sometimes ugly but necessary workarounds and mitigation strategies (a client bug fix affecting specific versions) without affecting the overall complexity of the platform.

The Bad

BFFs provide many advantages, but they can also be a source of problems if they’re not part of a broader service architecture that’s able to keep complexity and duplication at bay.

In service architectures with very small microservices that do little more than CRUD, and with no intermediate layers between these microservices and BFFs, feature integration (with all the associated business logic) tends to end up in the BFFs themselves. Although this problem also exists with other models, like centralized API gateways, it’s particularly problematic in architectures with multiple BFFs, since this logic can end up duplicated multiple times, with diverging and inconsistent implementations that drift apart over time.

This issue becomes critical for authorization rules that can only be applied at integration time (for example, because the necessary pieces of information required to make a decision are spread across multiple microservices). This model obviously doesn’t scale with the addition of more and more BFFs.

At SoundCloud, this problem manifested as the Track and Playlist core entities grew and were decomposed into multiple microservices serving parts of the final representations assembled in each of the BFFs. Suddenly, the authorization logic needed to be moved to the point of integration, which, at the time, was the BFF. This was not too concerning at first, with just a handful of BFFs and very simple authorization logic, but as the logic grew in complexity and the number of BFFs increased, it caused many problems. This will be the focus of the next posts in this series.

The Ugly

Effective operation of multiple BFFs requires a set of platform-wide capabilities that, in their absence, might lead to an unnecessary proliferation of BFFs. For example, application entitlements (in addition to user entitlements) are needed to restrict access to certain applications and third-party integrations to specific endpoints. In their absence, it’s tempting to spawn an entire new BFF for narrow use cases with specific access control requirements. There needs to be a strategy to decide how many BFFs are too many and when to create one versus when to reuse an existing one. Even though BFFs are designed to provide autonomy, there’s a tradeoff between autonomy and added maintenance and operational overhead that needs to be carefully managed.

We’ve also seen a tendency to push complex client-side logic to the BFF. This stems from the initial idea that a BFF is an extension of the client, and that therefore it should be treated as “the backend side of the client.” This has worked well in some cases, but in others it has led to problems. For example, pushing pagination to the server (recursively paginating to return an entire collection in one single request) — even though faster for basic use cases — can lead to timeouts, restrictive limits for collection sizes, and fan-out storms that may bring the entire system down.

Although BFFs enable some form of autonomy, it’s also important to recognize that BFFs are at the intersection of two worlds, and the idea of full autonomy for client developers is just an illusion. Extensive collaboration between frontend and backend engineers is required to ensure optimal API designs that are convenient for client developers to use, in addition to being optimized for distributed environments and their intricacies.

Summary

Backends for Frontends is an architectural pattern that can lead to a high degree of autonomy and pace of development. Like all engineering decisions, it comes with a set of tradeoffs that must be well understood and managed. In particular, a good service architecture is critical for scalability, security, and maintainability, and there are limits to how much autonomy can be achieved.

In future posts, we’ll dive into some of the unintended consequences of using the BFF pattern and discuss how our service architecture has evolved to address them. Stay tuned!

How We Share Knowledge as a Web Collective

Wed, 14 Jul 2021 00:00:00 GMT

There’s no single platform team that consists of only web engineers at SoundCloud, even though we consider ourselves to be part of the “Web Collective.” Instead, all web engineers are split into smaller, cross-functional teams across the organization. This structure definitely has its advantages, but it’s also left engineers in a situation where platform-specific experience and knowledge about platform-specific best practices are more difficult to access than they would be if we were all on the same team. New employees in particular can have difficulties breaking into unfamiliar tech stacks and SoundCloud internal frameworks.

We did and do still have forums to discuss technical challenges at SoundCloud to share knowledge across multiple teams: Technology Open House is a good example. This is where engineers from the entire organization come and learn from each other every week. While we enjoy these meetings, we as the Web Collective wanted to try out a slightly different format with topics dedicated to our platform. More specifically, we wanted to reduce the pressure to present as much as possible and instead give space to unprepared thoughts and raw ideas. We encourage all questions, and the environment fosters a culture of being able to ask questions that might feel silly or trivial to the people asking them.

We tried it out, it immediately felt natural to us, and it turned out to be extremely helpful, so we wanted to share a bit about the structure and format and our observations.

Meeting Structure

The Web Collective Knowledge Sharing meeting is an hour long, and it takes place every Friday at 16:00. This is one hour before our company-wide demos or all-hands sessions that we have on Fridays as well. Fridays are usually also days where developers take their self-allocated time (SAT) to work on their own projects, and since most of our web developers are based in Europe, the majority of us are winding down for the weekend at that point. This allows for a pretty relaxed atmosphere.

Even though this session primarily focuses on web topics, we occasionally also have non-web developers join to share more about web projects they’ve been involved in. This is because it’s important to us that the guest list stays as open as possible and that everyone feels welcome, no matter which platform they primarily work on.

People who are presenting don’t have to follow a specific format. We do this to keep a low as possible barrier of participation. Some people like to have slides prepared, while others like to use only some prepared browser tabs, or don’t prepare at all and just run through a piece of code they want to share. We found that by allowing low-key presentations, more people are presenting. There are also no limitations on time other than the one hour allocated for the meeting. (Of course, people shouldn’t and don’t use the full available hour when they know that there are other topics on the agenda.)

We often find ourselves using most of the time for the meeting, even for days with a light agenda. The Web Collective is pretty good at turning a single question into an hour-long discussion, albeit one that’s both enjoyable and valuable.

Speaking of agenda, there’s no fixed agenda for the sessions, and topics are often proposed in the meeting itself. We do keep a document with proposed sessions, and it’s good practice for presenters to add their topics to the agenda, but it’s also totally OK to jump the queue in case someone has an urgent need to (e.g. they’ll go on vacation afterward or the topic has a time constraint). These lax rules around the agenda also add to the casual environment of the meeting.

Things We Do in Knowledge Sharing

The flexibility of the meeting has allowed us to morph it into whatever we needed it to be.

One thing we regularly do is review PRs together. “I worked on this pull request the other day, let me show it to you…” or “I saw an interesting pull request, can somebody walk us through that?” is something we hear a lot during Knowledge Sharing. We find this really helpful for more complex PRs, because sometimes the collective spots details that other reviewers missed.

Every so often, Knowledge Sharing also means introducing the collective to new tools or codebases: Sessions that start with “We created a new app and integrated it into soundcloud.com, here’s how it works…” or “We have a new shared component library. It’s in its early days but here’s where we are right now…” are helpful for everyone to stay up to date on what’s new.

We also had times where we talked about processes and how we could improve them. A few months ago, for example, we were getting bombarded by automated alerts on Slack, and an engineer kicked off the session with “I feel like alerts are getting too noisy and out of control. Should we appoint a point person to deal with them?” Another conversation about how we do releases started with “Here’s an open source tool that could simplify our release processes. What do you think?”

Other times, we use Knowledge Sharing for technical deep dives. For example, an engineer who worked on our webpack config began a conversation with “I had to dive into how code splitting works for soundcloud.com and I can now explain it!” Other times, an engineer who worked on our famous waveform started with “We are making changes to the waveform and I found that it’s really well architected. Here are interesting patterns that we use there…” And another one who was profiling React went with “I measured the performance overhead of rendering many React roots and here’s what I found…”

Questions and open rounds are frequently on the table when the agenda is empty. In these cases, Knowledge Sharing is used as an opportunity to seek out expertise and feedback from others. For example, we recently had meetings that started with “It looks like we’re always running into issues with feature flags. What can we do to fix them?” and “Can anybody explain how tracking works? How exactly would I implement firing a click event?”

Observations

We’ve observed that developers have adapted a sharing mindset and are actively sharing their work and asking peers to share in this new forum. We’ve talked about a vast variety of topics, which results in an increase of the bus factor of the team.

With our current remote work setup, this meeting also helps make the platform team feel more connected. There are few opportunities to connect to platform peers in such a laid-back environment, and even before we started to work remotely, this meeting was one that web peers always looked forward to, making it even more important now.

Another thing we noticed is that the barrier to asking “basic” questions has gone down; people aren’t afraid to ask questions they might otherwise be scared to ask in other scenarios. And with engineers from all the different engineering levels present at the meeting, it has become a valuable forum for getting answers to these questions.

What’s Next?

There are some points that we want to improve upon, though we haven’t yet found the best solutions to these issues yet.

For example, sometimes it’s unclear to peers if they should have a discussion on Slack or whether they should propose a topic for the next Knowledge Sharing session. To date, we haven’t found a great guideline for this. However, when someone asks if a discussion should happen in our Knowledge Sharing meeting, it’s a great indication that it should be discussed there as well, even if it’s just in the format of a summary of the Slack discussion.

And this is a great segue to another issue: We haven’t yet found a good way to document the outcomes of these sessions. Semi-randomly, colleagues add their notes to the agenda document, but a lot has already been “lost.” We should definitely take on a better documentation cadence: Maybe declaring a dedicated note-taker for each meeting or encouraging presenters to sum up their findings in the agenda document afterward would be a good start. However, we don’t know if this would be detrimental to the casual environment of the meeting. But we’re sure we’ll find a good balance there.

Nevertheless, we’re very happy with the current setup of the meeting, and we plan to keep our weekly cadence to share knowledge.

If you have a similar setup at your company, we’d love to hear about it to understand what has worked best for you.

Photo by Papaioannou Kostas on Unsplash

Upcoming API Security Updates — Action Required

Thu, 01 Jul 2021 00:00:00 GMT

As part of our continuous effort toward making improvements to our API with the hope that we can relaunch API access to all developers, we’re making some critical security improvements. Please note that these are important and time sensitive and may cause disruptions in your app if you don’t make the necessary security upgrades.

TLS Enforcement — HTTPS Everywhere

Going forward, all traffic will be served via TLS. If your application doesn’t support TLS or cannot follow HTTP 301 redirects automatically, you should update it. This is effective immediately.

Move OAuth Tokens from URL to Header

While we currently support clients using oauth_token as a query parameter per the OAuth 2.0 RFC 6750, this poses a few security challenges. We’re stopping support for this to encourage users to provide the authentication header instead, and it’ll be in effect soon.

Here’s an example of sending an access token in a request header:

curl --request GET \
--url 'https://api.soundcloud.com/me/tracks?limit=1' \
--header 'Authorization: OAuth ACCESS_TOKEN'

Deprecated Password Grant

The password grant type will be deprecated. We recommend using authorization_code for client-side integrations and client_credentials for server-side integrations.

Changes in Authorization Code Flow

For an authorization_code grant type, we’ll only support one response type: code. Token and code_and_token are deprecated and will be considered invalid.

Use Client Credentials Grant for Server-Side Integrations

Currently, to access the public resources of the platform, server-side integrations with our API only require a client_id in the URL’s query parameter. We’ll be strengthening our authorization here by making all public resources on the API only accessible to apps that have been authorized with the client_credentials grant. This will enable the app to capture both the access_token and the refresh_token to then fetch the resources from the API. Please note that the use of client_id will be deprecated and deleted soon (from July 2021). Developers should provide the Authentication header for all their requests to the SoundCloud API going forward.

Here’s an example of getting an access token via the client_credentials grant type:

curl --request POST \
--url https://api.soundcloud.com/oauth2/token \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data client_id=CLIENT_ID \
--data client_secret=CLIENT_SECRET \
--data grant_type=client_credentials

Expiring Tokens Are Issued by Default

All access tokens issued by our servers will be expiring by default with the TTL set to 6 hours. If you’re currently using non-expiring tokens, you should make the necessary updates in your app to handle the refresh_token grant.

Here’s an example of getting an access token via the refresh_token grant type:

curl --request POST \
--url https://api.soundcloud.com/oauth2/token \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data client_id=CLIENT_ID \
--data client_secret=CLIENT_SECRET \
--data grant_type=refresh_token \
--data refresh_token=REFRESH_TOKEN

For future updates, please follow us on Twitter and on our blog.
For any bugs or feature requests, please visit our GitHub Issue Tracker.
You can also follow our API Release Notes page to stay up to date with minor changes.

Did I Break You? Reverse Dependency Verification

Tue, 25 May 2021 00:00:00 GMT

SoundCloud was founded 13 years ago, and throughout its history, the company and much of its tech stack has changed. We started with a monolithic Ruby on Rails app, and since then, have worked to extract, isolate, and reuse logic in many subsystems, and later on in separate microservices. We introduced different storage mechanisms and data analysis apps, and our frontend has also expanded from a single website to mobile apps, integrations with third parties, an Xbox app, and more recently, a full-fledged PWA to run on Chromebooks.

Golden Paths

Internally, we’re committed to giving teams flexibility to pick whatever tools they deem best to solve the technological challenges of everyday product development. While this means a lot of autonomy, it also gradually led to a fragmented environment of languages, frameworks, processes, and documents, making cross-team collaboration and onboardability harder. We have, thus, moved into a model of providing sensible recommendations and supporting an opinionated tech stack.

In that sense, complexity became opt-in: In general, libraries, tooling, and the engineering experience should be as simple as possible, with no configuration, but still provide the ability to support more complex use cases, depending on each problem-requirement set.

With time, we identified some similarities in the implementations of our solutions and naturally repeated choices. Engineers experienced with a given programming language were more likely to pick the same for a following project, while in other cases, the ease of integration with, say, a specific database engine pushed a new integrator to decide for it again. We built more tooling to help develop these existing apps, we wrote more documentation to cover the “gotchas” of a framework, standards arose to organize the system architecture, etc.

This cycle ended up consolidating some ideas as de facto Golden Paths for our different disciplines: web development, data science, infrastructure, etc. Engineers are still free to choose their tooling, but the positive reinforcement cycle was already set: Why spend time reinventing the wheel when lots has already been invested in building mature solutions? Drifts from the Golden Paths exist, but they’ve become increasingly rare.

JVMKit

We settled on the Golden Path for developing backend applications in the JVM ecosystem. Since adopting Twitter’s Finagle, we’ve also built extensive tooling on top, up to a point in which we could derive a library of our own: JVMKit, which helps building JVM applications by providing an implementation of common patterns and protocols used at SoundCloud.

Because Scala applications make up the vast majority of apps at SoundCloud, JVMKit mainly targets them, but it should be possible to use from any other JVM language as well — we currently run services in Java, Clojure, and JRuby as well, even though we try to steer everybody toward the Golden Path. It provides opt-in modules for serving public traffic in the shape of BFFs, as well as integration with MySQL, Kafka, Memcached, and Prometheus, among other functionality.

With such a broad audience, it’s no surprise that JVMKit was largely adopted across SoundCloud, and it powers much of what we provide to our users — it became a critical component of our technology stack, with hundreds of services depending on it and thousands of instances running it in production.

JVMKit’s repository sees active development: New features, bug fixes, and security patches land every week. Different from an open source project, in which dependent projects are unknown, within the SoundCloud organization, we can draw a transparent view on all callers to JVMKit’s APIs to see which apps are lagging behind and which are on the straight cutting edge.

We then ask ourselves: How exactly can we make sure the entire company follows JVMKit’s release cycle? As a platform team and developers of JVMKit, we want to minimize the disruptions to each team’s roadmap, onboarding into new APIs, replacement of deprecated calls, etc. In other words, we want to minimize the work we put on our colleagues’ plates so they can focus on higher objectives. For that, we have built automation to perform batch upgrades across all of our repositories — ideally, JVMKit upgrades are completely transparent.

Reverse Dependency Graph

To identify which services depend on JVMKit, we rely on these relationships being explicitly defined in terms of code. As part of the previously introduced concept of the Golden Path for backend development, we do that through the build scripts that are part of each repository. For the majority of our backend repositories, this means sbt builds. The same idea is applied to our other Golden Paths, with Gradle on Android and npm for web development, for example:

val jvmkitVersion = "15.0.0"

lazy val root = (project in file("."))
  .settings(
    libraryDependencies ++= Seq(
      "com.soundcloud" %% "jvmkit-admin-server" % jvmkitVersion,
      "com.soundcloud" %% "jvmkit-json-play" % jvmkitVersion,
      ...
     )
  )

Every five minutes, changes to the default branch of each of SoundCloud’s repositories hosted on GitHub are indexed by Zoekt, a search engine based on regular expression matching (read this article for more in-depth information). Out of the box, Zoekt provides a web interface that we then make available to all SoundCloud engineers, but we also added the ability to query its results through a simple scripting API.

By combining Zoekt’s API and the standardization across the company, we can write a simple query like "jvmkit" f:^build.sbt and list all the projects that depend on JVMKit — in other words, we can build a level of the reverse dependency graph in which JVMKit is the root.

Verification

With these tools at hand, we can now start thinking about how we can automatically verify whether a change in JVMKit will integrate smoothly with other projects. For the sake of the example, let’s take an arbitrary service called likes, which is responsible for managing which tracks each user liked on SoundCloud. If we were to write such an algorithm in pseudocode, this is how it would look:

1. Clone "likes" repository
2. Find build.sbt
3. Replace the jvmkitVersion value with the latest snapshot
4. Compile "likes"

If all steps in the algorithm succeed, we guarantee that the new changes in JVMKit didn’t cause issues with likes. We can generalize this idea and run the algorithm for each and every repository we previously identified as reverse dependencies of JVMKit — if all of them succeed, the new JVMKit version is good to go!

Our work is done, right? Not really. There are other cases that our algorithm currently doesn’t consider. For instance, what happens if we were planning to release a JVMKit version and coincidentally on that day, the main branch of likes was broken? We shouldn’t consider it the fault of that JVMKit version’s bump. To isolate that scenario, we can compile the target project with no changes whatsoever.

If JVMKit’s new version introduces an API-breaking change — requiring, thus, a major version bump — we allow for the engineers running the upgrade to automatically fix the breaking change by executing a set of processors directly in the repositories source code with tools like sed, awk, lint, etc.

Now, when the release is a minor or patch change, surely the project will compile, but does it behave as we expect it to? We should also run automated tests to ensure our business logic is still respected. Once again, here we can leverage the standardization provided by the Golden Path: All Scala projects that are built with sbt will execute their tests with the same command: sbt test.

Finally, the pseudocode presented earlier can be extended to become:

1. Clone "likes" repository
2. Compile "likes"
3. Run tests on "likes"
4. Find build.sbt
5. Replace the jvmkitVersion value with the latest snapshot
6. Run source code processors for automatic API fixing
7. Compile "likes"
8. Run tests on "likes"

With this, it becomes clearer who the culprit for a failure is, depending upon which step of this sequence the algorithm stops. We’re also more confident of a smooth integration if it completes successfully.

Taking It a (Pipeline) Step Further

Instead of going through the described process manually and in an ad hoc fashion every time we decide to release a new build of JVMKit, we can develop the automation even further and leverage a continuous integration system to perform the checks for every single commit in the library repository.

As mentioned earlier, at SoundCloud, we have hundreds of projects that rely on JVMKit, which makes compiling and running their tests a time-consuming task. To speed up JVMKit’s development and feedback loop, we decided to split the automation into two:

A nightly, long-running pipeline that verifies the new JVMKit against all projects in the organization
A faster pipeline, limited to a small allowlist of projects, that runs on every pull request

This is achieved through scripts that fetch project lists from Zoekt and translate them into pipeline descriptors that can be ingested by our CD system. The generated pipelines include steps to run tests, depending on whether these exist on each of the target projects.

On top of that, in the real world, not all projects are well behaved. There are different reasons we might want to skip specific steps of the verification algorithm: when the test suite takes too long, if it’s known to be flaky, or when the project needs a specific environment configuration or special dependencies that aren’t provided out of the box. For these cases, we also introduced a denylist that serves as input to the CI pipeline generator script.

Summing It All Up

Observing the running pipelines is an engaging activity. Every day, we can have a view on how our projects will behave when moving forward, and when failures occur, it’s also easy to pinpoint reasons: If a single project failed due to an API change, we can dive into possible shortcomings of its implementations or uses of JVMKit’s APIs that weren’t originally intended. If many (or all) projects fail, then it’s obvious that the issue is with JVMKit itself, and we can quickly act on providing fixes.

We’ve reached an interesting point in our development practices for JVMKit after years of investment: With good tooling and automation in every step of the process, we’re confident when releasing a change to the library that powers dozens of teams, hundreds of systems, thousands of instances, and millions of users. We’re fearless about releasing JVMKit more often than ever with no impact or disruption to our team’s planned roadmap, while also ensuring technological standardization across SoundCloud.

High-Tier Content in the SoundCloud API

Thu, 06 May 2021 00:00:00 GMT

You can now retrieve metadata for our full SoundCloud catalog through our API by providing the new access filter in your request.

Until recently, high-tier content (Go+ content) wasn’t served from our API. This removed the ability for clients to decide how to present restricted or blocked content, i.e. high-tier content. With this change, clients are now able to fully control the user experience:

https://api.soundcloud.com/playlists/833426378/tracks?access=playable,preview,blocked&linked_partitioning=true

In the above example, the access parameter indicates which level of playback accessibility for a track is available to the requesting user. There are three possible values supported:

playable — The user can listen to a full track.
preview — The user gets a snippet of a track (Go+ content for free users).
blocked — The user isn’t allowed to play the track but can see the metadata.

You can specify any number of allowed values depending on your use case.

For instance:

access=playable will return only tracks the user can listen to.
access=playable,preview will return all tracks that the user can either play fully or preview the snippets of.
access=playable,preview,blocked will return metadata for all tracks — even the blocked ones the user cannot listen to due to various restrictions.

If no filter is provided, the endpoint falls back to the default behavior and provides the content with the access=playable,preview value.

All track entities are now also annotated with this new field to indicate the level of playback accessibility available to the user:

{
  "artwork_url": "https://i1.sndcdn.com/artwork.jpg",
  "id": 123456,
  "isrc": null,
  "kind": "track",
  "permalink_url": "https://soundcloud.com/userPermalink/trackPermalink",
  "stream_url": "https://api.soundcloud.com/tracks/123456/stream",
  ...
  ...
  "access": "playable"
}

For an overview of full use and all the supported endpoints, please refer to our API specification.

Encountered an issue? Report it on the issue tracker.

For future updates, please follow us on Twitter and our blog.

The Journey of Corpus

Thu, 29 Apr 2021 00:00:00 GMT

It seems like a simple enough concept: You take data from how your users interact with your product, and you use it to make business and product decisions. Informed decisions will certainly be better than guesses, so in the end, this should pay off.

But that’s for anyone who hasn’t seen the dirty underbelly of data — from how broken tracking can lead to incorrect conclusions, to having lots and lots and lots of data but no documentation around what any of it means. If you’ve worked a day in your life as an analyst/data scientist/data engineer or in any other data-heavy role, you know how bad things can get.

In mid 2019, we at SoundCloud were in a position many companies have probably found themselves in: Our stakeholders wanted to make use of the wealth of data available to us, as we’re the world’s largest open audio platform, but access to data was hard, since our data warehouse was the accumulation of six years of decisions that made sense at the time but naturally grew outdated over the years. As a result, we just weren’t able to support the business and product teams in the data-driven decisions they wanted to make.

More concretely, a single (small) team owned more than 100 ETLs (extract, transform, load processes) on Amazon Redshift — most with little to no documentation and some inconsistencies between them (they were, after all, built at different points in time, by different people, for different purposes). However, we had signed a contract with Google to switch over to BigQuery (BQ) as our new data warehousing platform. This was our opportunity to start fresh and build something that would hopefully fulfill SoundCloud’s needs at the time and scale with time.

Corpus Is Born

So we set out on our journey to create what would become the Corpus BQ project: a centralized single source of truth for SoundCloud’s most relevant data. To do this, we created the Data Corpus team, whose mission statement is to “Implement data governance on the datasets that are most critical for product and business decision making.”

And just what exactly is data governance? According to Wikipedia, data governance is “a data management concept concerning the capability that enables an organization to ensure that high data quality exists throughout the complete lifecycle of the data, and data controls are implemented that support business objectives.” This was what we needed to become a more data-driven company.

Corpus as a team sits somewhere in the middle of what most people think of as Data Science and Data Engineering. At SoundCloud, there’s a dedicated team of data engineers known as Data Platform, which is responsible for maintaining all the infrastructure around data. We like to think of it as “Data Platform engineers the infrastructure, we engineer the data.” Our job titles are still Data Scientist or Data Engineer, depending on each person’s background and strengths, but we’re closer to what some companies now call analytics engineers.

Corpus Principles

After defining a team mission, we needed to decide how we were going to implement data governance. We came up with this list of six principles, which will be covered in detail in the rest of this post:

Quality
Compliance
Timeliness
Usability
Efficiency
Maintainability

This list is based on an ISO standard that we adjusted to our needs by removing characteristics that were outside our responsibilities and adding “maintainability,” for reasons we’ll explain below.

Quality

Quality is defined as “providing a consistent and accurate source of truth.” The bottom line is: Our stakeholders need to be able to trust our data, or they won’t use it. Exactly how we achieve this could be the topic of its own blog post, since it involves several different strategies. Here are a couple examples:

Whenever we perform a change on an existing ETL or table, we create a new version of it and compare it against the currently deployed version. We then use an automated tool we built to validate that the two versions are exactly the same, with the exception of the changes we introduced.
All our ETLs have data quality checks that run whenever new data is added to a table. These range from cardinality checks (e.g. “Does that table that’s supposed to be one row per user really have one row per user?”) to checking the input versus the output data (e.g. “Did all the clicks in the input make it into the output?”).
We have an outlier detection algorithm (based on Tukey’s interquartile range method) that runs daily and warns us of any unexpected data observations. We then investigate these outliers and notify stakeholders if there’s anything of note.

Compliance

As required by law, we make sure we comply with the appropriate regulations for data, such as GDPR. This means, for instance, that we don’t provide any personally identifiable information (PII) in any of our resources. If there’s a need for this sort of data, there are established internal procedures to go through that don’t involve our team.

Timeliness

Internally, we define timeliness as having our daily updates finished by 10AM CET. In actuality, most days, they’re finished before 8AM CET. This is important because it ensures the data is available by the time our stakeholders start working.

We ensure we meet this deadline by working on both the efficiency of our ETLs and the timeliness of their dependencies. When analyzing our overall timeliness, we map out the landing times of both our ETLs and their source data. We then work on improving our code to make it run faster, and we work with the teams that provide the source data to ensure there are no bottlenecks.

One example of an easy win: One table we needed had its completion unnecessarily coupled with the completion of another table we didn’t actually need, and which took much longer to complete than the first table. We worked with the owners of these tables to decouple them, and we removed four hours from our pipeline delivery time in one fell swoop.

When we fail to meet our 10AM deadline, which naturally happens from time to time, we get notified automatically, and whoever is on call investigates and works on solving whatever the issue is. If the issue isn’t quickly solvable, the matter is escalated to an incident and becomes an engineering-wide priority to resolve.

Usability

We define this principle as “providing easily accessible and consumable data.” One of the main problems with our old setup wasn’t that we didn’t have enough data; it was that it was very hard to know when to use what and how. As such, with Corpus, we set out to build (as much as possible) a self-service platform to provide resources that can be used by all our stakeholders, as long as they have basic SQL skills. There will always be the necessity for certain analysis to be done by experts (data scientists/data analysts), but this shouldn’t be the case for basic data needs.

We ensure usability by focusing on the following:

Descriptions — This might seem too simple to have a real impact, but it’s one of the features our stakeholders most appreciate. There’s no need for them to ping around between Slack channels asking what the meaning of a certain field is — it’s all explained in the same place they get our data from, in the BQ UI. We even add at least one example query.
Design documents — These focus on documenting more advanced use cases (more complex example queries), data anomalies (there are always some), and context around certain design decisions. This is what our stakeholders dig into if their questions cannot be answered directly by the descriptions.
Ease of use of resources — We don’t want to build something that’s technically perfect but hard to use. Sometimes we sacrifice on storage or efficiency to make sure the table is still queryable without expert SQL skills. Other times, when the price to pay for usability is too high, we build a table with a schema that’s non-trivial (because it saves a lot of storage, for instance). But then we build a view on top that turns it into a simpler schema and only show that to stakeholders, allowing us to get the best of both worlds.
Providing training — We provide (and record) company-wide training both on SQL and on how to use the resources we provide.
Accessibility — The fact that we use a cloud solution (BigQuery) that’s ready to use out of the box means no one actually needs to install anything or ask for access to use any of our resources; they just need to access the BQ console by authenticating through their @soundcloud.com email address. Note that our data is available across the company because it doesn’t have any PII, as explained in the Compliance section above.

Efficiency

As hinted at by some of the other characteristics, we want to make sure we optimize execution time and storage, as these relate to both performance and costs. For this, we try to make the most of BQ’s features like sketches, nesting, and UDFs.

Getting proficient with BQ was something that took us some time, since it works quite differently from what we were used to before (Redshift). A few things helped with this learning curve:

The fact that we were able to use it for some time with the old tables before we needed to start building the new versions allowed us to get our feet wet before we began making the big design decisions. We also saw what worked and didn’t work about our old designs in this new platform.
We hired a consultant who was already an expert on BQ to help us with the migration. Their suggestions opened our eyes to features we didn’t know existed, such as sketches, which made a big difference once we started using them.
We acknowledge that BigQuery is in constant development and improvement, so we keep a close eye on new releases and even work with Google from time to time to try to identify the features that would be the most helpful to us.

Maintainability

Knowing the pain of maintaining something we didn’t build and had no documentation on (who doesn’t?!), we wanted to make sure we were building code that was easy to read, extend, and maintain, so we decided to make this into its own principle.

We accomplish this by focusing on three concepts:

The “80 percent principle” — We aim to create tables that respond to 80 percent of stakeholder requests, not 100 percent. For 100 percent, one will often get into too many edge cases and end up with an over-complicated setup for something that’s rarely used. As such, we don’t try to answer every single data question out there — only 80 percent of them. This principle, maybe more than anything else, has become a team staple that we turn to often in discussions.
Shared set of best practices — We maintain a set of data best practices covering the different tools we use and use cases we have. The idea is that there’s a lot of commonality between different team members’ work, which helps us have broad ownership of everything we do. We also do knowledge sharing about these practices with other teams.
Scalability — We’re well aware that products change, and we’ve seen SoundCloud evolve a lot over the years: Features are launched or deprecated, and stakeholders change, and so do business priorities. We know our data needs to evolve over time as well. As such, we try to build our resources in a way that’s easy to extend (e.g. adding a new metric doesn’t require altering a table’s schema). We also have well-established versioning and release processes, both of which allow us to evolve the data we provide without ever breaking service to our stakeholders (this could be a blog post of its own too).

One aspect of our ETLs that might surprise people is the fact that they’re all in BQ SQL. In the past, we owned ETLs in PostgreSQL and Spark/Scala, some with Python mixed in, and even one in Haskell. However, in BQ, every data pipeline we’ve encountered (so far) is actually simpler and more efficient to build using plain old SQL. We’re open to exploring other options — BQ can support many different setups. We just haven’t found a need for this, and we also greatly value the consistency and readability benefits we get from using a language that most of the company is familiar with.

The Finished Product (So Far)

Putting all of these principles into practice, we were able to build a BQ project that takes up 90 percent less storage space than the raw data it reads. In turn, that empowers anyone in the company to answer most (we are, after all, aiming for 80 percent) of their data questions.

If you’re wondering what the rest of our stack looks like, it doesn’t stop at BQ:

For scheduling our ETLs, we use Apache Airflow, a versatile scheduler that makes sure our ETLs (along with their checks) run every day — and we’re notified if something goes wrong.
For dependency management, we use an internal tool (built by our Data Platform team) called Datapub.
For the BQ project infrastructure, we use Terraform to manage access rights, dataset and table/view/UDF creation/deletion, and schemas.
For powering the automated data validation tool mentioned in the Quality section above, we use pytest.
For version control, the whole company relies on GitHub.

These are of course just our main tools; others will come up from time to time.

Final Notes on the Journey

It’s only fair to point out that we didn’t have such a clearly defined mission and principles when we were in the midst of putting the first version of the Corpus together and trying to get the team up and running. We started smaller and simpler, but we always had certain principles in mind (like the “80 percent principle”), which made a big difference at times when we faced tough decisions on which direction to take.

This is to say that it might seem like a daunting task to come up with such clearly defined scope and principles when you start a project like this, but you still shouldn’t skip the step of thinking about these things. The temptation to just get the work started and figure it all out as you go along might be high. For us, though, making sure everyone had clear foundations to fall back on paid off in the long run. When we didn’t have a pre-existing principle to rely on, we discussed which way to go and extended our principles to make sure it was covered for the next time it came up.

These principles also served many times as a first line of defense when outside teams or stakeholders tried to change our scope. Often, once we explained our thoughtful reasoning behind why we thought we should or shouldn’t do certain things, they’d agree with us.

However, we are of course open to changing these principles if there’s a strong enough reason to do so. One big lesson we learned from this migration (a reminder, really) was that just because a decision made sense at a certain point in time doesn’t mean it still makes sense five years later. We try to never lose sight of the fact that this current setup can easily become as outdated as the one we had pre-migration and, hence, we keep an open mind about possible changes.

Another aspect of this journey that might be easy to forget is that, when we started the migration, we planned for a first implementation followed by a refactoring. Instead of waiting for the perfect conditions for us to be able to do exactly what we wanted, we got started with what we had. This allowed us to gain crucial experience in tools that were new to us (BQ and Airflow) and immensely improve from the first iteration of Corpus to the second. As a matter of fact, the road to a Data Corpus that we could proudly say abided by all of these principles was actually quite long:

Early 2018 — There was a first attempt at migrating using a lift and shift approach, which was later aborted for a variety of reasons, but provided many precious lessons.
Mid 2019 — We started planning a new migration from scratch.
December 2019 — The first intermediate version of Corpus was released, still using the old data warehouse as a source in some places.
April 2020 — The release of Corpus Alpha, the first iteration that was truly independent from our old data warehouse.
June 2020 — The release of Corpus Beta, where we did a major refactoring of the code, unified table schemas, and saw big gains on storage and execution time.
July 2020 — The release of Corpus 1.0, the first version of Corpus that abided by all the principles described in this post.

We recently released Corpus 2.2 in April 2021, with Corpus 2.0 and 2.1 coming before that, and we’re now starting the work on Corpus 3.0. At this point, our release work is mostly focused on extending Corpus’ coverage by adding tables that cover new entities or metrics, which are driven by the natural evolution of the product itself. We also fix the few bugs that get discovered from time to time, which is an inevitable reality of building anything. Corpus really is a continuously evolving project, which is what makes working on this team interesting and fun!

If all of this sounds like an exciting challenge to you, then apply for the Data Corpus team!

Acknowledgements

None of this would’ve been possible without the work of many SoundClouders outside the Data Corpus team, namely:

The old Data Science Analytics team, which later got separated into the Data Science and Business Research teams. Many members of these teams contributed to the first version of Corpus when we weren’t yet a dedicated team. Nowadays, they are our stakeholders and help us prioritize what to work on next.
Data Platform team — In addition to providing much helpful guidance and support during the migration, this team is continuously ensuring we have the necessary infrastructure to do our work.
Content Authorization and Payments teams — Producers of data sources we consume that worked closely with us to meet our requirements.

How to Successfully Hand Over Systems

Tue, 20 Apr 2021 00:00:00 GMT

In a product company, changes are inevitable so as to best support the strategy and the vision. Often during such a change, new teams are formed and other ones are restructured. While there are many challenges to be solved during a big change, there’s one in particular that’s often overlooked: system ownership.

Who will take ownership of the systems that were owned by a team that doesn’t exist anymore or that are better suited to be owned by another team? It’s in everyone’s interest that the ownership be given to a team familiar with the system’s domain, so that they can continue the maintenance and evolution.

Regardless of when the system handover will happen, how it’s executed is important, since the cost of failure can be high, and that could result in an outage or a significant amount of unplanned work.

Having experienced not-so-successful handovers — some of which took place over the course of a one-hour meeting — I was inspired to create a guideline that will help other teams do handovers differently. At the same time, my colleague Antonio N. went through an ownership change with his team. This also had few mishaps, so we joined forces to write a proposal document for doing system handovers at SoundCloud.

We used the RFC approach to gather input, experiences, and opinions from the entire organization. It was welcomed with enthusiasm and since then has been used multiple times.

I’m posting the complete guideline below not only to document what we did at SoundCloud, but also in hopes of providing a template for other companies to use when faced with a similar scenario.

Guideline for Internal System Handovers

The guideline is a list of questions, tasks, and actions that the involved parties should consider as part of the system handover. The topics listed can be covered in different ways: through documentation, meetings, pairing sessions, workshops, tasks, PR reviews, etc. The goal is to help the new team understand the what, why, and how of the system, and to empower them to maintain, change, and improve it.

General Recommendations

As the system ownership change is a process itself, we recommend that it’s driven by the new team with the help and support of the previous system owner. Both teams should collaborate on the planning and execution of the tasks.

There will be some new documents and artifacts produced as an outcome of the system ownership change. We recommend storing them in the system’s repo when possible, or else including a link in the repo (e.g. from the README file). This will help with both onboarding new team members and potential ownership changes in the future.

Why Are We Changing the Ownership?

Help everyone involved in the handover understand why there’s a system ownership change. This impacts the team engagement in the handover process.

What’s best is to document the reasoning and to add additional information to the history of the system. In turn, this can reveal different things, such as if the ownership changed or underwent restructuring multiple times in a short period of time, or if the current organization isn’t set up to own such a system or if the system doesn’t belong to any team. Uncovering this information helps us ask important questions, such as is the system too complex, is it not in any team’s domain, or is it even needed anymore?

What Does the System Do? What Problem Does It Solve? What Is the Vision?

Here, we’re looking to understand the system from a product perspective. It’s helpful to know some history about the system, how it evolved, and what its vision for the future is.

This could be a product session where product managers are involved. As an outcome of this session, it’d be nice to have a document to help onboard new people to the system.

High-Level Architecture of the System

Get an overview of the system, the main components, and their interfaces. A more detailed diagram will probably lead to more detailed discussions. It’s best to have an online diagram so that it’s easily available for future reference.

Use, Availability, and Criticality of the System

Get familiar with who’s using the system, what the criticality of the system is, and what it means if the system isn’t available. This is an opportunity to look into the available runbooks, metrics, and monitoring.

In cases when the previous and the new team owning the system are not part of the same on-call rotation, there must be an additional session where the system is introduced and explained to the engineers in the rotation group, so that all of them can respond to incidents related to the system. This helps prevent bigger outages and maintain the healthy on-call culture.

Maintenance

As part of the maintenance of our system, we have daily, weekly, and monthly tasks that need to be executed by the team. In this section, we need to identify what those tasks are and what their periodicity is (if needed).

Data Storage Overview

Most of the systems have their own data storage. When taking ownership of a system, the new team takes ownership of that data and the infrastructure that comes with it. This is to get an overview of what, how, and where that data is stored and/or purged.

Batch (Offline) Jobs Overview

Some systems are using data that’s an outcome of a batch job. Also, many systems have batch jobs that produce datasets to be consumed for analytics or reporting. Get an overview of the batch jobs, their usage, outcomes, and maintenance.

Decision History

This is helpful to understand the architecture choices and the evolution of the system, as well as to learn about any constraints the system might have. It’s best if this is documented.

We recommend using Lightweight Architecture Decisions Records, which has the following format: Context, Decision, and Consequences.

Tech Debt

Make sure you’re aware of the existing tech debt and you understand its implications. This is best if it’s documented.

Known Bugs

When ownership changes, user-facing bugs will be reported to the new team; however, many of them might already be known. Make sure you understand them and why they’re present. This helps not only by decreasing the time to investigate, but also by providing a good service to our users.

To Dos

In addition to the above, here are some tasks (not in order and not complete, since they are SoundCloud specific) that can ease the ownership change:

Project ownership update on GitHub
Grant permissions to the new team
Update offline jobs configuration
Local development
- Can the engineers build the project locally?
- Does the system have integration tests? If so, do they run in a local environment? Does the new owner need any additional information to run them?
Deployment
- How is the system deployed?
- Is there a CI/CD?
Monitoring and Alerting
- Check if the monitoring graphs need to be updated
- Update the system-related runbooks
- In case the runbook location changes, please reflect that change in the corresponding alerts to avoid broken links from the alerts
- Add the system to a corresponding on-call group and have a knowledge transfer session
- Update alerts
- Update PagerDuty
- Our suggestion is to update the on-call rotation at the end, once the team has gained sufficient knowledge and confidence in the system

💡 The list of things to do is quite long. Take your time with each task and don’t rush. Use the help of the previous team, and pay attention to details and to the alerts.

💡 You can use your project management tool (e.g. JIRA) to track the progress of the handover. That will help both involved teams stay up to date on the status of the handover, the next steps, and when it will be completed.

💡 If you’ve discovered other helpful tasks or topics, please update the guideline with them.

Usage

As the name suggests, the above document is a guideline, and it’s up to the parties involved in the system ownership change to decide if they’re going to use it and how they’re going to use it.

In most cases, ownership change is a collaborative process that enables the new owners to be motivated and have a solid understanding of the system to maintain and continue evolving it. In some cases, it can happen that there’s no one in the company that previously contributed to the system. However, even then, this guideline can help the team keep the focus on topics that are important to know, and not only on the codebase.

The most important thing is to not be judgmental of the choices others made and understand that, at that time, it was the best decision. For example, instead of making statements like “You could have used A instead of B,” or “You could have done it like this,” or even the harsher “That is wrong!” or “That is a huge mistake!”, try to be curious and ask open-ended questions like “What made you use A?” or similar.

I’d also recommend that the team taking ownership takes the time to go through each of the topics and gain understanding and knowledge — not only from an engineering perspective, but from the perspective of the product. One might think they can copy the guideline and fill in the sections, and that writing everything they know of and handing it over to the new team will complete the transfer. I would argue that this isn’t the intention, and I don’t believe it will have the same positive impact that can be seen when doing this collaboratively and dedicating time to exploration.

Additionally, the guideline is meant to be a live document, updated as teams are learning through the process.

Side Effects / Other Impacts

This guideline should inspire teams to have useful and up-to-date documentation; a README on how to contribute, test, and run locally; and high-level architecture diagrams. This helps not only when changing ownership, but also when onboarding new team members.

Furthermore, it’s important to embrace the use of architectural decision records and help to reason about them in the future.

Summary

This guideline exists to help engineering managers, product managers, and teams acknowledge that system ownership change is a process that should be well planned and done at a time that works best for everyone involved. It’s a process that requires effort and has its cost. However, it can inspire the organization to nurture a healthy engineering culture with a high bus factor and systems that are easy to maintain, evolve, and reason about.

SoundCloud’s New API Track Object

Tue, 16 Mar 2021 00:00:00 GMT

As a part of our efforts to improve API use, we’re introducing a new Track object that’s more up to date with our current data model. The following change is also intended to standardize and simplify the response data. A new object will be used for all endpoints returning track-related information, for example:

/playlist/:playlist_id/tracks
/tracks
/tracks/:track_id

New Track Object

The following shows a schema of our new Track model:

{
  "artwork_url": Option[String],
  "available_country_codes": Option[HashSet[String]],
  "bpm": Option[Int],
  "comment_count": Option[Int],
  "commentable": Boolean,
  "created_at": String,
  "description": Option[String],
  "download_url": Option[String],
  "download_count": Option[Int],
  "downloadable": Boolean,
  "duration": Int,
  "favoritings_count": Option[Int],
  "genre": Option[String],
  "id": Long,
  "isrc": Option[String],
  "is_explicit": Option[Boolean],
  "key_signature": Option[String],
  "kind": String,
  "label_name": Option[String],
  "license": String,
  "permalink_url": String,
  "playback_count": Option[Int],
  "purchase_title": Option[String],
  "purchase_url": Option[String],
  "release": Option[String],
  "release_day": Option[Int],
  "release_month": Option[Int],
  "release_year": option[Int],
  "reposts_count": Option[Int],
  "secret_uri": Option[String],
  "sharing": String,
  "stream_url": Option[String],
  "streamable": Boolean,
  "tag_list": String,
  "title": String,
  "uri": Option[String],
  "user": User,
  "user_favorite": Option[Boolean],
  "user_playback_count": Option[Int],
  "waveform_url": String
}

Deprecated Fields

In the next couple weeks, we’re planning on removing the following deprecated fields from the response. These fields are either superfluous, outdated, or not in use by developers:

downloads_remaining
domain_lockings
embeddable_by
label
label_id
last_modified
permalink
originalcontentsize
original_format
secret_token
state
track_type
user_id
user_uri
video_url

If your application is using the Track object and/or deprecated fields, please update your API in a timely manner. If you deem any of the above fields necessary for your use case and want them to be preserved, please let us know by filing a feature request on our issue tracker by 31 March 2021.

For future updates, please follow us on Twitter and our blog.

Building a Healthy On-Call Culture

Thu, 11 Mar 2021 00:00:00 GMT

Paging Doctor Software

In the past, on-call duty was often associated with doctors, but in recent years, it’s become common for software engineers to be asked to be available for support work on short notice. As software has grown to power so much of the world, the need for high availability and rapid incident response has likewise grown.

Another driver of the popularity of on-call work is the ubiquity of smartphones. The days of bulky, unreliable pagers are gone. Now when automated software monitoring detects a system anomaly, an on-call engineer gets an alert in the form of a phone call, a text, or a loud noise from a mobile app.

On-Call Work at SoundCloud

First and foremost, on-call duty for SoundCloud engineers is optional. We believe this is important to our engineering culture for reasons I’ll discuss below. Secondly, on-call duty outside of normal office hours is compensated at an hourly rate, with additional hourly payments when responding to pages.

On-call engineers are organized into rotations. Each rotation consists of a group of engineers representing a team or, more typically, several teams. At any time, there’s always one engineer on call for the rotation. That engineer is expected to provide first-level support for the systems belonging to all the teams in the rotation.

Additionally, every engineer in the rotation is always on call to provide second-level support for the systems belonging to that engineer’s team. Second-level support is on a best-efforts basis, meaning that while engineers can receive a second-level page at any time, they’re not required to answer if they’re unavailable to help for any reason.

Why On-Call Work Is Good for Engineers

Having a wide range of engineers, and not just DevOps and Site Reliability Engineers, on call has a number of benefits both for the company and for the engineers themselves.

Perhaps most obviously, it lightens the burden on the operational engineers, who often have substantial out-of-hours support commitments as part of their core job descriptions.

It also equips and motivates engineers to build reliable, well-documented systems. Seeing firsthand how things go wrong in production powers insights into how systems can be improved and made more robust.

And finally, supporting both their own and others’ systems is a great learning opportunity for engineers. It provides valuable hands-on experience with infrastructure such as databases, as well as experience diagnosing faults and making operational decisions.

Procedural Best Practices

Every engineering organization is different, but through trial and error we’ve found the following practices work well for SoundCloud.

Different rotations have different shift cadences, but most shifts last only one or two days.

The optimal frequency for being on call is about three days a month. More than that and people risk burning out over time. Less than that and people get rusty and aren’t as effective at dealing with incidents. This means the optimal size for a rotation is between eight and twelve engineers, with ten being just about perfect. In fact, I was once part of a rotation that had a waitlist to join because we collectively agreed to not grow bigger than twelve people.

Most rotations have a formal or informal rotation administrator drawn from the engineers in the rotation. The administrator maintains the shift schedule, deals with personnel changes, and performs other ad hoc tasks that assure the rotation’s health. For example, in many rotations, the administrator organizes a meeting to set the shift schedule for the holiday period. Deciding together how to cover times of low engineer availability has proved the fairest and least stressful way of handling these situations.

A Word about Rotations and Teams

Most SoundCloud on-call rotations started as groups of engineers from related teams in the same area of the engineering organization. But, as is the case with most engineering organizations, SoundCloud’s has evolved over time. Teams have merged or split, new teams have been created, teams have been moved into different divisions, and so on. However, the on-call rotations have generally not evolved at the same pace as the engineering organization. This means the rotation structure has come to bear less and less resemblance to the wider organization structure, to the point where many rotations now represent what seem like random groups of unrelated teams.

On the whole, this has not proved to be a problem. Furthermore, attempts to reorganize the rotations to match the current company structure have usually stalled due to objections from the on-call engineers. These engineers have, in some cases, supported the systems in their rotations for years and acquired deep knowledge of those systems. Change for the sake of matching the current organizational chart represents a significant upheaval for a comparatively modest benefit. (And when the organizational chart changes again, the upheaval repeats itself.)

Cultural Best Practices

We try hard to foster the following norms and attitudes for the benefit of the on-call engineers, and by extension, the company as a whole.

The people who are on call want to be on call. Engineers who’ve freely taken on the obligation (and are being compensated for it) are more motivated when responding to incidents. Having experienced workplaces where on-call work was mandatory or close to mandatory, I can personally attest to the positive atmosphere engendered by a voluntary on-call policy.

Practical matters like shift cadences are decided collectively by the engineers in the rotation. The rotation administrator leads the decision-making process but does not act unilaterally. This is why not all of SoundCloud’s rotations have the same shift patterns, shift handover times, procedures for trading or giving away shifts, and so on. Each rotation is free to do what works best for its members.

On-call engineers often spend some of their normal working day looking into minor operational issues to prevent these issues worsening and having to page someone out of hours.

When responding to incidents, engineers can always ask for help by paging other engineers. This is something we stress multiple times during on-call onboarding. It’s a cornerstone of a healthy, high-trust on-call culture. Nobody enjoys getting a second-level support page in the middle of the night, but responding (if possible) is an important act of empathy for the engineer who needs help. It’s also an investment in training them to handle the situation autonomously in the future.

In addition to feeling free to ask for help, engineers also feel free to hand work over to others after a reasonable length of time. For major or long-running incidents, we encourage engineers to hand over after four hours — or sooner if they become too tired to work effectively.

The most important cultural practice — which influences all the other cultural practices I’ve discussed — is fostering a learning culture rather than a blame culture. This truly can’t be emphasized enough. Mistakes are an inevitable part of incident response. Learning from mistakes builds a stronger, more technically proficient engineering organization. Punishing people for mistakes makes engineers afraid to act in new situations, afraid to ask for help when they need it, and afraid to be transparent. Ultimately, engineers will choose to leave the on-call rotations or even the company if there’s a culture of blame.

When a Major Incident Happens

Responding to a total site outage or other severe incident is stressful for all involved. It’s also a stress test for a company’s on-call culture. It’s more important than ever for engineers to work well together and trust each other. They’ll resolve the incident faster if they feel comfortable admitting what they don’t know, asking for help from others, being honest if they make mistakes, and speaking up if they’re too tired to carry on.

The time to foster these behaviors is before a major incident occurs. Engineers learn them from experience, by responding to smaller incidents and interacting with their colleagues. In this sense, smaller incidents are essentially practice for larger incidents.

Conclusions

Underpinning all the practices and behaviors I’ve described in this article is one thing: respect. Companies that respect their engineers institute policies that enable those engineers to do their best work. Engineers who respect each other help each other succeed, and in turn, help the company succeed. This applies to on-call rotations as well as the wider engineering culture. When mutual respect exists, everyone wins.

Dependency Inversion as a Driver to Scale Mobile Development

Wed, 24 Feb 2021 00:00:00 GMT

SoundCloud’s iOS codebase faced a radical change a couple of years ago. The team had decided to modularize the codebase into frameworks, mainly to speed up development and give it a shape where it scales as new contributors join. The reasoning and roadmap of this transformation are explained in our Leveraging Frameworks to Speed Up Our Development on iOS — Part 1 blog post.

In this move toward modularized architecture, the team has benefited directly from faster development every day. However, when making architectural decisions on such a large scale, tradeoffs are almost always inevitable. Ours was dependency management. Without supervision of dependencies, dependency graphs end up being convoluted. The famous Spaghetti Code analogy stands out as one of the outcomes of these kinds of graphs.

Restricting dependencies is an excellent start toward loosely coupled frameworks. Modules — frameworks in our case — need other modules for interoperability. This relationship can be cleaned up by providing an API of a module as needed instead of exposing the entire module. This technique is called dependency inversion.

The dependency inversion principle (DIP) is the D in the widely regarded SOLID principles. We’ll begin with a quick definition of the concept. A Swift example will follow to show how the concept works in real-life examples. At the end, we’ll explain how we’re using DIP among frameworks in our modularized architecture.

The Dependency Inversion Principle

The dependency inversion principle suggests that a module shouldn’t directly depend on another module. Direct dependency leads to a tight coupling between high-level (consumer) and low-level (consumed) modules. With tightly coupled modules, chances are that a high-level module needs to reflect every alteration in the low-level module. Autonomous modules are better for maintainability, as we already know from microservice architectures. High- to low-level dependency looks like this:

class ModuleA { //High Level
	let foo: ModuleB //Low Level
}

Loosening the connection is key, but modules are meant to work together, so we still need a connection. The principle solves the problem with a supervisor contract that has the rules of the connection. It says modules should depend on the contract instead of each other. This contract determines what goes out of the higher module and what goes into the lower module, and it guarantees both modules that the communication will be done according to its terms. So, the initial dependency is broken into two and reverted to the contract. This is what we call dependency inversion.

A Swift Example of DIP

The best way to understand is by applying the principle to a real-life example. Imagine we have a module called ProfilePresenter. This module is responsible for logical decisions about what to show on our screen, and it’s the standard architectural component in most modern mobile development.

ProfilePresenter is our high-level module, or the consumer module. It needs a view or views to present what it needs to present. ProfileViewController (the low-level module) is our view layer for visualizing profiles under the service of the ProfilePresenter. Let’s see how this looks without DIP:

class ProfileViewController: UIViewController {
    func showFullName(_ fullName: String) { }
}

class ProfilePresenter {
    private weak var profileView: ProfileViewController?

    init(profileView: ProfileViewController) {
        self.profileView = profileView
    }

    func presentProfile(_ profile: Profile) {
        let fullName = profile.name + " " + profile.surname
        profileView?.showFullName(fullName)
    }
}

struct Profile {
    let name: String
    let surname: String
    let artworkUrl: String
}

The ProfilePresenter receives the full profile entity, extracts the full name, and commands the view to show it. It’s the responsibility of the ProfileViewController to show the full name. In other words, a unidirectional flow.

What’s the problem here? Using concrete types seems straightforward. The ProfilePresenter knows which component it’s talking to, so there are no surprises. But this is what we call an overly attached relationship or a tightly coupled dependency: We can’t easily separate the low-level module from the high-level module right now, in spite of this example being basic.

If we need an entirely different profile layout for our premium users (which we’d accomplish by injecting different views for different subscription models), and if we need to create a new view for it, we cannot easily replace the current view with the new premium user view.

Currently, our dependency relationship looks like this:

So the dependency is from ProfilePresenter to ProfileViewController. DIP says that this dependency needs to be broken and inverted — not all the way back to ProfilePresenter, but as a new contract between these two, i.e. an abstraction. So we break the current dependency into two weaker dependencies and point them to the new abstraction:

ProfileViewing is the contract that enforces the relationship rules of both parties. What ProfileViewing does not guarantee is the concrete types of both modules. This is intended by the principle, in order to have flexibility:

protocol ProfileViewing: class {
    func showFullName(_ fullName: String)
}

class ProfileViewController: UIViewController, ProfileViewing {
    func showFullName(_ fullName: String) { }
}

class ProfilePresenter {
    private weak var profileView: ProfileViewing?

    init(profileView: ProfileViewing) {
        self.profileView = profileView
    }

    func presentProfile(_ profile: Profile) {
        let fullName = profile.name + " " + profile.surname
        profileView?.showFullName(fullName)
    }
}

Not much has changed, actually. There’s now a contract visible to both modules to break the unhealthy relationship. If we have a new profile view — something like PremiumProfileViewController — we need to conform to ProfileViewing. We can then hook it up to ProfilePresenter with no additional work, and it would work just like the original profile view.

The above example is a standard use of DIP for Swift. The initial behavior of creating relationships between modules should follow this approach. There may be exceptions, but it’s good to start with the DIP approach and tighten the relationship if needed. Direct dependency is tempting because it’s straightforward, whereas we create a lot of boilerplate code to have DIP. And it might seem like it requires a lot of work to take the DIP route, but we’re actually saving some refactoring time in the long term.

DIP for Framework Dependencies

Now that we understand the mechanics of DIP, let’s see how we can use it on the level of frameworks.

If you’ve been modularizing your project into frameworks, you probably had issues with the dependencies among them. When the number of frameworks in a project increases, their dependency on each other becomes complicated, and it introduces problems ranging from broken syntax highlighting to longer compile times. By inverting some dependencies, we can save ourselves from this hectic dependency graph.

The problem is similar to the previous example. Framework-A needs a piece of Framework-B, but Framework-A doesn’t want to depend on the entirety of Framework-B. Who can blame Framework-A, right? Instead, Framework-A can be given the exact piece it needs from Framework-B: an injection per se.

Again, the two modules don’t like their direct dependency, so they need a lightweight, more manageable contract. Since we’re working on the framework level, the contract should be placed in its own framework. This new framework holds the contract and a minimum number of pieces that the contract needs. It’s important to keep this framework thin because other frameworks will depend on it. Since certain pieces need to be injected into the framework, and since the main app target is the one injecting, the main app also needs to depend on the new contract framework:

To visualize this, let’s say we have two frameworks named Authentication and Payment, each of which contains code associated with its name. Payment needs an authenticator when the user wants to make the payment. We have a class named Authenticator, located in the Authentication framework, exactly for this purpose:

public class Authenticator {
    public init() { }
    public func authenticate() { }
}

The Payment framework needs this piece in its PaymentManager class:

public class PaymentManager {
    private let authenticator: Authenticator

    public init(authenticator: Authenticator) {
        self.authenticator = authenticator
    }

    func doFoo() {
        authenticator.authenticate()
    }
}

So far, it looks almost exactly like the previous example. We learned from the previous example that we shouldn’t use the concrete Authenticator type; we need to invert this dependency to a contract. So we need a new framework called AuthenticationContract to put the protocol in. The AuthenticationContract framework needs to be imported to Payments, Authentication, and the Main App:

public protocol Authenticating {
    func authenticate()
}

import AuthenticationContract

public class PaymentManager {
    private let authenticator: Authenticating

    public init(authenticator: Authenticating) {
        self.authenticator = authenticator
    }

    func doFoo() {
        authenticator.authenticate()
    }
}

import AuthenticationContract

public class Authenticator: Authenticating {
    public init() { }
    public func authenticate() { }
}

So dependencies are inverted to a new protocol called Authenticating, and it’s safe to import AuthenticationContract to both frameworks. One last thing to do here is to give Authenticator to PaymentManager, which will be done in the Main App:

import AuthenticationContract
import Payment
import Authentication

let authenticator = Authenticator()
let paymentManager = PaymentManager(authenticator: authenticator)

Briefly, PaymentManager needed a piece of the Authentication framework, the Authenticator. Instead of creating a dependency from the Payment framework to the Authentication framework, we inverted this dependency to the new AuthenticationContract framework, which is way more lightweight than Authentication. We saved ourselves from the tightly coupled relationship between two rather heavy frameworks.

This process can be replicated for every framework that needs to serve others. For example, a framework named Subscriptions needs some parts from Payment, and then a contract framework named PaymentContract can be created for Payment. Ultimately, frameworks come with their own companion contract frameworks.

So What?

When building software on any scale, individual modules need to work in harmony; hence we need dependencies. The dependency inversion principle doesn’t remove the dependency need for modules; they still need to work together. Rather, it sets up contracts that regulate the traffic between modules while assuring them that their internal changes won’t affect the other modules.

Inverted dependencies increase the chance that changes won’t be propagated into our software. Even if we want to replace a module whose dependency is inverted entirely, a new module assembly and integration will be way easier than a tightly coupled version. This is a good practice to lighten the burden of dependencies, but of course it’s not the only one!

Tests Under the Magnifying Lens

Tue, 02 Feb 2021 00:00:00 GMT

Testing is at the heart of engineering practices at SoundCloud. We strive to build well-balanced test pyramids within our code repositories and have as much test coverage as possible for different service use cases.

Here’s an x-ray view of the test pyramid in one of our largest codebases, the Android application:

As test suites grow and, more interestingly, cover more complicated integration cases, engineers have had difficulties when answering questions about the value, validity, and correctness of their tests. Some of the main challenges in building and testing at SoundCloud are pipeline-waiting times and test flakiness, respectively.

Data First

Test flakiness impacts teams in different technology stacks, both frontend and backend. In parallel, a challenge we commonly face is weighing tech health work against feature work to help product and engineering teams prioritize their work iterations.

As one of the efforts in leveling up our testing infrastructure — inspired by our peers in the industry (e.g. Spotify, GitHub, Google, Fowler, and Cypress) — we identified a potential improvement to our tooling, and we came up with a path to enabling cross-repository, cross-team, and cross-feature data aggregation and analysis that could potentially help us solve the aforementioned challenges by informing engineers and other non-technical stakeholders.

To do that, we must flip a common engineering view that test reports are outputs of the build process, and instead look at build artifacts as inputs to a next processing step in the engineering workflow.

In other words, every test report becomes a data point to help draw a curve that will support driving insights from the build pipelines themselves; track the test suites’ health, retries, and execution times; and more interestingly, help predict or bias an expected movement.

Finally, we aim to help prioritize refactoring or fixing tests so that they speed up test pipelines to reduce running cost and developer waiting times.

Backstage

With these targets in mind, we’ve come up with a service architecture backed by a data storage. For our requirements, all we need is a CRUD for data ingestion and manipulation, with some extended capabilities for rendering.

Ingestion

Given the service is aimed at integration with pretty much any stack, framework, or programming language — so as to attain wide internal adoption — we picked an agnostic scheme and designed an interface with as few assumptions about the data format as possible.

The first implementation for the tool is a thin Scala application with a Finagle+MySQL setup that ingests data via a REST API. It’s similar in skeleton to our production microservices, which means we leverage a lot of our existing toolkit for building and deploying the service, while also empowering engineers in the organization to contribute to improving the system by collaborating on a stack they already have experience with.

To minimize the effort of integration even further, we also provided a command-line tool, distributed through our private Docker registry, that knows how to translate some common test report formats (such as JUnit) into the expected schema for the service’s API.

That covers most of our use cases, since many of the testing libraries we rely on — such as the ones we run for our Android and pure Java applications — are capable of generating these XML reports. For other codebases, with ScalaTest, RSpec, and even Xcode, we rely on third-party and open-source integrations.

In practical terms, most teams simply add a step to their build pipelines, specifying a few parameters for the command-line uploader. Here’s an example from one of our GoCD pipelines:

stages:
...
- acceptance-test:
 jobs:
  acceptance-test:
   tasks:
   - exec:
    command: make acceptance-test
   - exec:
    run_if: any
    command: <docker run> -- automated-test-monitor-uploader \
     --system=api-mobile \
     --reports_dir=target/test-reports

At the end of the day, each of the teams can opt for a “vanilla” integration of their system by using the Docker image, but they can also choose to write their own report-uploading script to maximize the information they store in the database for querying later. This has even greater potential to allow teams to group and slice data through test cases’ metadata — for example, our Web Collective might tag their tests according to the browser they were run on, while our iOS team can use the device/OS version metadata for their use case.

Rendering

Once the data is stored in a structured way, the next step is to provide rendering capabilities so that teams can follow trends and the evolution of their datasets.

Here, we rely on first-party and third-party Grafana data source plugins, with some generic prebaked metrics we believe are useful for most cases. The plugins talk back to the ingestion service, executing queries to manipulate the data. This includes the number of builds and a drill down of test case outcomes, as well as rankings for the flakiest tests or the ones that might represent a bottleneck for the entire build pipeline.

Tests that take too long to run, on average, rapidly add up to decrease the overall engineer productivity by increasing waiting times in build queues and when running the suite locally. Not only that, but this slowdown potentially means higher costs for SoundCloud due to virtual/physical device time allocation in third-party device farms.

We found that Spotify’s Odeneye was a great way to identify both test flakiness and infrastructure problems, as the company itself describes:

If you see a scattering of orange dots this usually means test flakiness. If you see a solid column of failures this usually represents infrastructure problems such as network failures.

Leveraging the thin Scala application layer in front of the Grafana data source gives us extra power, allowing for further filtering and grouping to help understand outage scenarios before they hit developer productivity. On top of that, we have a better overview of the assumptions our code is making and where coverage might be lacking.

Alternatively, we also provide engineers, data scientists, managers, and other SoundClouders with direct access to our SQL entry point into the schema for a more “heads-down” approach, in which more flexibility allows for discoveries of team-specific metrics or even extraction of the whole dataset via dumping.

Action

Having data is the first step to be able to act on it, and each team has different practices for how they deal with test coverage, flakiness, and failures.

Some may visit the graph weekly to check on the trends, while others may only check when they experience build failures; some could create tasks in their ticketing system to regularly increase the codebase’s overall quality, while others might decide to set up alerts when specific metrics go above or below a certain threshold. The tool doesn’t enforce a given workflow, but it’s ready as support when teams decide to invest in technical health.

Eventually, we want to make the tracker for test execution available in a holistic way, so that we can answer questions like “How many failures did we have across all of SoundCloud over the last two weeks?” or “How has the rate of flakiness moved across all of SoundCloud in the last year?”

Leveraging a Manager Weekly Newsletter for Team Communication

Wed, 25 Nov 2020 00:00:00 GMT

I started my journey as an Engineering Manager at SoundCloud close to a year ago. This came after working as a Software Engineer for more than a decade, most of which I spent as an iOS Engineer. Since then, I’ve been trying to figure out my role and expand my knowledge on the subject. This entails constantly meeting with other peers, along with reading books I’ve found or that have been recommended to me about how to be a better people manager.

One of these books is “The Making of a Manager: What to Do When Everyone Looks to You” by Julie Zhuo. In this book, the author mentions that during the week, she jots down her thoughts about her team, what she hears others say about it, how the projects are going, etc; basically anything that’s happening during the week that is directly or indirectly related to her team, she writes down.¹

Zhuo said she does this to properly digest the daily events — if possible, she writes them as they happen, if not within hours of them happening. Then, at the end of the week, she reviews them, which allows her to digest the different situations and process stuff she might have forgotten or that she remembers but which are a bit fuzzy. For her to cement this new habit, she started sending the resulting compilation to her team.

When I first read this, it struck a chord in me, as a few weeks prior to reading this book, I had started working on my own journaling habit. I did this by leveraging it with Obsidian for my notetaking and daily jots in a way that allowed me to connect topics organically and learn from the patterns that emerged.

Next Steps

Thanks to my nature that pushes me to try new things and introduce changes and tools to my system to see if they can bring me any value, it didn’t take much for me to get in the habit of writing my daily notes in a way similar to Zhuo. As mentioned above, I was already taking daily notes as a reference for my weekly reviews, as well as to not forget what was important during meetings. However, I wasn’t sharing them with the team. After reading Zhuo’s book, the idea of sharing them with the team made a lot of sense and required little adapting on my part, as I was already compiling a daily journal.

Obsidian has a plugin for daily note taking; you can provide a template in Markdown and it’ll be used any time you create a new daily note. So with this in place, I can quickly, and with just one click, be on my daily note.

The image above shows a sample of my notes and the format I’ve chosen for naming them (this is also automatically filled by Obsidian). I just set up the date, and then at the end, I include the name of the day (this is useful and important for my newsletter, since I only need to take a look at the notes from Monday to Friday for it).

What Does a Daily Note Look Like?

Honestly, there’s not much magic behind the daily notes. I literally write down anything and everything that happens; later on, when I’m revisiting my notes, I can ignore what’s not relevant or even delete something if I feel like it’s not adding value or it isn’t clear enough.

My template is divided into two sections: Happenings and Grateful For. Originally, it also had action items at the bottom, but since I follow the Getting Things Done principle, I write every action item that comes my way in my tracking system (at the moment, it’s a combination of Todoist and Notion), so it doesn’t make much sense to duplicate efforts.

Happenings

This is where everything goes. It’s usually a list of one or two liners describing things that happened in a meeting, a conversation I noticed on Slack, an email that caught my attention, etc. I try to not write a lot per topic, but I do tend to make a lot of entries (usually ending up with 20 or more per day). The idea is just a single line that will jog my memory and put me in the same state of mind as when the thing it’s describing happened.

Grateful For

This is part of the journaling exercise that has come in handy for the newsletter. Here I try to write two to three things that I’m grateful for each day. The idea is to not have to think too hard, which gives me the freedom to write small things like “Having flexibility today to move some meetings and get more work done.” Wherever it is that I’m grateful for, no matter how minor, I write it down.

Wiring Everything Together for Team Communication

At the end of the week, I go through all my notes related to work and to the team and compile them into a list. Here I try to be succinct and just provide a small update for the team.

This way, I can highlight the strengths and remarkable qualities the team showed during the week, as well as point to some areas where we can do better.

I also copy my manager in the email to take the opportunity to show them all the things the team is doing that might not be as visible (Jira tasks and meetings can be visible for externals, but the other most important topics could go unnoticed) and make my manager aware of all the great things the individuals on the team are doing. The purpose here is to bring to light what the team is doing in a more pointed way by highlighting specific things they accomplish during the week.

I find this important because it reminds me of things that might not be too “flashy” but that were super important. It has also given me a tool to recognize weekly work that the team executed.

One key aspect of this for me is to keep it casual, since the objective is not a report but rather a sharing of one’s thoughts and ideas during the week, and also to let the team know how their work and the team are being perceived by others outside of the group of people that they directly work with on a daily basis.

On that note, it’s important that the email is short and concise; I aim to exclude details that are either too small to bear any relevance or that are just part of the operational overhead of running the team and adjacent projects. I’m trying to keep everyone up to date on the latest happenings of the team but not achieve such a granular level of details that the reader might get lost without active technical knowledge of each and every project.

We’re starting to adopt this technique at SoundCloud outside of my team (other teams, project-specific updates, etc.) and so far, it has proven quite successful. Often, team members feel happy when something they did gets mentioned in the email; they might have not thought it was that remarkable or they might think it went unnoticed, and then reading it in the weekly newsletter definitely brings some unexpected joy. This was expressed to me by a couple of team members during our 1:1s, confirming the hypothesis that highlighting these achievements helps build the team’s morale.

On a parting note, one last detail I derived from this exercise is that while I’m compiling the list of achievements, improvements, and random thoughts, I get to do a “retrospective” of the week. This helps me understand how things are going; define a better trajectory for the team and the projects; and adjust direction more quickly, because I take the time to process things regularly and within a “short” period of time so as to avoid having to go through lots of different events. This obviously has the potential to keep the team on track and informed constantly and regularly. Only time will tell how beneficial this practice might be, but at least for the time being, it’s proving to be quite useful.

Note: It seems a similar concept is also mentioned in the book “Soft Skills: The Software Developer’s Life Manual” by John Z. Sonmez.
↩

Testing SQL for BigQuery

Fri, 16 Oct 2020 00:00:00 GMT

“To me, legacy code is simply code without tests.” — Michael Feathers

If untested code is legacy code, why aren’t we testing data pipelines or ETLs (extract, transform, load)? In particular, data pipelines built in SQL are rarely tested. However, as software engineers, we know all our code should be tested. So in this post, I’ll describe how we started testing SQL data pipelines at SoundCloud.

Now when I talked to our data scientists or data engineers, I heard some of them say “Oh, we do have tests! After creating a dataset and ideally before using the data, we run anomaly detection on it/check that the dataset size has not changed by more than 10 percent compared to yesterday etc.” That’s not what I would call a test, though; I would call that a validation.

Validations are important and useful, but they’re not what I want to talk about here. Validations are what increase confidence in data, and tests are what increase confidence in code used to produce the data. And SQL is code. Ideally, validations are run regularly at the end of an ETL to produce the data, while tests are run as part of a continuous integration pipeline to publish the code that will be used to run the ETL.

Some of the advantages of having tests and not only validations are:

Some bugs can’t be detected using validations alone. For example, if your query transforms some input data and then aggregates it, you may not be able to detect bugs in the transformation purely by looking at the aggregated query result. Hence you need to test the transformation code directly.
Depending on how long processing all the data takes, tests provide a quicker feedback loop in development than validations do.
Validations are code too, which means they also need tests.
Improved development experience through quick test-driven development (TDD) feedback loops.

The Backstory

My team, the Content Rights Team, used to be an almost pure backend team. We handle translating the music industry’s concepts into authorization logic for tracks on our apps, which can be complicated enough. However, since the shift toward data-producing teams owning datasets — which took place about three years ago — we’ve been responsible for providing published datasets with a clearly defined interface to consuming teams like the Insights and Reporting Team, content operations teams, and data scientists.

We’ve been using technology and best practices close to what we’re used to for live backend services in our dataset, including:

Spark to conveniently code in the same language as our backend service, Scala
Prometheus to monitor, using custom metrics pushed through the Pushgateway (these metrics give us insight through alerting and visualization)
Tests built like the tests for our live backend services

However, Spark has its drawbacks. Even though the framework advertises its speed as “lightning-fast,” it’s still slow for the size of some of our datasets. This affects not only performance in production — which we could often but not always live with — but also the feedback cycle in development and the speed of backfills if business logic has to be changed retrospectively for months or even years of data.

Of course, we educated ourselves, optimized our code and configuration, and threw resources at the problem, but this cost time and money. For some of the datasets, we instead filter and only process the data most critical to the business (e.g. only export data for selected territories), or we use more complicated logic so that we need to process less data (e.g. rolling up incrementally or not writing the rows with the most frequent value).

Other teams were fighting the same problems, too, and the Insights and Reporting Team tried moving to Google BigQuery first. Inspired by their initial successes, they gradually left Spark behind and moved all of their batch jobs to SQL queries in BigQuery. But with Spark, they also left tests and monitoring behind. In their case, they had good automated validations, business people verifying their results, and an advanced development environment to increase the confidence in their datasets. Through BigQuery, they also had the possibility to backfill much more quickly when there was a bug.

In the meantime, the Data Platform Team had also introduced some monitoring for the timeliness and size of datasets. But still, SoundCloud didn’t have a single (fully) tested batch job written in SQL against BigQuery, and it also lacked best practices on how to test SQL queries. I searched some corners of the internet I knew of for examples of what other people and companies were doing, but I didn’t find a lot (I am sure there must be some out there; if you’ve encountered or written good examples, I’m interested in learning about them).

It struck me as a cultural problem: Testing didn’t seem to be a standard for production-ready data pipelines, and SQL didn’t seem to be considered code.

The Project

After I demoed our latest dataset we had built in Spark and mentioned my frustration about both Spark and the lack of SQL testing (best) practices in passing, Björn Pollex from Insights and Reporting — the team that was already using BigQuery for its datasets — approached me, and we started a collaboration to spike a fully tested dataset. His motivation was to add tests to his team’s untested ETLs, while mine was to possibly move our datasets without losing the tests.

We used our “self-allocated time” (SAT, 20 percent of engineers’ work time, usually Fridays), which is one of my favorite perks of working at SoundCloud, to collaborate on this project. As the dataset, we chose one: the last transformation job of our track authorization dataset (called the “projector”), and its validation step, which was also written in Spark. We already had test cases for example-based testing for this job in Spark; its location of consumption was BigQuery anyway; the track authorization dataset is one of the datasets for which we don’t expose all data for performance reasons, so we have a reason to move it; and by migrating an existing dataset, we made sure we’d be able to compare the results. Also, it was small enough to tackle in our SAT, but complex enough to need tests.

Challenges and Solutions

Not all of the challenges were technical.

Organizationally, we had to add our tests to a continuous integration pipeline owned by another team and used throughout the company. Fortunately, the owners appreciated the initiative and helped us.

I’ve already touched on the cultural point that testing SQL is not common and not many examples exist. We shared our proof of concept project at an internal Tech Open House and hope to contribute a tiny bit to a cultural shift through this blog post.

The technical challenges weren’t necessarily hard; there were just several, and we had to do something about them. I don’t claim whatsoever that the solutions we came up with in this first iteration are perfect or even good — but they’re a starting point.

The tests had to be run in BigQuery, for which there is no containerized environment available (unlike e.g. MySQL, which can be tested against Docker images). We at least mitigated security concerns by not giving the test account access to any tables. The next point will show how we could do this.
Test execution performance had been the biggest blocker of previous attempts to add tests against BigQuery. BigQuery is performant for, well, big queries, but it has high constant time overhead, especially for loading data from and into tables. Single queries also take two to three seconds, which is still a lot, but not as much as loading a table first. Therefore, our tests don’t access any tables at all, and we inject all test data as literals in our queries. An example would be:

WITH launch_dates AS (
  SELECT territory_code, TIMESTAMP(launched) AS launched
  FROM (SELECT
          'US'               AS territory_code,
          DATE('2000-01-01') AS launched
        UNION ALL
        SELECT
          'DE'               AS territory_code,
          DATE('2050-01-01') AS launched) [...]

SQL itself does not have language support or frameworks for testing. OK, one could argue that SQL’s ability to throw exceptions is all one needs, but that’s pretty basic. So we decided to use Python and pytest. Extra benefits we get from this are that it’s the same language as our scheduler (Airflow), it has good failure reporting, and it has easy execution of e.g. a single test file and dependency injection through fixtures.
Still, SQL has no language support for queries to be parametrized in the table. This is necessary for switching between the literals as tables in tests and the actual table name in production. We use the templating engine Jinja, which is conveniently also used in Airflow.
Already for Spark, it’s a challenge to express test data and assertions in a _simple-to-understand way_ — tests are for reading. We tried our best, using Python for abstraction, speaking names for the tests, and extracting common concerns (e.g. connecting to BigQuery and rendering templates) into pytest fixtures. We created dataclasses for the data schema with to_sql and from_row methods for better readability, e.g.:

@dataclass(eq=True, frozen=True)
class TrackPolicyInfo:
    track_urn: str
    content_policy: str

    @classmethod
    def from_row(cls, row):
        return cls(track_urn=row.track_urn, content_policy=row.content_policy)

    @property
    def as_sql(self):
        return f"""
          (
            SELECT
            '{self.track_urn}'      AS track_urn,
            '{self.content_policy}' AS content_policy
          )
            """

The production queries use Airflow templating (e.g. {{ execution_date }} or {{ ds_nodash }} ) to be runnable in Airflow. As we run the tests independently from Airflow, this part of the queries is unfortunately not yet tested. As a substitute, the team that maintains our Airflow now supports running Airflow locally, so we can at least manually check the queries after rendering (i.e. expansion of the template).
SQL is not as decomposable as we’re used to for other high-level programming languages. Thus writing unit tests isn’t trivial. We’ve approached this by using common table expressions (CTEs aka “WITH clauses”) extensively and adding templating with defaults to be able to only test single CTEs. Example:

WITH raw_rollup AS (SELECT * FROM {{ params.rollup_data }}{{ ds_nodash }}),

  -- replace monetizing policies in non-monetizing territories and split intervals
  [...]
  policies AS (...),

  -- now deduplicate / merge consecutive intervals with same values
  windowed_policies AS(
    SELECT [...] FROM {{ params.policies_to_deduplicate|default('policies') }}),
  [...]
  deduplicated_policies AS (...),
  SELECT * FROM {{ params.selected_cte|default('deduplicated_policies') }}

Now we can test the CTE

policies by setting params={'rollup_data': example.input_query, selected_cte': 'policies'} and
windowed_policies by setting params={'rollup_data': empty_rollup_data, 'policies_to_deduplicate': example.input_query} with the right selection of example and empty_rollup_data.

Conclusion

Although this approach requires some fiddling — e.g. for testing single CTEs while mocking the input for a single CTE — and can certainly be improved upon, it was great to develop an SQL query using TDD, to have regression tests, and to gain confidence through evidence. When I finally deleted the old Spark code, it was a net delete of almost 1,700 lines of code; the resulting two SQL queries have, respectively, 155 and 81 lines of SQL code; and the new tests have about 1,231 lines of Python code. A substantial part of this is boilerplate that could be extracted to a library.

I’m looking forward to getting rid of the limitations in size and development speed that Spark imposed on us, and I’m excited to see how people inside and outside of our company are going to evolve testing of SQL, especially in BigQuery.

Pagination Updates on Our API

Mon, 21 Sep 2020 00:00:00 GMT

As part of our efforts to improve our APIs, we’re introducing updates on how we paginate over tracks. This only affects developers and apps that aren’t strictly relying on the next_href field, but rather custom manipulating the offset to request a page.

Affected Endpoints

The following endpoints are affected:

/users/:userId/tracks
/me/tracks
/users/:userId/playlists
/me/playlists

Going forward, the paginated responses for the above endpoints will be represented via a cursor in the next_href field. The cursor field will replace the need for an offset to paginate, while also improving the performance of this endpoint. We recommend that developers strictly rely on the next_href field to request paginated results.

Current Format

Here’s the current format, which uses offset for next_href:

https://api.soundcloud.com/users/123/tracks?linked_partitioning=true&limit=1&offset=1

New Format

Here’s the new format, which uses cursor for next_href:

https://api.soundcloud.com/users/123/tracks?linked_partitioning=true&page_size=10&cursor=2018-02-27T14:37:10.000Z,tracks,00405959469

Example Response

Here’s an example response:

{
  "collection": [
    {
      //... track
    }
  ],
  "next_href": "https://api.soundcloud.com/users/123/tracks?linked_partitioning=true&page_size=10&cursor=2018-02-27T14:37:10.000Z,tracks,00405959469"
}

This change will be in effect by 28 September 2020.

A Quick Note about the linked_partitioning Field

Setting the linked_partitioning field to true is our recommended way of fetching paginating responses. The shape of the response is an object with a cursor and the collection of results, rather than an array of the collection. In the near future, all responses for a collection will be paginated by default and served as an object. We highly recommended that developers include this parameter in their requests, as it’s the only way the API can guarantee paginated responses.

To get information about future updates, please follow us on Twitter and subscribe to the Backstage Blog.

Breaking Loose from Third-Party Lock-In with Custom Refactoring Tools

Tue, 11 Aug 2020 00:00:00 GMT

Code refactoring is an essential part of the job of software developers. As time goes on, technology evolves, product requirements change, and new features are built into a codebase. These things have the potential to negatively impact both the codebase’s quality and its capacity to be extended to retain its value for as long as possible. In addition to refactoring, it’s also part of our job to timely detect when changes in our code are necessary, so as to avoid code becoming too complex or growing to unmanageable levels.

However, even though it’s essential, refactoring can at times be intimidating, and introducing improvements to a codebase without affecting its behavior can impose huge challenges on software developers.

Deprecating Specta

At SoundCloud, we create all our new features in our iOS app using Swift, but since our codebase was started several years ago, we still have a significant amount of Objective-C code to maintain (including third-party libraries). Years ago, before adopting Swift, we decided to use Specta, “A light-weight TDD / BDD framework for Objective-C,” to write our unit tests. It brought us a lot of value for a long time, but around two years ago, after experiencing issues like not being able to run a single test without the entire test suite being run, and being unhappy with Xcode’s support for Specta, we decided to deprecate it.

From the moment of deprecation onward, we decided to write all our new tests using plain XCTests, but we were still locked in to using Specta, as we had more than 900 spec files in our codebase.

The number of specs in our project was not negligible, and manually transforming those specs into XCTests was a very tedious, error-prone, and time-consuming task; even though we would have loved to transform all the specs into XCTest at once, it would have taken too much time to be viable. Given this interesting challenge, I decided to take advantage of one of the benefits of working at SoundCloud and set out to find a way to automate the transformation of these specs during my self-allocated time (SAT). Luckily, the project succeeded and Specta is no longer part of our codebase. So in this post, I’ll cover how it was done.

Specta vs. XCTest

Let’s start by comparing how tests are structured in both Specta and XCTest. Due to Specta’s domain-specific language (DSL), you can structure your tests in complex ways. This can be a blessing if you’re trying to avoid duplication and a curse if it’s taken too far, as it makes specs hard to navigate with all the nesting. For the sake of this blog post, we’ll use a rather simple spec to demonstrate the work done.

The image below shows how testing the same simple User class is done using both formats:

Test Declaration

In Specta, tests are declared with the use of the SpecBegin and SpecEnd macros, while in XCTest, tests are classes that inherit from the XCTestCase class.

Scope of Variables and Properties

In Specta, the scope of variables is determined by the enclosing scope they’re declared in. The Specta DSL works by using functions that get passed blocks, where every block defines a new context. To make variables mutable from within blocks nested at a deeper level, they need to be declared with the __block keyword. In XCTest, member properties or ivars are declared to make them accessible by methods, just like a regular Objective-C class.

Setup and Teardown

When writing tests, we need to prepare the initial state before our tests run and clean it up after they complete. In XCTest, XCTestCase provides class and instance methods that can be overridden to set and clean the state. In Specta, the before and after methods serve the same purpose.

Test Methods

In XCTest, test methods need to comply with a set of conventions in order for Xcode to find and execute them. As per Apple documentation, “a test method is an instance method on an XCTestCase subclass, with no parameters, no return value, and a name that begins with the lowercase word test.” When using Specta, we call the it function and pass the block that contains the assertions for our test (specs are processed at runtime to dynamically generate test classes that comply with these conventions, which makes it possible for Xcode to execute them).

From Specta to XCTest

Knowing how these two ways of writing tests differ, I figured that to systematically transform a Specta spec into an XCTestCase class, we needed to do the following:

Find top-level calls to the before and after methods and use them to generate setUp and tearDown methods.
Find calls to the it method and generate test methods from them.
Find calls to the before method that are within a context and generate setup methods from them that can be called from test methods to set the initial state as needed.

In order to generate intuitive names for our test methods, the context names needed to be extracted from calls to the describe and context functions and then processed to make them valid Objective-C method names. In the same manner, calls to before and after methods from within the associated contexts needed to be detected to make sure the state was being prepared accordingly before the tests were run. If this sounds confusing, the image below might give you a better idea of how elements from the spec were translated into XCTest test methods.

Code transformations can be done in multiple ways, and depending on the problem, one tool might be better than another. Some of the options I considered were:

Text replacement using regular expressions.
Refactoring features available in Xcode.
Low-level libraries that operate on the abstract syntax tree (AST).

While possible, using regular expressions to perform the type of source transformations described above would have been incredibly cumbersome and error-prone. The available Xcode features for refactoring were also not advanced enough for the task at hand, so I was left with option three. I started investigating different alternatives that allowed me to operate on the AST. Fortunately, I found LibTooling, which provides a way to conveniently analyze the abstract syntax tree using a set of high-level abstractions to match against sections of the code you’re interested in. In the following sections, I’ll expand on what LibTooling is and how I used it.

LLVM’s LibTooling

LibTooling is a library you can use to create your own standalone tools based on the Clang compiler. The LLVM compiler takes our code through three different stages to create the executable:

The frontend stage takes the code through the lexical, syntactic, and semantic analysis phases to generate the IR (intermediate representation) that is used as an input for the subsequent phases.
The middle-end stage applies target-independent optimizations — like the removal of useless or unreachable code — on the IR.
Finally, the backend stage converts the IR into machine code to generate the target program.

In order to efficiently perform a source transformation, we need to be able to operate on the AST, which means we need to somehow be able to tap into the compiler’s frontend to have it generate the AST for our source and later be able to analyze it. Fortunately, this is exactly what LibTooling’s frontend actions do!

ASTMatchFinder

ASTMatchFinder is a frontend action class that can be used to match against the AST. To better explain what is meant by “matching against the AST,” I’ll provide a concrete example.

Consider the following function:

int f(int x) {
  int result = (x / 42);
  return result;
}

Now let’s see what its AST looks like:

`-FunctionDecl 0x5aeab50 <test.cc:1:1, line:4:1> f 'int (int)'
  |-ParmVarDecl 0x5aeaa90 <line:1:7, col:11> x 'int'
  `-CompoundStmt 0x5aead88 <col:14, line:4:1>
    |-DeclStmt 0x5aead10 <line:2:3, col:24>
    | `-VarDecl 0x5aeac10 <col:3, col:23> result 'int'
    |   `-ParenExpr 0x5aeacf0 <col:16, col:23> 'int'
    |     `-BinaryOperator 0x5aeacc8 <col:17, col:21> 'int' '/'
    |       |-ImplicitCastExpr 0x5aeacb0 <col:17> 'int' <LValueToRValue>
    |       | `-DeclRefExpr 0x5aeac68 <col:17> 'int' lvalue ParmVar 0x5aeaa90 'x' 'int'
    |       `-IntegerLiteral 0x5aeac90 <col:21> 'int' 42
    `-ReturnStmt 0x5aead68 <line:3:3, col:10>
      `-ImplicitCastExpr 0x5aead50 <col:10> 'int' <LValueToRValue>
        `-DeclRefExpr 0x5aead28 <col:10> 'int' lvalue Var 0x5aeac10 'result' 'int'

The AST shows what type of nodes are generated when compiling the function above. With MatchFinder, you can attach instances of AstMatcher to selectively match nodes of this AST. Some of the matchers we can use are callExpr, declStmt, functionDecl, and CompoundStmt. For more matchers, have a look at the reference site.

Refactoring with ASTMatchers

As described in the section above, ASTMatchers help us easily identify the presence of specific nodes in our source code. Once we’ve identified the nodes, we can also find out their specific locations. This allows us to analyze our spec source code and extract the sections we’re interested in. Once extracted, we can use the output to build a new source — in this case, our XCTestCase class. Let’s look at some of the ASTMatchers I created for matching against parts of a Specta spec. The matcher used to match against the it functions looks as follows:

const auto ItExprMatcher = callExpr(callee(functionDecl(hasName("it"))),
                                    hasArgument(0, hasDescendant(stringLiteral().bind("testName"))),
                                    hasArgument(1, blockExpr().bind("block"))
                                    ).bind("itExpr");

We can see how the DSL allows us to express in a very natural way that we want to find a call expression, where the callee is a function with the name it. The function should have two arguments — the first one being of type string literal, and the second being a block expression. Also, notice the calls to the bind function; as we’ll see in a moment, calling this function allows us to refer to specific nodes via the names that are passed to it. Let’s look at a more complex example to better illustrate what ASTMatchers can do:

const auto TopLevelBeforeExpr = callExpr(
                                         callee(functionDecl(hasName("before"))),
                                         hasArgument(0, blockExpr().bind("beforeExpr")),
                                         unless(hasAncestor(callExpr(callee(functionDecl(anyOf(hasName("describe"),  hasName("context")))))))
                                         ).bind("topLevelBeforeExpr");

As mentioned before, we need to distinguish between calls to before that are within a context and those that are top-level ones. This is necessary in order to produce setup methods that will set the initial state according to our tests. To identify top-level before calls, we can add a condition to our matcher where it’ll accept calls to functions with the name “before” that take a block expression as the first argument as long as they have an ancestor within a context or describe expression.

Handling Matches

Every time we register a matcher in our MatchFinder, we also need to pass an object that implements the MatchCallback abstract class. This class declares the following methods:

  class MatchCallback {
  public:
    virtual ~MatchCallback();
    virtual void run(const MatchResult &Result) = 0;
    virtual void onStartOfTranslationUnit() {}
    virtual void onEndOfTranslationUnit() {}
  };

As their names suggest, the StartOfTranslationUnit and onEndOfTranslationUnit methods are called at the start and end of processing a translation unit (source file). More importantly, the run method is the one that will be called whenever a match is detected. In order to know what type we need to process, we can check every result for nodes we’re trying to match using the type of node and the string we passed to the bind function, like this:

void ASTHandler::run(const MatchFinder::MatchResult &Result) {
     if (Result.Nodes.getNodeAs<clang::Expr>("itExpr")) {
         ...
    }
}

Here we’re checking if the result we got contains a node of type Expr that’s bound to the itExpr identifier. Once we have identified the type of result, we can get more details from it — for instance, when we match a call to the it function, we want to be able to extract the first parameter in order to use it to generate our XCTestCase method. We can extract this string as follows:

  if (Result.Nodes.getNodeAs<clang::Expr>("itExprWrapper")) {
    const auto itText = Result.Nodes.getNodeAs<clang::StringLiteral>("testName")->getString().str();
  }

Unique Identifiers

In order to identify sibling nodes after extracting the code from the AST, I needed to mark nodes in a way that allowed me to identify the ones that shared the same parent. For this, I created a helper function that generates a unique ID based on the beginning and end location of a node:

string Utils::getIdForNode(const std::string key, const MatchFinder::MatchResult Result) {
    const clang::Stmt *node = Result.Nodes.getNodeAs<clang::Stmt>(key);
    const auto beginLoc = node->getBeginLoc();
    const auto endLoc = node->getEndLoc();
    return (std::to_string(Result.SourceManager->getFileOffset(beginLoc)) + ":" + std::to_string(Result.SourceManager->getFileOffset(endLoc)));
}

Matching Preprocessors

In addition to matching nodes in the AST, I also needed to handle preprocessors in order to find inclusion directives that needed to be copied over to the XCTest output file. LibTooling provides the PreprocessOnlyAction class, which allows us to attach preprocess actions capable of matching against preprocessors like inclusion and pragma directives, among others.

Collecting Results

To make things simpler for me given my limited C++ experience, I decided to parse the match results into JSON and do the rest of the work to generate the XCTest source using a different set of tools. The example below shows the resulting JSON for the previously presented User sample spec:

Sample json result

{
  "afters": [
    {
      "body": "\n  subject = nil;\n",
      "id": "314:341"
    }
  ],
  "befores": [
    {
      "body": "\n  subject = [[User alloc] init];\n",
      "id": "266:310"
    },
    {
      "body": "\n      subject.hasConfirmedEmail = NO;\n    ",
      "id": "424:477",
      "parentId": "389:579"
    },
    {
      "body": "\n      subject.hasConfirmedEmail = YES;\n    ",
      "id": "629:683",
      "parentId": "585:787"
    }
  ],
  "contexts": [
    {
      "id": "389:579",
      "isSharedExample": false,
      "name": "email confirmed",
      "parentId": "345:794"
    },
    {
      "id": "585:787",
      "isSharedExample": false,
      "name": "email not confirmed",
      "parentId": "345:794"
    },
    {
      "id": "345:794",
      "isSharedExample": false,
      "name": "-canSubscribeToNewsletter"
    }
  ],
  "includes": [],
  "its": [
    {
      "body": "\n      NSAssertTrue([subject canSubscribeToNewsletter]);\n    ",
      "id": "489:568",
      "name": "is true",
      "parentId": "389:579"
    },
    {
      "body": "\n      NSAssertFalse([subject canSubscribeToNewsletter]);\n    ",
      "id": "695:776",
      "name": "is false",
      "parentId": "585:787"
    }
  ],
  "spec-name": "UserSpec",
  "vars": [
    {
      "body": "__block User *subject",
      "id": "0:263"
    }
  ]
}

Interpreting JSON Results

To correctly generate an XCTest test from a spec, we needed to perform multiple tasks on matched nodes in order to get valid Objective-C code that Xcode would be able to execute. For the sake of convenience, I decided to do the rest of the work in Ruby, given the vast number of gems — like Thor and activesupport — available to create command-line tools and easily manipulate strings.

Generating an XCTestCase Class

It’d make for an excessively long post to go over all the tasks involved in the transformation process whose main purpose is to replicate the flow of control of a spec. So instead, here’s a list of the most relevant ones I had to perform on the resulting JSON file to generate the XCTestCase class:

Generate the XCTestCase skeleton class.
Copy over the inclusion directives.
Copy the variable declarations and handle potential duplicates coming from different contexts of the same spec.
Generate a main setup method that overrides XCTestCase’s setUp method and other setup methods specific to every test from the elements inside befores in the JSON file.
Generate a teardown method from the spec’s after calls.
Generate test methods with names computed according to the context they’re in and insert calls to the corresponding setup methods to configure the state before running the validations.
Format the output using clang-format.

To take care of these requirements, I created multiple parsers for processing the JSON and outputting different parts of the XCTest test. The resulting tool is a Ruby gem that:

Executes the parser based on LibTooling that generates the JSON file.
Calls all different parsers to generate the XCTestCase class and merges their results together.
Formats the resulting XCTestCase class using clang-format.

Results

Once the tool was created, transforming a spec into XCTest went from taking hours to taking only a couple of minutes. This allowed us to transform our 900+ specs with confidence and ease. The last spec was recently transformed and we were consequently able to remove Specta from our codebase and free ourselves from this third-party coupling, making the project a total success! 🎉

But as is the case with everything, it wasn’t all perfect. So here are some points to keep in mind if you are considering building a refactoring tool with LibTooling:

Poor documentation — The documentation on how to create these tools is limited, and on top of that, the documentation available is often outdated since the C++ API offers no API stability guarantees.
Edge cases — Even after having created the ASTMatchers to match the specs, we had to go back to and expand them to support edge cases we hadn’t detected before. This is natural when refactoring so many different sources — especially when they’re written in a language that allows for equivalent expressions to be written in several different ways.
Missing matchers — Despite the large number of matchers available, sometimes you might want to match something a matcher is not available for, forcing you to write your own custom matcher.
Working on C++ — It depends on your level of writing and reading C++, but if, like me, you’re used to working with more high-level languages, using C++ to write a refactoring tool might feel like adding complexity to a task that can, by itself, be quite daunting.

Conclusion

Creating your own refactoring tools requires an upfront work investment, and whether or not this investment is justified depends on each case. For the right situations, building a refactoring tool that is tailored to your problem can prove to be a high-return investment. For us, we went from needing hours of tedious work to transform a spec to being able to just run a command and get the output instantly! This type of automation results in freed-up resources for other tasks like building new features, which is a huge benefit for companies dealing with large amounts of legacy code.

DeveloperBridge: SoundCloud’s Program for Training People from Diverse Backgrounds to Become Engineers

Fri, 17 Jul 2020 00:00:00 GMT

DeveloperBridge is a year-long, full-time, paid traineeship program where participants learn from and work with engineering teams at SoundCloud. The DeveloperBridge program was launched in 2017 with the goal of further supporting diversity and inclusion at SoundCloud and giving back to the tech community. Now it’s 2020, and we just kicked off the second edition of the initiative.

The program is designed to:

Give back to the community and support the Berlin tech scene by creating junior engineering roles that require no previous experience.
Provide an opportunity to people passionate about engineering and who haven’t yet had a chance to kickstart their careers.
Support underrepresented groups and people from marginalized communities.
Promote junior talent.
Improve diversity in the organization and create a more inclusive environment.
Benefit from fresh perspectives and be reflective of our diverse user base.
Give our engineers an opportunity to mentor and coach.
Hire trainees to full-time positions at the end of the program.

Organizing such a program is a logistical challenge that requires cross-team effort and collaboration. We started recruitment in January 2020, and in the job description, we were transparent about our offering, process, and timeline. To create a diverse pipeline and attract our target audience, we promoted the program on different platforms and at various organizations, such as ReDI School of Digital Integration, WomenTech, Women Who Code, and the Women Who Go course, which SoundCloud hosted in the past.

After one month, we received an astonishing number of applications — 400 to be exact. Together with volunteers from Engineering, we screened the CVs and filtered out applicants who were already professional or full-time developers. The next step was a technical one, consisting of two take-home programming tasks. Since our participants are not experienced engineers, we didn’t require them to actually complete the tasks. Instead, we focused on their thought processes and problem-solving skills.

At the end of this, we invited the 21 highest scorers to the final onsite day. During the day, the applicants learned about SoundCloud, took part in introduction sessions with engineering leadership and DeveloperBridge 2017 alumni, and met members of the SoundCloud team for informal lunches. They also had two short interview sessions: pair programming and barkeeper (barkeepers are senior engineers who represent the technical excellency, culture, and mindset of SoundCloud).

Organizing the onsite day was a huge logistical task that required support from volunteers across the whole organization. In the end, 37 people supported the onsite day by interviewing, conducting introductory sessions, chaperoning, and hanging out with the participants during lunch. It was great to receive this support from the engineering team.

Best of all, the onsite day was successful: We made eight offers instead of the seven we had initially planned for, and all of them were accepted!

Here is what one of the finalists, Kaylee, thought of the recruitment process: “From the beginning until now, it has been an exciting, challenging, fair, and rewarding experience. The programming challenges brought an extensive range of brain stimulation, while the personal interview session at SoundCloud was welcoming and relaxing. Live coding with the senior developer was also fun and interesting.”

The next step was finding projects and teams for the trainees. We wanted this process to be as inclusive and interactive as possible, in order to cater to the interests of the trainees. We came up with the idea of a “Pitch Day,” which is where engineering managers would pitch their teams and projects to the trainees. That way, the new joiners could also meet their future colleagues and learn more about the actual tasks and projects. Neslihan, one of the participants said: “I am impressed with the genuine interest in finding us the project that we would be passionate about.”

The preparations didn’t stop there. Volunteers from Engineering started working on the training program to onboard the new joiners to all things SoundCloud: tech stack and systems, ways of working, internal policies and processes, and career progressions. In addition to the training program, the new joiners will have mentors and buddies who are also volunteers from the engineering team.

In addition to the benefits for the new joiners, DeveloperBridge has already had huge positive effects on the company. For example, the training coordination group is planning on scaling its training program into a regular onboarding, junior colleagues had the chance to participate in a recruiting process, and colleagues were excited about running the program — this is something they drew additional motivation from in their day-to-day work.

Organizing and executing DeveloperBridge has been a great learning experience for the recruitment team. It is also extremely rewarding to give back to the community by supporting junior talent and giving opportunities to people passionate about programming. We so excited for our eight trainees — Ekaterina, Kaylee, Margaret, Tristan, Oscar, Matan, Neslihan, and Evgenija — to start their SoundCloud journeys.

Changing the Interview Process during Remote Working

Tue, 30 Jun 2020 00:00:00 GMT

Please also see Part 1: Rethinking the Backend Engineering Interview Take-Home Challenge and Part 2: The Recruiting Perspective and Results.

In the last two blog posts, we talked about how we rethought our backend engineering challenge as a way of improving numbers for recruitment. In the same vein, this post talks about how we changed our hiring strategies to adapt to hiring remotely during COVID-19.

Onsite engineering interviews at SoundCloud include whiteboarding, pair programming, barkeeper,¹ product manager, and engineering manager interviews. Because of COVID-19, SoundCloud switched to working completely remotely in March of 2020. However, we didn’t let this new setup put a stop to our recruiting efforts.

Scheduling Interviews

In the past, we had all of a candidate’s interviews scheduled onsite for the same day at our SoundCloud office space. As we moved to a full remote work environment, we made all the interviews virtual. Virtual interviews made us more flexible, as we could spread the interviews across multiple days to better accommodate the availability of the candidates and interviewers.

Video Calls for Interviews

Out of the five aforementioned interview types, the latter three types were the simplest to make work in a remote setup because they mostly consist of a verbal dialogue. To carry out these interviews, we set up video calls between the interviewers and the candidates. The candidates were given details about this in advance so that they could try out the setup and make sure they had good audio and video connectivity.

When doing this, we discovered we could get similar signals about a candidate’s proficiency through a video call as we would in an in-person interview. The interviewers also found it was easier to take notes during the call without it being awkward. And the candidates seemed more comfortable taking part in the interview in spaces they were familiar with.

We used Slack to coordinate with one another internally when someone finished an interview to make sure we allowed enough time for the candidate to take a quick break and refresh before the next interview began. This was also a helpful tool for communicating with recruiters — for example, if the candidate couldn’t join on time or had to drop off the call because of any connectivity issues. There was even an instance where we sent the incorrect VC link to the candidate, and having a running conversation on Slack helped us quickly communicate with the recruiter and thereby the candidate.

Tools for Whiteboard and Pairing Interviews

Whiteboard and pairing interviews were a bit more challenging to adapt to a remote setting, as a big portion of our technical assessment happens over these sessions. Running these interviews in a remote context forced us to invest in better tools, familiarize the interviewers with them, and send out material to candidates in advance to allow them to also understand the tools.

For the system design interview, we used Miro as a whiteboard tool. The candidates could use it to sketch out the system design and walk through their ideas. The first time, we ran into unanticipated permission issues that had to be resolved just before the call. We quickly learned from this and started setting up the Miro board and sharing it with candidates in advance. These virtual boards also made it easier for interviewers to go back and review the work if necessary before filling out the feedback for candidates.

For the code pairing session, we opted to use video conferencing along with screen sharing. This allowed candidates to use the IDE of their choice. However, the interviewer missed out on being able to contribute or point to specific lines of code. It was also difficult when candidates paused and explained before or while making their code changes. As a result, the interviewers needed to be explicit about asking probing questions and clarifying things to be able to keep up with a candidate’s train of thought.

Feedback

In addition to us learning directly through this experience, we tailored our post-interview feedback forms from the candidates to get their feedback on the interviewing experience. We also collected feedback from the interviewers to make possible improvements and reevaluate the tools.

Ongoing Challenges

The entire experience of running interviews has been an ongoing learning process for us and we are still trying to find optimal solutions to some of the challenges. For example, we haven’t found a good replacement for the casual lunch session we had in our onsite interview day. We tried having a casual video call as an alternative, but we haven’t found it to be as equally informal as having lunch together.

Another challenge has been to find a best practice for sharing access to our remote interview tools with the candidates within a reasonable timeframe and in a way that makes it clear how to set up and use them.

Results

When initially faced with the challenges of moving our interviews to a remote setting, it seemed like the overhead of setting up tools and video conferencing in advance would be a lot of work on our part. However, through feedback and iteration, this became a seamless part of our hiring flow. As a result, we were able to continue to hit our hiring goals, even in the midst of a global pandemic.

Overall, running our full onsite interview days in a completely remote setup has been a huge learning experience. It allowed us to be more flexible, empathetic, and inclusive, and we will continue to use these learnings to improve our hiring moving forward.

Barkeepers bring an outside perspective to the hiring team. They represent the technical excellency, culture, and mindset of SoundCloud. They have a history of hiring effective engineers, experience running interviews, and technical depth in multiple areas.
↩

Technical Interview Reform, Part 2: The Recruiting Perspective and Results

Fri, 26 Jun 2020 00:00:00 GMT

Please also see Part 1: Rethinking the Backend Engineering Interview Take-Home Challenge

Among the engineering groups at SoundCloud, backend engineering has always had the highest number of open roles — at times reaching 10 positions. To fulfill headcount plans in the current fast-paced market, recruitment needs to move quickly and put efficient processes in place.

At SoundCloud, we have designed a three-step process to enable us to move candidates through the pipeline quickly: recruitment screening, challenge, and final onsite round. Each step has to provide a clear signal as to if the candidate is a good match for the role we’re trying to fill.

However, in the past, we’d experienced challenges with high dropout and time-to-hire rates, as well as with the low ratio of challenges passed to offers made. Having analyzed the data, we identified the code challenge step as the bottleneck from the recruitment perspective.

According to the candidate feedback, it took anywhere from 10 to 20 hours to complete the backend code challenge task. The turnaround between a candidate receiving the test and sending it back was three weeks on average, and it took around a week for engineers to review a submission.

This is a big time investment to ask from candidates, and it is hard for them to commit to such a time-consuming task while having a full-time job. It is even less inclusive of people with families and kids or more junior candidates. As a result, the dropout rates during the challenge steps were high. All these factors contributed to the lengthy process and not always the best candidate experience.

On a positive note, people who invested the time to complete the challenge were definitely motivated to join SoundCloud. :)

Process Changes

To initiate the change in the process, the recruitment team worked closely with the members of the leadership and engineering teams. We analyzed the data, proposed a possible solution, and issued an internal RFC (“request for comments”). Recruitment and engineering collaborated on developing an efficient process and constantly iterating based on the candidate feedback.

To onboard challenge reviewers and interviewers, we introduced challenge review squads. A squad consists of more experienced and less experienced reviewers who pair and learn from each other. The squads change every 6–8 weeks so that everyone can participate and get onboarded to become reviewers of the new challenge.

In addition, engineers are able to better manage their time with this process in place: they commit for 6–8 weeks, and when that time has passed, they can be free of the challenge reviews for the rest of the year if they want to.

Results

The new backend challenge was implemented in October 2019, and the candidate feedback has been great: The time commitment is now 5–6 hours. The new PR review tasks received a positive response as well: Candidates often find this part of the challenge interesting, and they consider it a refreshing change from the norms they encounter in technical interviewing.

The turnaround time between the candidate receiving the challenge and sending back the solution went down to an average of seven days (a decrease of 60 percent). The new challenge also provides stronger signals: Conversion rates (challenge completed/offer made) increased from 35 percent to 50 percent.

As an interesting side effect, the new backend code challenge positively contributed to the company’s focus on diversity in hiring: Since implementing it in October 2019, 30 percent of the offers we’ve made have been to non-male engineers.

After seeing such significant positive results of the new backend challenge process, we are now working on making similar changes in other engineering coding interviews.

What about COVID-19?

An upcoming bonus post will outline changes we’ve had to make and challenges we’ve faced adjusting our technical interviewing process in the COVID-19 era.

Technical Interview Reform, Part 1: Rethinking the Backend Engineering Interview Take-Home Challenge

Thu, 25 Jun 2020 00:00:00 GMT

Most SoundCloud backend engineers have good feelings about the old backend engineering take-home challenge. It’s commonly been characterized as a fun, thought-provoking coding problem.

However, most of us took a significant amount of time to complete it. In fact, reports of having worked on it for 12 hours or more, over a timespan of two or more weeks, are not uncommon to hear. Candidates coming through the hiring process reported in feedback to their recruiters that they were straining under the time burden, and we clearly saw this reflected in the form of a significant dropoff at this stage in our hiring pipeline data.

This makes intuitive sense: Many great candidates have good jobs and busy personal lives. We want to talk to as many qualified candidates as possible, but to do that, we need to minimize the chances that our interview process itself gets in the way.

As a result, engineering management challenged engineers and recruiters to improve the situation. We set out to patch the obvious problems, but we ended up comprehensively rethinking the backend engineering take-home challenge.

The Tryout

The ability to compose, change, and critique programs is only one dimension of an engineer’s job — but it’s an essential one. Similar to an orchestra or acting audition, we use the take-home test to get an initial sense of how a candidate performs. We judge the performance with respect to what we believe it takes to be successful as an engineer in this organization.

The form a “code tryout” should take is a hotly debated topic, in our view, partly because contextual factors heavily constrain the format possibilities. It’s clear to us that there’s no one ideal answer that fits all engineering organizations.

The take-home challenge format has historically been a natural fit for SoundCloud for a couple of key reasons:

We hire engineers from all over the world, so the decision to bring a candidate onsite means there’s a significant time investment to be made by both the candidate and SoundCloud. While the onsite is certainly no guarantee of an offer, a key goal is to maximize the odds that candidates brought onsite will excel and get an offer.
This format presents a unique opportunity — and in the very first stage of a candidacy — to eliminate unwanted bias in evaluating candidate performance. We treat the take-home challenge as a kind of anonymous audition, and we do our best to ensure reviewers do not have any knowledge of irrelevant personal attributes of a candidate when judging submissions.

In addition:

We want our initial evaluation of a candidate to be based on what they can do given more ideal conditions than short onsite coding interviews allow for.¹ This means having enough time to really concentrate and produce work they’re proud of. Candidates often report that this is a refreshing change when compared to industry norms.
SoundCloud engineers who review submissions are more confident in their ability to be fair and consistent if they are able to compare and contrast work that is technically consistent from candidate to candidate over time. In part because the programming problems are consistent, experienced reviewers can productively “train up” those who are new to the review process.

Same Coding Problem, but Faster to Complete and Review

After much discussion, we guessed we could take the same problem, change the format a little, put some modest constraints on how the solution could be solved, and still get as good of an indication of a candidate’s likelihood to succeed in the onsite interview as we did with the old challenge.

Here’s what we changed:

Instead of building from scratch, it’s now a refactoring challenge plus a small feature extension. It comes complete with a tester program that passes “out of the box.” The candidate’s first goal is to keep that tester passing while making the code a lot better.
Partly as a consequence, we offer the choice of selecting from six languages to solve the problem, with an emphasis on languages that are popular in the industry and academia. In the past, candidates could choose any language they wanted, but with a refactoring exercise, this is no longer possible.
We decided to restrict solutions to using only the standard library for the chosen language.
We revised instructions and READMEs to be more thorough and clearer in what our expectations are.

This plan was motivated by a goal to reduce the burden on the candidate. But we quickly realized that reducing the burden on reviewers is almost as important. Efficient and consistent review means we can get back to candidates with a response more quickly — but it also opens up opportunities for us to be fairer.

Each submission is evaluated by two people independently. They then meet up and file a “consensus review” signed off by both of them. But code review is a human activity, and reviewers will often end up in disagreement over a submission. When reviews are quick, it’s easy to pull in tiebreaker reviewers and still promptly get back to the candidate with a response.

Adding Pull Request Review

Inspired by Slack engineering’s pull request-based challenge, we decided to add a small PR challenge of our own to the backend engineering take-home challenge.

PR review allows us to get some insight into how candidates give critique, and the additional code problem allows us to probe a couple other technical areas. In exchange for a modest amount of extra time to complete the code challenge, candidates get an important new channel through which they can show reviewers how they approach these important aspects of the day-to-day work in engineering.

Results

Please see part 2 of this series for the recruiting take on the coding challenge reform, along with a report on both quantitative and qualitative results.

Thank You

Much of SoundCloud engineering, recruiting, and management contributed to this in some way. Big thanks go out to:

Anya Voronova
Ben Stahl
Matthias Käppler

We would also like to thank Slack Engineering for their time and advice as we built our version of the PR challenge, in particular:

Chase Rutherford-Jenkins
Maude Lemaire
Will Kimeria

You know the drill: You have 60 minutes, a web browser coding environment or whiteboard, the problem is straight out of an Algorithms and Data Structures class, and the interviewer mostly sits there in awkward silence. Some companies see such high-pressure conditions as a good thing. We don’t.
↩

Announcing Twinagle: Twirp and Protobuf for Finagle

Fri, 12 Jun 2020 00:00:00 GMT

A previous post on this blog ended with the following paragraph:

“We might also replace JSON with a more efficient serialization protocol like protocol buffers. This would further improve performance and should result in less handwritten serialization and deserialization code but this still needs more experimentation and investigation.”

To follow up on the above, this article announces the release of Twinagle, an open source implementation of the Twirp protocol for Scala/Finagle.

Background

The backend of SoundCloud is designed around a set of microservices. As detailed previously, services expose a REST-like interface and use HTTP and JSON to communicate. Our services are usually implemented in Scala, making use of Twitter’s Finagle library, but we have a fair amount of Ruby and Go powering production as well.

This choice has worked well for us, because tooling around HTTP and JSON is widespread and allows for easy debugging by humans. However, the very flexibility offered by the stack has, in significant ways, slowed down development. This is because, for each endpoint, developers need to decide:

Which HTTP method to use
Whether to expose parameters in the path or as query parameters
How to structure the response
How to name fields in JSON objects
How to represent common data types, such as time stamps and durations
How to represent error conditions

Internal guidelines and recommendations have only helped so much — variations tend to creep in between teams and over time. Small inconsistencies between server and client implementations have caused outages (e.g. a client mistakenly made use of camelCase for field names but the server expected snake_case).

The “artisanal” approach to HTTP API development and consumption can be frustrating: Even where guidelines are followed perfectly, an engineer still needs to forensically examine application code in order to get comprehensive information about how any given endpoint works.

As a result of discussion in our weekly engineering forum, a subgroup of engineers decided to explore alternatives. The group honed in on Twirp.

Why Twirp?

Twitch engineering created the Twirp project, which is a Go-based server implementation of the Twirp wire protocol:

“The Twirp wire protocol is a simple RPC protocol based on HTTP and Protocol Buffers (proto). The protocol uses HTTP URLs to specify the RPC endpoints, and sends/receives proto messages as HTTP request/response bodies.”

Engineers compose API contracts in the protobuf Interface Description Language (IDL). They then use those contracts to generate server stubs and clients using protobuf tooling. The generated code takes care of HTTP request/response parsing and serialization to either JSON or the binary protobuf format. All requests are simple HTTP POST. And the wire protocol itself is simple and well-thought-out (thank you, Twirp engineering).

Twirp seemed like a great fit: It enabled us to start introducing IDLs to simplify microservice development in a way that incrementally “slotted into” a complex backend. We looked at gRPC but decided against it, and our reasons for this decision match very well with those stated in the original Twirp blog post). We already run some Go services, and Twirp has a Go implementation we could just drop in and use.

However, more than 70 percent of microservices we deploy are based on Scala + Finagle…

Twinagle = Twirp + Finagle

Due to the simplicity of the Twirp protocol, we decided to implement it ourselves. And we are pleased to announce the release of Twinagle, our implementation of Twirp for Finagle.

The project is split into two components, outlined below.

An SBT plugin that takes a protobuf interface and uses it to generate:
- a trait that defines the service’s interface
- code that implements both the server-side and client-side of this interface. This includes (de)serialization of requests and response bodies, error handling, and transmitting data over the wire.
A runtime library that implements the logic to route requests, performs (de)serialization of messages, and integrates with Finagle.

For instructions and code examples, please see the Twinagle project page.

Migrating a Service

About the Track Authorization Service

The first high-impact production service at SoundCloud to adopt Twinagle was our track authorization service. This service is responsible for deciding how users can access tracks on SoundCloud based on the user’s location, client application, and subscription status. Should they hear an ad? Should they be able to see the track exists in other countries? Every time the SoundCloud backend handles a track identifier, this service gets a request. Needless to say, this service receives a large number of requests.

In addition, JSON response bodies for track lookups can be large: Up to 100 tracks may be requested at a time, and in practice, the average request is for 25 tracks. However, the structure of these responses is not complex; it consists of a long list of relatively simple objects.

The track authorization service is only used by one client application, meaning we only needed to perform one client migration. A combination of systems-integration simplicity and handling of significant request load made this service a natural early candidate for conversion to Twinagle.

Code Conversion

First we implemented a new Twinagle-based endpoint in the authorization service. We defined messages in the protobuf file that are based on existing domain model classes found in the service’s codebase, and we implemented a thin layer to convert between both. In exchange for this small amount of extra boilerplate code, we were able to reuse the existing business logic code, untouched. As a result, we could limit potential bugs to serialization and deserialization code.

Once we deployed the new endpoint, we brought the Twirp library and protobuf file into the client application. We took a similar approach to the client side as we did on the server, building in a thin (de)serialization layer that isolates the existing business logic.

Rollout

Then we had the option of using both the REST and Twirp versions of the endpoint from the client app. We redirected client traffic using a percentage-based rollout flag in two phases. In the first phase, we made duplicate requests and compared the results. This meant that with a rollout percentage of 1 percent, the client made a request to both the REST and Twirp endpoints 1 percent of the time and reported discrepancies in the responses, but returned only the REST endpoint’s response. We caught a few critical bugs this way. We then moved on to use the Twirp endpoint only — and to stop duplicating requests. We rolled out to 100 percent with no outages and no bugs on the first attempt as a result of this process.

Findings

The results of this migration were positive but produced some mild surprises. We expected a drop in response time, but we only saw marginal benefits in the 90th percentile, and no benefits in the 50th or 99th percentiles. This was likely because most of the response time comes from accessing the database, and not from request/response serialization and deserialization. Additionally, the variability of 90th percentile latency dropped significantly.

We saw a huge drop in bandwidth between the two services; network usage decreased by more than 40 percent when using binary protobuf instead of JSON! We also discovered that many fields in the response were being ignored by the client application, and we were able to confidently clean things up in the new Twirp endpoint. Finally, we deleted all of the custom JSON serialization and deserialization code in both services, which is always a satisfying developer experience!

Conclusion

This initial service conversion confirmed our hopes for Twinagle: Protobuf contracts make services easier to understand, allow us to get rid of an entire class of low-value code, and even result in some nice efficiency benefits. We are actively converting old services to use Twinagle, and all new SoundCloud services are now Twinagle-based. To try it out for yourself, check out the project page.

Thank You

The implementation and rollout of Twinagle has been a great cross-team effort. Special thanks to:

Bernd Louis
Frank Calma
Jorge Creixell
Niko Dziemba
Sean Treadway

And extra special thanks to Twitch engineering, for Twirp!

Stretch Opportunities for Engineers

Wed, 20 May 2020 00:00:00 GMT

Stretch opportunities are tasks or projects that are a bit beyond your current skill or knowledge level and that allow you to improve and learn new things. They are also essential for growth. At SoundCloud, some teams introduced a process so engineers can take on stretch opportunities. In this blog post, I’ll explain how this works for the Content Team.

The need for stretch opportunities within the Content Team was raised in a retrospective. An engineer stated that they wanted to broaden their role and be exposed to requirement gathering, meetings with stakeholders, and driving new projects. Other engineers also showed interest, so we began discussing as a team what that would mean and how we could accomplish this. Based on these discussions, we ended up introducing the Project Lead role.¹

The Project Lead role is a voluntary non-permanent role that a team member gets assigned for the duration of a project. We assign the role for a new project when the current project is coming to completion and the new project will become the next main focus for the team.

Role Responsibilities

The team member in the Project Lead role takes over some of the responsibilities previously held by the Engineering Manager (EM) while still being supported by them. The responsibilities of a Project Lead are to:

Be the technical representative for the project within the team but also externally. We expect that they answer questions about the implementation and participate in meetings with stakeholders.
Write the Request for Comments (RFC) for the project and request and respond to feedback from the team and external stakeholders.
Be the driver for the execution and delivery of the project.
- Surface the work that is left and break it up in smaller chunks together with the team.
- Identify what work can happen in parallel and encourage team members to help out.
- Estimate and communicate how many iterations are still needed to complete the project.
- Identify blockers and resolve them or ask for help.
Make sure a retrospective happens when the project comes to an end, and find a facilitator for it.

Role Restrictions

In addition to the responsibilities of the Project Lead, we also came up with a list of restrictions:

The Project Lead can’t unilaterally make decisions. In case of unresolved conflict, the EM and/or Product Manager (PM) will make the final decision (for example, for different opinions about technical implementations or product/prioritization or anything else).
The Project Lead shouldn’t work alone. There can be some preparation work that happens in isolation for some time — for example, when writing the RFC — but we want to work as a team on all projects, as we think that getting different perspectives and leveraging the diversity of the team leads to better outcomes.
We stick to one Project Lead per project. In case of unplanned absence, the EM and PM should be able to cover all the expectations and responsibilities. In case of planned absence (vacation), the Project Lead should find a replacement or alternative solution before leaving, and they should do a handover.

Learnings

After using the Project Lead role for several years, we’ve seen many advantages.

Because the role is temporary and there is support from the EM and PM, it is a safe way for team members to take on additional responsibilities, and it’s an opportunity for growth and getting feedback.

The voluntary nature of it means that team members can step into the role for a project they feel comfortable with. We have no restrictions on the experience level needed to step into the role, however, we noticed less experienced engineers will typically volunteer for smaller projects, while more experienced engineers are challenged by taking on larger projects with potentially more stakeholders and sometimes dependencies on other teams.

We see that engineers who took on the Project Lead role, especially for larger projects, are in a better place for getting promoted because of the extra responsibilities they took on and the additional exposure they got outside the team. In fact, some of the Project Lead role responsibilities and expectations match well with some of our Level 3 Engineering Level expectations, in particular:

Drives larger projects reliably by delegating and coordinating
Coordinates work across team boundaries
Keeps the team honest and accountable to commitments
Constantly puts effort into understanding the business/product big picture

Furthermore, delegating some of the responsibilities previously taken on by the EM means that the EM will end up with more time for other things, which can be welcome, especially when leading larger teams.

Conclusion

The Project Lead role allows engineers to take on stretch opportunities. It’s also a way of acknowledging all the extra work that might not always be visible otherwise (what Tanya Reilly calls glue work).

The fact that the role is voluntary and on a per-project basis is great, because it means that everyone who wants to take on the role will have a chance to sooner or later. Furthermore, it also works for people who prefer to focus on contributing but not leading.

The Project Lead role concept may or may not work in your environment, but I believe it is important to create stretch opportunities for your engineers so they can get exposed to new skills, remain engaged, and grow.

When talking about a project in the context of this blog post, I mean a software development project that has a purpose of reaching a certain outcome. A project can be part of a larger program consisting of multiple projects focused around a common theme.
↩

Open Sourcing Intervene

Tue, 05 May 2020 00:00:00 GMT

A little while back, the web team at SoundCloud got an urgent report that our upload page looked weird in the US. Web engineering is based in Berlin, and it looked fine outside the US. However, we quickly identified that a change we’d made recently was conflicting with some custom visuals that were only being run for US visitors.

Rolling back the change was the easy part, but we needed to emulate the API response in Berlin in order to replicate and fix the issue properly. We develop against a single HTTPS test instance of our BFF, and it’s not trivial to add extra mocking directly into that BFF code (it’s the same code that gets deployed to production). We ended up adding a breakpoint in the browser dev tools where the response was received, and we were able to manually inject the US version of the response. It wasn’t pretty and we weren’t proud, but we were able to replicate and fix the issue.

Although we’d resolved the issue, I still thought: “There has to be an easier way. After all, we just need to override one endpoint and proxy the rest to the real server, so how hard can it be?”

Finding a Solution

I looked into a few existing solutions, but none were as all-in-one as I’d have liked. I wanted to be able to start a command locally and override specific endpoints and have the HTTPS certificates generated and trusted automatically. I also wanted overriding and editing responses to be possible in languages frontend engineers are familiar with, and I wanted that this configuration could be checked in to the repository to be shared with other engineers.

And so intervene was born.

Intervene

intervene is a node.js process that works like a man-in-the-middle proxy (except it’s not an HTTP proxy, as it only proxies to a single host and appears to be a single HTTP server). It enables the altering of requests and responses for mocking purposes. The configuration is written in TypeScript, so that frontend engineers are using a familiar language (as TypeScript is a superset of JavaScript, using plain JavaScript is fine too).

It automatically generates and trusts self-signed certificates for HTTPS, so in every scenario, only a single command is necessary to create and start a proxy, with no extra steps to perform manually.

At SoundCloud, we used it to build the frontend for a major creator feature before the backend was ready. We added full typing support to intervene in order to use it to discuss the APIs as they were being built. Our mock became the documentation around which discussions were had on what the API should look like. We were able to slowly turn the mocking off as the backend endpoints started to work, and we were even able to patch up responses when the backend implementation wasn’t finished and was only partially working. It’s now become a key piece of our development tooling, with many mocks checked in to frontend repos so that engineers can quickly turn on mocking for common scenarios.

Open Source

We’re really excited to be able to open source this tool today. It’s been invaluable not only in development, but also in recreating edge cases and a whole bunch of other scenarios where we just need to alter a header, request, or response to quickly try something out.

To install globally (so you can run intervene from anywhere):

npm install -g intervene

You can also install it as a devDependency of your project:

npm install --save-dev intervene

Configuration

A configuration can be automatically generated using intervene create https://api.mycompany.com (or whatever host you want to create a configuration for). In this example, it generates a file called api.mycompany.com.ts.

So what does a minimal configuration look like?

import { ProxyConfig } from 'intervene';

const config: ProxyConfig = {
  target: 'https://api.mycompany.com'
};

export default config;

Let’s add a static JSON response to an endpoint:

import { ProxyConfig } from 'intervene';

const config: ProxyConfig = {
  target: 'https://api.mycompany.com',

  routes: {
    'GET /cats': {
      collection: [{ name: 'Charlie' }, { name: 'Whiskers' }]
    }
  }
};

export default config;

Saving the file automatically reloads the configuration, so there’s no need to restart after saving. GET in the route name is the default, so it can be excluded for the sake of brevity.

Now if we make a GET request to https://api.mycompany.com/cats, we’ll get some static JSON back:

$ curl -k https://api.mycompany.com/cats
{"collection":[{"name":"Charlie"},{"name":"Whiskers"}]}

The -k is needed in the curl request, because otherwise curl will complain that the certificate is self-signed and not trusted. However, if the URL is opened in Chrome, it will Just Work(™), because intervene has already added the certificate to the operating system’s list of trusted certificates (this list isn’t used by curl).

Note that it doesn’t matter if there was an existing /cats endpoint — either way, the JSON specified in the configuration is what is returned and the real server is not called. CORS is automatically enabled for all endpoints from all origins. This can be configured if more specific CORS rules are required.

Altering Responses

So intervene can easily return static JSON, but what about if we want to modify the response from the server? Let’s imagine there’s a GET /dogs endpoint that returns the names of some dogs in the same format as the /cats endpoint we mocked earlier:

const config: ProxyConfig = {
  target: 'https://api.mycompany.com',

  routes: {
    '/dogs': async (req, h, proxy) => {
      // Get the response from the real server.
      const response = await proxy();

      // Mutate the response (add an `age` property to each dog).
      response.body.collection.forEach(dog => {
        dog.age = Math.ceil(Math.random() * 18);
      });

      // Return the response.
      return response;
    }
  }
};

If a function is used as a route handler, it gets called with the request details, an instance of the hapi.js response toolkit, and a function that returns a promise of the response from the server.

Changing the req object before calling the proxy() function will change the request that gets made to the server. For instance, changing req.url.pathname to /cats before calling the function will mean that the proxy() function will actually call the /cats endpoint on the real server when a request is made to /dogs.

Custom Responses

Using the hapi.js response toolkit, it’s possible to create custom responses. We have used this extensively at SoundCloud to develop error handling routines by mocking an endpoint to return an error.

For example:

const config: ProxyConfig = {
  target: 'https://api.mycompany.com',

  routes: {
    '/whoops': (req, h, proxy) => {

      // Create a response.
      const response = h.response({ error: “whoops” });

      // Set the status code.
      response.code(500);

      // Add a custom header.
      response.header(‘x-error-id’, ‘123456’);

      // Return the response.
      return response;
    }
  }
};

How Does It Work?

intervene initially writes an entry to /etc/hosts, adding the address of the server to mock as 127.0.0.1 (localhost). This means requests to that server now get routed to 127.0.0.1. If the server is HTTPS, it generates a self-signed certificate and calls operating system commands to add it to the set of trusted certificates. It then starts an HTTP(s) server (based on the fantastic hapi framework). Routes from the configuration file are added, and then default routes are added to proxy all remaining requests to the real server.

In order to proxy requests, when a request comes in, intervene performs a real DNS lookup to ensure it ignores the entry in /etc/hosts. These lookups are cached and use the TTLs returned in the DNS response.

Where to Find out More

There’s a lot more documentation on the intervene.dev site. Additionally, feel free to raise an issue or send a PR to the GitHub project if you have questions or suggestions.

Periskop: Exception Monitoring Service

Mon, 16 Mar 2020 00:00:00 GMT

Periskop is an exception monitoring service that we built here at SoundCloud. It was designed with microservice environments in mind, but it’s also useful for long-running services in general. It has been architected for scale and is heavily influenced by what we’ve learned with Prometheus. It uses a pull-based model to efficiently aggregate exceptions in clients, after which point a server component collects exceptions and further aggregates them across instances.

Why Yet Another Exception Aggregation System?

In the past, SoundCloud relied on Airbrake to help troubleshoot and identify problems happening in production. As traffic increased and our services started receiving requests in the tens of thousands per second, small hiccups would send huge bursts of exception reports, in turn depleting our entire monthly budget in just a few minutes, and causing loss of information. grepping or indexing the logs was not an option given the huge volume generated by our services.

Considering the increasing amount of services SoundCloud runs, we needed a solution that would easily discover and scrape services with the minimum amount of manual effort.

Since SoundCloud employees are given the opportunity to use up to 20 percent of time to experiment with new ideas (known here as self-allocated time), this was a perfect opportunity for a group of engineers to get together and build something fun and useful to improve tooling at SoundCloud, and hopefully, for the entire open source community.

How Does It Work?

Periskop is composed of both a client library component and a server component.

The client library provides an exception collector. Every time an exception is handled, it can be added to the collector. Unhandled exceptions can be added to the collector by a top-level exception handler. Once an exception is added:

The collector builds a unique key with the exception’s message and a hash of the stack trace. This key represents a concrete exception type.
The collector aggregates the exception using the unique key and stores the aggregate using an in-memory data structure.
The concrete exception occurrence is added to the aggregate’s list of recent exceptions for inspection. A queue data structure is used to implement a moving window of the most recent exceptions with a configurable size.

Collected exception aggregates are exported via an HTTP endpoint using a well-known schema.

The server component uses service discovery to find all instances of a service. It then scrapes and further aggregates exported exceptions across instances. A simple UI allows navigating and inspecting exceptions as they occur.

Design Tradeoffs

Every design decision comes with a set of tradeoffs that are important to be aware of. In the case of Periskop, we opted for a pull model for collecting exceptions as opposed to the usual push model of similar tools like Sentry. This is advantageous because the pull model scales well with the number of exceptions and instances, and it provides interesting capabilities with little effort, specifically:

Memory used by the client library is bounded by the number of different types of exceptions occurring instead of the total number of exceptions overall. Memory on the server side is similarly bounded, as it uses the same aggregation logic.
If the server component needs to scrape a large amount of service instances, it might end up skipping scrape cycles, thereby reducing the freshness of the data, but it does not require additional resources. In practice, this means that Periskop can handle large numbers of services and instances with few resources.
Creating hierarchies of Periskop instances (known as federation) becomes trivial. A primary Periskop instance could aggregate exceptions collected by secondary, data center-specific instances, thus providing a global view of the entire service stack across data centers.

This model also has some disadvantages when compared to the push model, namely:

If a fatal exception occurs and the process dies, the exception won’t be collected. However, this can be mitigated by the reporting capabilities of orchestration services like Kubernetes or other forms of logging.
The pull model is not well suited for short-lived processes like scheduled jobs. This could be solved by the use of a push-based event gateway, although we haven’t yet had the need at SoundCloud, as it is usually more convenient to use logs to inspect failed jobs.

What Is Coming Next

Periskop is in its early stages and lacks some capabilities. Some of the ideas for improving and extending it that are on the roadmap include:

Server-side persistence of exception occurrences over time
UI filtering and sorting
Federation (hierarchical Periskop servers)
Pluggable service discovery mechanisms
Connectors to other error monitoring services like Sentry

In addition, high-level libraries for widely used web frameworks and client libraries for more programming languages need to be developed for Periskop to realize its full potential.

Periskop is open source and we are happy to accept external contributions! If you find this project useful, we would love to hear from you. Please drop us a line at periskop-maintainers@soundcloud.com.

How (Not) to Build Datasets and Consume Data at Your Company

Tue, 03 Mar 2020 00:00:00 GMT

The topic of datasets is not new, and they have been successfully used at SoundCloud for analytics purposes for a long time. In this blog post, however, we’ll delve into how the culture of building and providing datasets is being adopted in more areas of engineering beyond the domain of analytics.

When we talk about datasets, we’re talking about published datasets. In this case, a team owning the data in question would also own the published dataset and be responsible for delivering it in a particular shape, as agreed upon with the internal consumers, i.e. the end users of the dataset. As a result, the dataset encapsulates any domain intricacies that should not leak into the consumer’s domain.

This post shares our experience of adopting “A Better Model of Data Ownership,” which we blogged about more than two years ago. It discusses our learnings and digs deeper into the specifics and technical details.

In particular, we’ll focus on exploring three questions:

How not to consume data
How to consume data and who should provide it
How to build datasets and what to consider before you start

The process of data management in a data privacy-compliant organization is beyond the scope of this blog post. Whenever the access to or use of personally identifiable information is not justified by the business case, further treatment of data is required, but this is not discussed here.

Examples of Datasets at SoundCloud

Before going any further, here are a couple of examples of datasets we will refer to in the following sections. We deal with artists, tracks, and plays here at SoundCloud, so we’ll draw examples from that domain

Track Metadata Dataset (holds information about a given track)

`track_id`	`title`	`artist`	`valid_from`	`valid_to`	`is_current`
1	Longer, Better, Stronger Passwords	Adept Surfer	2019-02-05	2060-01-01	`true`
2	Think Before You Click	Cautious Mouse	2019-02-05	2019-05-01	`false`
2	Think Before You Click (On My Heart)	Cautious Mouse	2019-05-01	2060-01-01	`true`

Track Rightsholders Dataset (records the label or individual who owns rights to a track in a given territory)

`track_id`	`territory`	`rightsholder_id`	`valid_from`	`valid_to`	`is_current`
1	DE	182	2019-02-05	2060-01-01	`true`
1	FR	182	2019-02-05	2060-01-01	`true`
2	DE	42	2019-02-05	2060-01-01	`true`

The particular representation shown above is called a slowly changing dimension table, since it captures the historical changes of an entity, e.g. a track, over time. For many datasets, SoundCloud has settled on the following shape:

Primary keys — one or more columns that uniquely identify an entity (e.g, track_id, territory). However, there may be multiple entries for the same entity as it changes over time.
valid_from: datetime — the start of the validity period for the given row (technically also part of the primary key)
valid_to: datetime — the end of the validity period for the given row (or a time in the far future)
is_current: boolean — true if the given row is the valid one at the time of dataset creation, and false otherwise.

The dataset may or may not contain gaps in the time dimension for a single entity. This depends on whether or not it makes sense in the domain.

One word on representing entities that have been deleted: Deleted records appear as rows, with the latest valid_to in the past and is_current = false. In that sense, these rows look similar to old versions of an entity with no is_current row. This makes it harder to retrieve all entities, regardless of whether or not they have been deleted. If retrieval of deleted entries is required, it should also be possible to introduce an is_deleted column in the dataset.

Naturally, not all datasets need to have or can have a time dimension. A plays dataset is an example of a simple fact table:

Plays Dataset

`timestamp`	`listener_id`	`track_id`	`territory`
2019-02-15 11:20:00 UTC	1349	1	DE
2019-02-15 11:30:00 UTC	1349	1	DE
2019-02-15 12:05:00 UTC	1066	2	FR

Now that we’ve seen what the datasets actually look like, we will now consider all the various ways in which the company exchanged and consumed data before adopting the idea of published datasets.

How Not to Consume Data

The shift toward using datasets at SoundCloud arrived together with the shift toward batch-oriented processing where it makes sense. This means replacing online services with batch jobs for tasks such as: royalties calculation, report generation for labels and artists, processing content delivered by labels, some aspects of search and recommender systems, etc. All of these examples rely on data owned by other teams in the company, which brings up questions of contracts, SLAs, domain encapsulation, and team interdependency.

Data Consumption: A Story of One Team

This post will now highlight the experience of the Royalties & Reporting team here at SoundCloud, a team that has always been a heavy consumer of data and has undergone a shift toward the use of published datasets. The responsibilities of the team include calculating royalties and building reports for artists and labels.

An example of a report is a CSV file that shows the breakdown of tracks, track metadata, play counts, and associated royalty earnings. In relational database terms, you could think of it as a join between a plays table, a track metadata table, a track rightsholder table, and a royalties table. Usually, these reports are generated on a schedule (as opposed to on demand), so they fit the batch-processing model quite well.

With reference to the following figure, consider the various ways in which these reporting jobs used to consume data in the past. We will discuss the pros and cons of consuming events, snapshotting databases, and consuming live service APIs, all of which we have tried but eventually moved away from.

Consuming Events

Events are great. They are a solid approach for storing the changes of state of a system without imposing a way of how such events should later be consumed or interpreted. Many SoundCloud systems already publish events.

Events can be consumed in batch jobs directly, and were, in fact, the principle way of consuming data in our reporting jobs until very recently. But this approach carries with it a handful of tradeoffs, which are outlined below.

Pros

Flexibility: Events allow for ultimate flexibility, as they capture the most amount of detail and the entire history. This can also be a bad thing, as the details may expose domain complexity.
Excellent time resolution: Each event is timestamped.
Auditing: Investigating and debugging historical changes is free, and nothing is ever deleted. As such, it is possible to recreate system state at any point in time.

Cons

Expensive infrastructure: Each service must support event infrastructure, and this cost is not justified for all uses.
Difficult to consume: Consumers must read all events from the beginning of time. It is impossible to get the current state without reading everything. This does not scale well.
Heavyweight: At SoundCloud, events are encoded as Google Protobuf messages and stored in Hadoop Sequence Files. This makes them opaque, with no ability to selectively consume individual fields without decoding complete events. This limitation is implementation-specific.
Exposes domain complexity: Events tend to include all of the information available for interpretation later on. This interpretation is better handled by the data owner than by every consumer.
Potentially lower quality data: This occurs in cases where the data owner does not consume their own events and instead relies on their databases and APIs. The same may be true of datasets.

Events seemed very promising at first. However, as time went on, we realized they were difficult to work with and the jobs did not scale well after a year or two. The limited ad hoc querying abilities and increasingly slow jobs made development and operations frustrating and costly.

Snapshotting Databases

We did this. We performed full dumps of the database table backing the live service we don’t own.

Pros

Accurate: The data owner and data consumer consume the same data.
Easy to consume: Structured data that can be consumed cheaply and easily.
Easy to develop with: Can be queried in an ad hoc manner for development and debugging purposes.

Cons

Bad coupling: The database model becomes an implicit contract and the consumer needs to be consulted regarding any changes.
No SLA: No explicit contract with the data owner around the quality and availability of data.
Bad domain encapsulation: This exposes internal domain complexity that may be misinterpreted by the consumer. It bypasses any domain processing that happens in the service.
No time dimension: There is no visibility of changes of data over time or deleted entities.

It’s quite clear how snapshotting databases breaks many principles of encapsulation. Additionally, we were spending a lot of time building and maintaining all the datasets we needed ourselves. They were not the responsibility of the data owner, and they were not available to other consumers.

To be clear, snapshotting is a viable option for cheaply and easily building datasets. However, it’s a completely different story when the data owner, and not the consumer, is doing it with full knowledge and understanding of their databases.

Consuming Live Services through APIs

Yes, we did this too. We would make API calls to a live service from inside a batch job.

Pros

Accuracy: This was the most accurate and up-to-date data.

Cons

Bad fit for batch-processing paradigm: Depending on the API design, it would not normally be possible to work with the data in bulk with all the flexibility you’d expect.
Poor performance: It does not scale well and has the possibility of introducing unnecessary strain on the consumed API and its database.
Hard to develop against: Ad hoc querying is easy, but it may not be possible to do in bulk.
No time dimension: There is no visibility of changes of data over time or deleted entities unless the service keeps all the history.

Enter Published Datasets

Published datasets combine some of the positive aspects of the above approaches, but they practically eliminate all the negative ones. To reiterate, published datasets are produced by the data owner following an agreement with the consumer or consumers. We’ll discuss the pros and cons of these below.

Pros

Domain encapsulation: The dataset producer can preprocess the data to hide domain complexities.
Contract and SLA: Follows a well-defined contract between the producer and the consumer of what the shape, correctness, and availability should be.
May be easy to build: Internally built tooling can help build datasets for many use cases with low effort.
Easy to consume: Structured data that can be cheaply and easily consumed.
Easy to develop with: Can be queried in an ad hoc manner for development and debugging purposes.
Clear ownership: The producer assumes the responsibility for maintaining the dataset and fixing issues.
Decoupling: The producer is free to evolve the schema of their databases and events, so long as they maintain the dataset interface the same, when it makes sense.

Cons

Potentially lower quality data: The producer doesn’t usually consume their own datasets — data problems are still possible.
Takes effort to build: Datasets go beyond what can be built using standard tooling (more on that in the next section), and they may still take effort to build. In the cases where a need for a dataset is blocking the consumer but the producer has other competing priorities, the consumer may take on the responsibility of building the dataset for the producer to own and maintain.

How We Build Datasets

There are two primary ways in which we build our datasets: from events, and from database snapshots. As discussed in the previous section, we have experience consuming both events and snapshots in our batch jobs. What we wanted to do differently was to (a) establish a coherent way of building these datasets, (b) create standard tooling to cover the most common cases, and (c) make it the responsibility of the data owner to provide a dataset that was clean, consistent, and timely.

The two methods of building datasets (from events and from snapshots) are not as dissimilar as they may seem. In fact, they share many of the same steps, which becomes apparent when looking at the figure below. Most datasets we produce at SoundCloud take the shape of dimension tables, and they capture the historical changes of an entity over time, as shown in the introduction section. We will assume we’re building such a dataset in the discussion below.

Building a Dataset from Events

In case of events, the workflow is a single step, outlined below.

Roll up events on top of the previous dataset: This step reads the previous version of the published dataset, as well as the events emitted since the last dataset was published. It applies the new events on top of the old dataset, performing any necessary transformations, and writes a new dataset. This is a release candidate for the published dataset.

Note: If you have a very small number of events, you may choose to roll them up from the beginning of time every time you produce the dataset. This gives you the most flexibility and ensures the most recent code is used for rollup.

Building a Dataset from Database Snapshots

The workflow to produce a release candidate is covered below.

Dump a table to a snapshot: The tool reads from the database and writes this snapshot to a temporary table. This is a direct copy of the table, with no further processing. This is not the published dataset.
Transform the snapshot (optional): This is where you’d implement steps such as filtering/removing or renaming columns, or transforming values in the dataset. This is also not the published dataset.
Roll up snapshot on top of the previous dataset (optional): This step reads the transformed snapshot and the previous version of the published dataset and writes a new dataset. This step is required when building slowly changing dimension tables or when the entities deleted from the database must be preserved in the dataset. This is a release candidate for the published dataset.

This multistep procedure allows you to decouple your published dataset from the source table. In the past, we’ve had problems caused by coupling schemas in the database with schemas in the data warehouse, because changing the source becomes impossible without coordinating changes far downstream.

Whether the dataset is produced from snapshots or from events, the shape of the release candidate is the same, so the validation and publishing/promoting steps are the same.

Validate dataset: This is a very important step that’s often overlooked. Testing the code, e.g. through unit testing, lets us gain confidence in the logic of the various steps in dataset production. However, it is usually done using a contrived set of toy data. But when dealing with large amounts of data and moving parts, it is important to also test the data that is actually produced:
Does it contain the expected values in each field?
Does it satisfy the dimension table rules (see the Examples of Datasets at SoundCloud section above)?
Is the dataset a truthful representation of the upstream data? Even a simple comparison of the number of rows in the input and the output can go a long way.
Publish dataset: Usually the dataset is published to the “latest location” (thereby overwriting the previous version), as well as to a location corresponding to the time of publishing (for safekeeping).

At SoundCloud, we have standard tooling for building datasets from snapshots of MySQL databases. This greatly simplifies the creation of new datasets, and as a side effect, it promotes a consistent and common shape of datasets.

Considerations for Building a Dataset

Now that we’ve looked at the two ways of building a dataset, let’s look at the considerations that the dataset producer and consumers need to agree on that would inform the approach taken.

Do you need to record the previous versions of an entity?
Do you need information about deleted entities?
Can entities be deleted and reinstated? How would you like to handle gaps?
How often do you need the dataset updated with the newest data?
Is there a specific time by which an updated version of the dataset needs to be available?
For a dataset with a time dimension, what is the time resolution required? Hourly, every six hours, daily? Highest precision possible?
- Generally, the datasets based on snapshots suffer from the loss of precision.
- In some cases, it may be possible to keep the complete history of changes with exact timestamps in the database, though it makes the service more complex.
- Events can provide high time accuracy, though there may still be a time difference between the event timestamp and the time that the change became effective in the other systems.
For a dataset with a time dimension, how far back is the historical information required? Where do you get this historical data from (e.g. from events, from a different dataset, or inferred from the current value)?

Finally, for some datasets, it may be important to distinguish between two types of changes to an entity:

It may be a retrospective fix or a correction, in which case the change should be considered effective before it is made in the service/database; or
It may be a real brand-new change, effective from the time the change is made.

Conclusion

As more datasets were built at SoundCloud, common patterns led to standard semantics and standard tooling for the most common cases. Even in those cases where standard tooling did not apply, there was already an established pattern for what the dataset could look like and how it could be published.

For heavy consumers of datasets, that also meant that the burden of building and maintaining the datasets was now shared with those who knew most about the data, albeit not always those who had the most experience with building the datasets. But even that was changing quickly. The additional effort and the initial learning curve on the part of data owners led to the organization-wide benefits of shared, well-specified, and well-maintained datasets.

The Magic of Generating an Xcode Project

Thu, 20 Feb 2020 00:00:00 GMT

Planet Earth vs. Jupiter. There is the same order of magnitude between the ratio of these two planets and the amount of code that defined the SoundCloud iOS project before — Jupiter — and after — Earth — generating it using Tuist.

In this blog post, we will discuss the problems of maintaining a complex iOS project, how we managed to simplify the process by writing less and more consistent code, and how we ended up with lower build times as an unexpected result.

What’s the Problem with Xcode Projects?

Xcode is the official tool for developing apps on Apple platforms. Every time you make a change to an Xcode project, such as adding a new source file or changing a setting, the IDE generates some code into a file called project.pbxproj.

For example, here is an extract of changes that are automatically generated by Xcode when creating a new file:

The first thing you may notice is that the generated code is not easy to read or change, as it uses unique random identifiers to refer to every entry of the project. Furthermore, a single file addition creates changes in different sections of the project file, which needs to respect a specific format and particular conventions, or else it would result in an invalid project that can’t be opened.

Project Files Are for Computers, Not for Humans

Now let’s try to imagine how large this file would be in a project of the size of the SoundCloud iOS app, which is composed of thousands of source files and split across dozens of frameworks with multiple targets, hundreds of build settings, and complex linking rules. To give you a better understanding of this, in December 2019, we had more than 80,000 lines of code representing our project. That’s Jupiter!

Maintaining project files has always been a problem for iOS developers, especially as the number of contributors working on the same codebase grows. As an example of one possible issue, if multiple engineers are working in parallel and trigger changes on the same project file, there is a high chance that this will result in painful git conflicts.

Project Generation

To solve most of these problems, some engineers in the iOS community came up with the idea of automatically generating the project files. In this way, the projects can be redefined using a “user friendly” format composed of more concise and readable rules. A tool will then run to create a pbxproj file that adheres to Apple’s standards.

The main benefits of this process are:

Consistency — one rule applies to all modules
Simplicity — it’s easier to understand how your project is defined
Microprojects — you can generate a subset of your project on demand, in order to work on a feature in isolation and get lower build times
No conflicts — you’ll avoid having to solve difficult conflicts on project files
Extensibility — you can easily add new modules

Introducing Tuist

When we decided to find a new way of generating projects, we opted to use Tuist, an open source project started by one of our former employees, Pedro Piñera. Tuist generates Xcode projects based on some rules expressed on a Swift manifest file named Project.swift and on the current state of the filesystem.

Before using Tuist, creating a framework for a new μFeature¹ at SoundCloud was manual, error-prone work. Those days of work are now behind us, as it takes just one line of code and a couple of seconds to run the tuist generate command.

For example, this is the project definition of our μFeature Search:

The result of running Tuist is the creation of a non-trivial project all set up with our common dependencies, build settings, multiple targets, an example app, etc. Everything is generated by a single line of code:

With Tuist, We Finally Got Control of Our Project

By using Tuist, we didn’t just improve our productivity, but we also upped the consistency of our projects. All the modules are now generated using the same rules, expressed in Swift, using Tuist project helpers.

If we want to change a build setting or a linking rule across all our frameworks, we can simply do it in a single place and regenerate all our projects. It is a faster and more robust approach.

This is an example of a project helper function that defines the generation of the main target of our μFeatures:

There are also other popular open source tools for generating Xcode projects, such as XcodeGen. But we decided to use Tuist because of its ability to express rules in a powerful language like Swift, as opposed to the more verbose approach of using a data interchange format like YAML or JSON. In addition, our developers prefer to edit projects in Swift on Xcode to benefit from code completion and all the features the IDE provides.

Serendipity

On our way to generate the SoundCloud project, two unplanned yet fortunate discoveries occurred. The first one is that we got better compilation times for our μFeatures. This is because, when using the tuist generate command, the tool also generates a separate workspace on which developers can work in isolation.

So for example, when we want to make some changes on the Search feature, we can quickly generate a workspace that contains only the Search framework and all the modules needed to compile it:

Surprisingly, a smaller workspace resulted in faster compilation times, probably because Xcode and its compiler have an easier job understanding which source files should be indexed and recompiled and which ones don’t need to be.

The second unplanned advantage was that we cleaned up our project. While writing the rules to generate SoundCloud, we got rid of: files that were not part of any targets, duplicate resources, and dependencies that should not have been linked. Also, the generated project looks tidier: for example, all of our dependencies are now ordered both by type and alphabetically.

Conclusion

Generating the project was not an easy journey, as the process of writing the manifest files can be tedious and error-prone. Debugging problems at the project level is not always easy, nor is it easy to validate if the generated project is semantically equal to the previously existing one, even with the support of a great open source tool like xcdiff.

However, the benefits clearly overcame the costs for us. Being able to quickly change the setup of all our projects is a huge step toward further modularization of the SoundCloud app. It enables our iOS engineers to quickly create more modules while we are still able to keep the architecture under control.

Looking to the future, we are planning on using Tuist to continue our process of extracting code from the main application to feature modules, since it’s increasing the productivity of our engineers. We are proud to contribute to open source projects like Tuist, and we are very interested in the further development of the tool that has already proven to be a game changer!

A μFeature is an architectural approach to structuring iOS applications to enable scalability, optimize build and testing cycles, and ensure good practices on your team.
↩

A Happy New Employee

Fri, 07 Feb 2020 00:00:00 GMT

My first six months at SoundCloud as an iOS engineer on the Recommendations team have just finished. In that time, I’ve already contributed to around 15-20 repositories. I have made a couple of changes to our frontend and backend, ranging from a minor improvement in tooling to a newly baked feature. I feel I contribute to the team and grow every day. And I’m happy.

Is this because I’m smart and hardworking? Well, this is just one part of it. 😉 But there’s something about the culture and practices here that enable me to succeed. So I decided to take some time to articulate exactly what these things are.

Engineering Practices That Make the Onboarding Experience Better

If you are new to a company, it likely means you have a myriad of random questions every day, perhaps every hour. But I’m an introvert — maybe you can relate. I don’t want to bother people, especially for things I don’t necessarily need to know. And I don’t want to ask questions without knowing how to ask them. Would I go to someone and say “I’m curious about this. I have no idea how it works or what it is for. Could you explain this to me?” No, never.

Even so, I’ve found that something like this is possible at SoundCloud because people here are very supportive. However, what really helped me were the engineering practices in place that allowed me to explore the code base, documentation, and infrastructure on my own, following my curiosity and learning what I could until I needed someone’s help.

So, here’s a collection of random things that I found useful for quickly onboarding. Some of them may be widely known practices in the context of SRE and microservices. Some of them may be so common that no one bothers to mention them. Nevertheless, I’d like to share them while I still maintain an outsider’s view.

Standardized Technology Stack

SoundCloud is a distributed system built on top of a gazillion microservices, as explained in many previous posts. These microservices enable product teams to work autonomously, but doesn’t it mean that individual services grow so differently that I need to start over whenever I look into a new project? Well that’s not what’s happening at SoundCloud. Instead, I see common patterns appear in many places, again and again.

What do I mean by this? Well, we have shared frameworks, with which one can get many things for free. We have shared scripts that package and deploy applications. Most importantly, we have a dedicated team that continuously applies the latest practices to different services so that none of them falls behind too much and ends up confusing engineers with outdated information.

This means that once I learn one thing, I can transfer it to other projects, and the learning curve is not just linear.

Example Applications as a Showcase of Best Practices

In addition to the above, we have internal example applications that capture our best practices. These allow us to understand the essentials of how individual apps are configured, deployed, and monitored without being distracted by subtleties.

We can also quickly and easily start a new project using the example applications as templates. For the simplest application, we could start it in a day just by copying an example application and making the relevant tweaks to it.

Practical Hands-On Working

No matter how good the documentation is, we sometimes still need to mess up things in order to know what we don’t know. But how can we do that without causing havoc?

Here at SoundCloud, we have a hands-on program that everyone can try. This program allows developers to provision a new machine, deploy the aforementioned example application to it, and set up monitoring for it in the same way our production services do.

First Responder

As a new hire, you may have questions, some of which seem too basic that you are almost ashamed of asking. In fact, you might not even know where to ask them. So you finally work up the courage to post the question in a chat room that looks relevant. You keep coming back to the thread, hoping your question made sense and someone eventually replies to you, but no one responds in the end. Everyone knows this feeling. Understandably, people are often too busy to spend time clarifying what you want to ask, or maybe they expect someone else who knows the subject better will answer it.

At SoundCloud, many teams have a “first responder.” This is a person designated on a rotational basis to answer all the questions asked while they are in charge. Having someone like this greatly reduces the mental cost of asking questions because we know who we can bother, and there’s at least one person responsible for answering any questions.

Content Management Tool Open to Everyone

If your company operates with a fairly big number of users, you probably have an internal admin tool where you can manage users and their contents. What I experienced in the past was that it was open only to operational teams.

At SoundCloud, everyone can access this tool, though its capabilities are limited by the permissions set for the individual user. When selecting an account, one can quickly tell which experimental features it is getting, what its contents are, when it was created, which program it subscribes to, and so on. This information often explains why a problem happens only to specific users, and it helps facilitate communication with other teams while debugging.

Infrastructure as Code and a Code Search Tool

It’s common for new people to face permission issues, and if you need to talk to different people whenever you have these problems in order to gain access to something and solve the problem, it will likely discourage you from exploring new things.

Here at SoundCloud, in most cases, the way systems are provisioned can be found as code in a single place. Along with an internal code search tool, which enables fast cross-repository search, we can easily identify what we’re looking for by searching with an error message or our account name and fixing it ourselves by updating the relevant configuration.

Monitoring with User-Friendly Metadata

When you start in a new company, it’s a great opportunity to find issues that many people have gotten used to and ignore. However, if you can’t narrow them down, your report probably won’t help. Imagine that you are a mobile engineer and find a 500 error that happens intermittently. If you report it ambiguously like “I got a 500 error when I opened screen X and it never happens again,” it probably just adds another ticket that no one ever looks into.

We have been making a rigorous effort to make debugging information useful and user friendly. For example, you can find an improvement we made to the tracing metadata in the Using Kubernetes Pod Metadata to Improve Zipkin Traces blog post. In a scenario like this, I can check the trace and tell exactly where things go wrong and who is in charge of the issue. With this information, I can rewrite the occasional 500 error report with more specific information, which can be like “An upstream service X timeouts when requesting information of user Y.”

SAT (Self-Allocated Time)

Allowing people to work on side projects in working hours has been a popular concept since Google shared more about its 20-percent time policy, but it easily becomes a window dressing, and people don’t do it for many reasons. For example, they can’t find a suitable project or people to work with, or they can’t find time for it because project deadlines are always tight. And once the majority of people stop doing it, it’s hard for other people to justify continuing.

We have this same concept here at SoundCloud, but we refer to it as SAT, or self-allocated time. And I’m happy to report that I see many people actually taking time on Fridays for this. I myself use this time to work on the things I know the least about. It also gives me a chance to collaborate with people whom I don’t usually work with. It often pays off because it adds a new tool to one’s toolbox, which, sooner or later, becomes useful in regular projects too.

Documentation

I put this at the end of the list because this looks fundamental, but it works as collective knowledge only if the subject can be accessed and tested by everyone, and if it’s supported by the things I mentioned above.

What I like about how we update the documentation at SoundCloud is that it’s not authoritative. Although people ask other’s reviews for things they are not too sure about or when they think that other people should be aware of a change, the documentation can always be updated straight away. This removes the mental cost of updating documents and encourages more people to do so, which, in the end, also allows for minor errors which might be introduced by this non-authoritative model to be quickly fixed.

Conclusion

As engineers, we’re all familiar with the idea of being driven by our curiosity so profoundly that we forget to sleep or feel like we can work forever (which of course is untrue). And while smarter people often perform well, there are many high performers who do so because they’re the ones most excited about learning and putting that learning into practice.

What I’d like to emphasize with this post is that you and your company can leverage those people by having good engineering practices in place and allowing people to explore things freely without barriers to learning.

Or, more simply put, let good people do their jobs.

Speeding Up Builds with Dagger Reflect

Thu, 30 Jan 2020 00:00:00 GMT

A large portion of an Android app’s build time can consist of Dagger¹ annotation processing,² and most developers agree that productivity is important, so we decided to experiment to see if we could save time when compiling the SoundCloud Android app. This blog post covers how we used Dagger Reflect to save developer time with minimal changes to our codebase. Before we dive deep into how we saved Dagger annotation processing time, let’s make sure we’re on the same page in understanding what dependency injection is.

Basics of Dependency Injection

Dependency injection is a great way to write maintainable, testable code by using the dependency inversion principle. For more information on the basics of dependency injection, take a look at this introduction tutorial.

History of Dependency Injection

Dependency injection frameworks in the Android world started by bringing practices from the Java server side to Android. At the time, these frameworks, e.g. Guice, were reflection based, meaning they inspected the program’s code during runtime. This approach was especially slow on Android. Then Dagger came along and alleviated this problem by generating code at compile time instead of using reflection at runtime. However, the tradeoff to the faster app startup time was a longer time spent compiling, which was especially noticeable as apps grew in size.

In the above example from our app, a 59-second build spent 42 seconds processing annotations. More specifically, the 42-second processing time is spent checking the code, analyzing the object graph, and generating the code. To put this in perspective, in the SoundCloud Android app, Dagger generates a very large component class with more than 10,000 lines of code.

So this is how we as developers using Dagger started to suffer. What this meant for us here at SoundCloud is that we needed to find a solution that saved time. This sparked looking into Dagger Reflect. We did this because it offered the possibility of the best app startup time for users and the best compile time for developers, with little changes to the production code. Another option would have been to migrate our entire codebase to a runtime- or reflection-based dependency injection framework. However, we decided against it because the cost of migration and the slower startup times were not worth it.

How Dagger Reflect Works

Dagger Reflect is an open source project by Jake Wharton that mimics the API of Dagger.³ However, instead of generating code at compile time, it parses injectable objects and the component dependencies at runtime. This skips generating and regenerating the same code for every build. It instead uses a dynamic proxy class, which implements an interface specified at runtime. Dagger Reflect proxies an application’s Dagger Component to the Dagger Reflect runtime.

How to Use Dagger Reflect in Your Project

There are a few ways to use Dagger Reflect in your project. One approach is full reflection, and the other approach is partial reflection, as outlined in the Dagger Reflect README. You can also swap the dependencies manually for every module. The simplest approach is to use the Delect Gradle Plugin, something that was created here at SoundCloud to make integrating Dagger Reflect simple and painless. For this blog post, we are going to cover the partial reflection approach using Delect.

The partial reflection approach requires the least amount of changes to your existing codebase, and almost no downside — it takes only 60 ms of code generation to glue your code to Dagger Reflect. Here’s how to start using Dagger Reflect in your project.

Step 1

Make sure to use the Dagger Component Factory or Component Module annotations:

@Component
  public interface ApplicationComponent {
    @Component.Factory
    interface Builder { }
  }
}

Step 2

Ensure that all Qualifier annotations have AnnotationRetention.RUNTIME. Additionally, all MapKey annotations must have runtime retention:

@Qualifier
@Retention(AnnotationRetention.RUNTIME)
annotation class LibraryStorage

Step 3

Apply the Delect plugin to your root build script:

buildscript {
  classpath 'com.soundcloud.delect:delect-plugin:<latest_version>'
}
apply plugin: 'com.soundcloud.delect'

This will automatically swap Dagger for Dagger Reflect when the Gradle property dagger.reflect is set to true.

Delect — the Dagger Reflect Gradle Plugin

Delect is our open source Gradle plugin that makes it much easier to switch out all the regular Dagger dependencies for the Dagger Reflect versions in your project. Here is an example of the following dependency swap you would need to perform in each Gradle module:

implementation “com.google.dagger:dagger:<latest-version>”
implementation “com.google.dagger:dagger-android-support:<latest-version>”
lintChecks “com.jakewharton.dagger:dagger-reflect-lint:<latest-version>”
if (useDaggerReflect()) {
  implementation “com.jakewharton.dagger:dagger-reflect:<latest-version>”
  kapt “com.jakewharton.dagger:dagger-reflect-compiler:<latest-version>”
} else {
  kapt “com.google.dagger:dagger-compiler:<latest-version>”
  kapt “com.google.dagger:dagger-android-processor:<latest-version>”
}

The plugin works by using the Gradle Dependency Substitution API. When it detects that there is a dependency on the Dagger runtime, it adds the Dagger Reflect runtime. When it detects the Dagger compiler or the Dagger Android compiler, it swaps them out for the Dagger Reflect compiler. There is no need for a separate runtime or compiler component in order to use the Dagger Android processor.

To give you an idea of how it works, here’s a small snippet of code from the plugin where the Dagger annotation processor is substituted for the Dagger Reflect annotation processor:

substitute(
  module("$com.google.dagger:dagger-compiler")
).apply {
  with(module("$com.jakewharton.dagger:dagger-reflect-compiler:${extension.daggerReflectVersion}"))
}

In some initial trials, we noticed that developers’ workflows switched from the IDE to the command line while working on the same task. Only enabling Dagger Reflect for IDE builds triggered a full rebuild when switching and vice versa. Because of this painful rebuild, we chose not to enable Dagger Reflect in the Gradle Plugin only in IDE builds. Instead, Delect is enabled or disabled via a user controlled Gradle property, (dagger.reflect).

A quick way to enable Dagger Reflect when the plugin is applied is the following:

echo "dagger.reflect=true" > ~/.gradle/gradle.properties

We also seed our Gradle build cache with Dagger Reflect enabled; otherwise, building with Dagger Reflect would always be the slower option.

Performance Benchmark

Before we dig into the numbers, we want to note that we’ve spent a lot of time optimizing our builds by using incremental annotation processors and enabling caching, and as a result, have solved as many build cache misses as we possibly could.

Though on average, local developer builds are under 60 seconds, some builds can take as long as 3-5 minutes here at SoundCloud. As an example, the following build scan shows a build that took 5 minutes and 44 seconds.

Out of all the annotation processors in use here, Dagger takes the most amount of time by far. It took a total of 3m 9s in the above example.⁴

When using Dagger Reflect, the compiler never spends more than 80 ms processing annotations — a huge decrease in build times.

Our Dagger Reflect benchmark in our Android project led to a decrease in build times from 62 seconds to 31 seconds for an ABI change to a module. Dagger Reflect makes builds take half as long!

But what about overall build speeds for all our developers? In a four-week period, our average build time for all local builds without Dagger Reflect was 81.13 seconds. The build time with Dagger Reflect enabled was 55.33 seconds. This is, on average, a 26 second and 32% decrease in build speeds!

No Dagger Reflect	With Dagger Reflect

However, we see the biggest wins on the longest builds because those are the ones that spend the most amount of time in annotation processing. For simply applying a Gradle plugin in the root of a project, this is probably the easiest build speed win!

To sum it up, we saw that Dagger was taking a large portion of our build times. We saved up to 50 percent of our local build times with little maintenance cost by swapping regular Dagger for Dagger Reflect. Try it out in your project!

Additional Reading

For more information about Dagger Reflect, see here.
For more information about Delect, check out the GitHub repository.
Click for recorded version of this talk at Droidcon SF as well as corresponding slides.

Dagger is a popular dependency injection framework in the Android and JVM world that leverages annotation processing.
↩
Annotation processing is a way of hooking in to the compilation to analyze source code and generate code.
↩
Dagger Reflect is designed to be used for development builds only.
↩
This time is cumulative, not wall clock time. Multiple threads or Gradle tasks could be compiling using Dagger in parallel and the time spent in parallel might be double counted.
↩

The Power of Types for Errors

Mon, 20 Jan 2020 00:00:00 GMT

At KotlinConf 2019, I talked about the power of types. In essence, I discussed limiting the number of primitives we use in our code in favor of using custom types instead. In this way, we as developers will not only reduce the possibility of bugs by using the compiler, but we will also achieve more readable and self-documented types.

Today I ran into a situation which reminded me of the things I said in this talk.

Where It All Started

I was working on some tracking code that’s part of the signup flow of our Android app. The goal was to improve the insights from the error states that users might run into. These could be local UX flaws, but they could also be issues that arise when trying to log in via third parties.

My initial code looked like this:

fun onError(type: ErrorType, details: String? = null) {
  // ...
}

The ErrorType is an enum that specifies the exact error type:

enum class ErrorType {
   IO,
   PASSWORD_INVALID,
   RECAPTCHA_FAILED
// ...
}

The additional parameter details are for extra information some errors might need. But let’s take a closer look at the method signature we had:

fun onError(type: ErrorType, details: String? = null)

I’d say these two parameters belong together, as the details describe the error. This means they have a very tight coupling, but they still don’t share any relationship! Rather, they are just two independent parameters.

Let’s Express This Relationship

The first idea for improving this situation is to wrap both parameters into one type:

data class Error(val type: ErrorType, val details: String?)
Our tracking method now looks much nicer:
fun onError(error: Error) {
  // ...
}

Once we do this, there will only be one parameter in our method, making it clear to the reader what to pass:

fun onError(error: Error) {
  //...
}

And they can’t pass in something wrong, as was possible when we had two independent parameters.

This only confirms the quote I shared in my talk from the late Robin Milner, a British computer scientist who once said:

Well-typed expressions cannot “go wrong.”

We Can Do Better

But we should not stop here! The method is now saved, but the two parameters have simply been moved to the Error class, so there is still a hidden relationship!

So, we should express this. We should not let the developer guess when to pass what and when or what not to!

An enum type could not hold these individual details, as enum values are simply constants, which are by definition immutable. However, with sealed classes, we can model this more precisely:

sealed class Error {
  object InvalidPassword : Error()
  object RecaptchaFailed : Error()
  class IOError(val details: String): Error()
// ...
}

The Next Step

Another thing that enums do not support is a hierarchy. Enum values are flat on one level of abstraction.

To compensate, we normally express this via the name:

enum class ErrorType {
   IO,
   EMAIL_DENIED,
   EMAIL_EXISTING,
   EMAIL_INVALID,
   PASSWORD_INVALID,
   RECAPTCHA_FAILED
}

But as I mentioned in my talk:

Don’t put in a name what can be in a type.

Now, once we get rid of the enum, it’s easy to introduce a hierarchy:

sealed class Error {
  sealed class SignInError {
     sealed class EmailError : SignInError() {
        object Denied : EmailError()
        object Invalid : EmailError()
        object Existing : EmailError()
     }
     class IOError(val message: String): SignInError()
  }
//...
}

As you might have noticed, we could even restrict the email-related errors to sign-in use cases. This way, we could handle all errors related to one use case at one point simply by using types.

And Then I Saw…

There was another area of the code that wasn’t using the full power of types. When trying to log in via social networks, the code normally first handles third-party library calls and then proceeds with the login flow to your own system with those results. On the other hand, when you log in directly via an email, you won’t have this first step.

The code had something like this:

// An error on Facebook/Google appeared.
fun onAuthenticationError(method: Method, error: Error)

Here, Method is another enum:

enum class Method {
   FACEBOOK, GOOGLE, EMAIL
}

But as the code comment says, onAuthenticationError() is meant only for Google and Facebook. A developer should not even be able to call it with Method.EMAIL. But nothing prevents us from doing this. We could add an assertion and crash on invalid input, but both runtime crashes and hidden code comments are not ideal. The ideal scenario would be that the compiler could check this upfront.

Thanks to our new error hierarchy, this can be solved very easily with types:

fun onAuthenticationError(error: Error.SocialError)

Here, SocialError is just another specialty in our sealed Error:

sealed class Error {
 sealed class SignInError {
   sealed class SocialError(val message: String) : SignInError() {
       class GoogleError(message: String) : SocialError(message)
       class FacebookError(message: String) : SocialError(message)
   }
  // ...
 }
}

Conclusion

In summary, types are very powerful. With the changes mentioned above, our code became easier to read and more expressive, and a new developer will know how to use the tracking methods simply by the types they need as arguments.

Here are a few key takeaways from this blog post to keep in mind:

If you see parameters that belong together but are treated separately, check if there is a hidden type.
If you see classes with fields that are only used in certain cases, maybe there is a hidden type hierarchy.
If you see methods that only allow a certain combination, secure them with a type.

And, of course, if you’re interested in learning more about this, you’ll find more examples in my slides and by watching the recording of the talk.

Implementing Dark Mode Using the Observer Pattern

Fri, 08 Nov 2019 00:00:00 GMT

Last week’s update to the SoundCloud iOS app includes support for Dark Mode. This took several months of work and collaboration between design and engineering teams across the company, so we wanted to share our approach to implementing Dark Mode and some of the obstacles we encountered along the way.

Apple did an excellent job implementing Dark Mode in iOS 13, but because it’s important for us to support compatibility with previous versions of iOS, we decided to build our own implementation of themes. To achieve this, we built a fast, type-safe, and powerful observer pattern in Swift.

Observer Pattern

The observer pattern is a one-to-many relationship between objects, meaning that when one object changes its state, all of its dependent objects are notified and updated with the new state. Looking at this design in another way, the observer pattern can be seen as a delegation pattern with one-to-many delegates. We can think of it as the foundation of reactive programming.

The key objects in this pattern are the subject and the observers. A subject may have any number of dependent observers, and all observers are notified whenever the subject undergoes a change in state. It’s important to note that the observers do not access the state directly, because this can lead to dangerous race conditions.

How? It could be that while an observer is accessing the subject’s state, the state is changed from another thread. But the possible combinations of things that can go wrong here are exponential, so it’s a requirement that the subject informs the observers about state changes and never the other way around.

State to Be Notified

But first we need something to notify the observers of, which is the state that we want to spread across X number of observers. In our case: We want to notify the observers when a color theme changes across the app.

Let’s start by defining an abstract Theme as our state:

protocol Theme {
    var main: UIColor { get }
}

Now we can define, for example, a concrete DarkTheme:

final class DarkTheme: Theme {
    var main: UIColor = .black
}

Observers

In our case, an observer is going to be a View or a ViewController that is interested in using colors. Following the “program to an interface” philosophy, let’s first define an observer as a protocol:

protocol ThemeObserver {
    func apply(theme: Theme)
}

Now we can implement a concrete observer — for example, in a LoginViewController:

final class LoginViewController: UIViewController, ThemeObserver {
    func apply(theme: Theme) {
        backgroundColor = theme.main
    }
}

That’s great! Now our LoginView is an observer that can be notified about Theme updates.

Subject

Here is when things get really interesting. We need to define an object that will allow observers to subscribe to state changes by providing an interface for attaching and detaching observers. In addition, the subject will provide an interface to reach observer objects when its internal state changes. In response to a state change, the subject will broadcast the new state to all subscribed observers.

Abstract Subject

Following the interface segregation principle, let’s start defining the interface for a subject that allows observers to subscribe to and detach from it. This interface is going to be visible from any observer that wants to subscribe to notifications, however, the interface will make it impossible to access the state directly, thereby preventing race conditions or undesirable state updates from the clients:

protocol Subject {
    func subscribe(observer: Themable)
    func detach(observer: Themable)
}

Concrete Subject

Internal Data Structure

We want to be able to keep a reference to each observer, but without taking care of the memory management of these objects. In essence, we need to find a data structure that holds objects weakly, in order to prevent memory leaks.

First and foremost, we want to keep one (and only one) reference to each object to prevent a double subscription of the same observer. The main reason is that multiple subscriptions of the same observer can result in multiple notifications of new states.

In addition, we need to provide a fast and cheap way to subscribe new observers.

If you’re thinking this sounds like a Set data structure, you’re correct! We needed to build a Set that can only hold objects weakly. So after some research, we found a Foundation class called NSHashTable<ObjectType> that can reference objects weakly and add and access objects in constant time, O(1):

private var observers = NSHashTable<T>.weakObjects()

To make the data structure more useful, we can extend it and create a reusable generic WeakSet class:

import Foundation

final class WeakSet<T: AnyObject>: Sequence, ExpressibleByArrayLiteral {

    private var objects = NSHashTable<T>.weakObjects()

    public init(_ objects: [T]) {
        for object in objects {
            insert(object)
        }
    }

    public required convenience init(arrayLiteral elements: T...) {
        self.init(elements)
    }

    public var allObjects: [T] {
        return objects.allObjects
    }

    public var count: Int {
        return objects.count // Note that this count may be an overestimate, as entries are not necessarily purged from the `NSHashTable` immediately.
    }

    public func contains(_ object: T) -> Bool {
        return objects.contains(object)
    }

    public func insert(_ object: T) {
        objects.add(object)
    }

    public func remove(_ object: T) {
        objects.remove(object)
    }

    public func makeIterator() -> AnyIterator<T> {
        let iterator = objects.objectEnumerator()
        return AnyIterator {
            iterator.nextObject() as? T
        }
    }
}

Unit Test

And of course, let’s unit test it!

Concrete Subject

Now that we have a solid data structure to work with, we can define a concrete subject:

struct ThemeSubject: Subject  {

    private var observers: WeakSet<ThemeObserver> = []

    func subscribe(observer: ThemeObserver) {
        observers.insert(observer)
    }

    func detach(observer: ThemeObserver) {
        observers.remove(observer)
    }
}

Now Let’s Bring Everything Together

Sweet! So far, we have created a ThemeObserver and our ThemeSubject. Now let’s provide a default way to subscribe our observers to the subject. We can do this as a protocol extension of ThemeObserver. We will define a subscribe method that will access a shared instance of ThemeSubject to subscribe our view to the subject:

extension ThemeObserver {
    func subscribeForThemeUpdates() {
        ThemeSubject.shared.subscribe(observer: self)
    }
}

Now we can add a state to our ThemeSubject. In our example, this will be Theme:

struct ThemeSubject: Subject  {

    private var observers: WeakSet<ThemeObserver> = []

    func subscribe(observer: ThemeObserver) {
        observers.insert(observer)
    }

    func detach(observer: ThemeObserver) {
        observers.remove(observer)
    }

    var state: Theme {
        didSet {
            notify()
        }
    }

    init(state: Theme) {
        self.state = state
    }

    private func notify() {
        observers.compactMap { observer in
            observer.apply(theme: state)
        }
    }
}

That’s awesome! Now every time we set a new state to ThemeSubject, all our observers’ views will receive the notification with the new theme. The code below shows how we can use this:

final class LoginViewController: UIViewController, ThemeObserver {

    override func viewDidLoad() {
        super.viewDidLoad()
        subscribeForThemeUpdates()
    }

    func apply(theme: Theme) {
        backgroundColor = theme.main
    }
}

Remember that adding new views as theme observers will be in constant time, and changing the theme across our app will be linear. Quite powerful, right?

Obstacles We Encountered along the Way

While implementing themes on the iOS app, the major problem we encountered was the inconsistency of UIColor use. This is because some parts of the app set colors on Storyboards or XIB files, while in other parts, the colors were used as global variables. This led to us putting a large amount of effort into refactoring the UI layers in order to conform to the ThemeObservable protocol.

Another problem we discovered was the inconsistency and lack of UI components. For example, we found eight different implementations of the loading spinner. We thought this was a good opportunity to unify several views into common UI components.

When We Should Apply the Observer Pattern

The observer pattern is a powerful tool that we should all have in our toolset. So I’d like to share some tips on how to identify opportunities to apply it:

When an abstraction has two aspects, one depends on the other. Encapsulating these aspects in separate objects lets you vary and reuse them independently.
When a change in one object requires changing others and we don’t know how many objects need to be changed.
When an object should be able to notify other objects without making assumptions about what these objects are. Or, put another way, when you don’t want these objects tightly coupled.
When there are a lot of layers between the subject and observer so that the delegate pattern is not practical.

Benefits

There are a handful of benefits that come from making use of the observer pattern. They include:

Abstract coupling between the subject and the observers. All a subject knows is that it has a list of observers, each conforming to a simple interface of the abstract Observer protocol. The subject doesn’t know the concrete class of any observer. As a result, the coupling between a subject and observers is abstract and minimal. Consequently, the observers can belong to different layers of abstraction in a system.
Support for broadcast communication. The notification is automatically broadcast to all interested objects that have subscribed to it.
Benefits from reactive programming, without a huge third-party dependency.
Compared to NotificationCenter, the type-safe approach that we propose can prevent runtime crashes.

Conclusion

This post details how we created a dark mode for the SoundCloud iOS app using the observer pattern. But the pattern is not just limited to this kind of scenario. In fact, here are other real-world examples of where you can apply the observer pattern:

A music player that wants to notify different architectural layers of the app that playback has started.
When views need to observe and react to changes on a database.
When sending notification about the app lifecycle, e.g. when the app is in the background, foreground, etc.

Do you have more ideas of other problems that can be solved with this pattern?

for question in questions {
    question.ask()!
}

Solving Remote Build Cache Misses by Annoying Your Colleagues

Tue, 01 Oct 2019 00:00:00 GMT

This is part two in a series about solving Gradle remote build cache misses. Solving build cache misses is important to both avoid work that has already been compiled and improve build speeds. For more information, check out part one here.

To understand how these cache misses happen, it is important to understand how we seed our build cache. Gradle recommends that your CI system populates the build cache from clean builds and that developers only load from it. At SoundCloud, we also pull from the build cache on CI in order to make our CI builds faster. This is a diagram of what our remote build cache setup looks like.

For the same task, differences in the list of inputs when the task is running on the CI system vs. on the local machine causes these build cache misses.

Blocking Slow Builds

So now on to the explanation of the title. Part of a build engineer’s role is to speed up builds. Improving build performance and avoiding work with caching is one way to achieve this, but another tool in the build engineer’s belt is that of disallowing slow builds. Sometimes, there are two paths that produce the same outcome. Let’s call them the slow path and the fast path. If we can always take the fast path, we can improve our build speed.

Often it isn’t obvious which path is the “slow path” and which path is the “fast path.” When confronted with two paths in real life, we usually ask Google Maps which path has the most traffic so that we can avoid it. With Android Builds, there may be limited guidance from Google. So, it is the build engineer’s role to investigate and disallow the slow paths in order to make engineers happy and productive.

Instant Run

One of the slow paths we discovered at SoundCloud was Instant Run. At first, we didn’t realize that it was causing problems; we only saw that some of our build scans showed that some of our developers were building with the Gradle build cache disabled. Since we have org.gradle.caching=true checked in to our repository in the root gradle.properties, we were puzzled as to how it could happen. We paired with developers to troubleshoot these slow builds, and we quickly realized that they were being built with Instant Run.

Instant Run was a tool designed to allow developers to deploy changes to devices faster, but it didn’t always work in large projects. One big limitation is that it disabled the build cache. This sometimes led to painfully slow builds, as everything had to be recompiled. Additionally, this meant that when building with Instant Run, both the local and remote build cache were disabled. We did a performance analysis of Instant Run in different scenarios, and it showed that in our codebase, Instant Run performed either equally as fast as a build without it, or slower — sometimes significantly slower.

We already had a page in our internal wiki recommending our team disable Instant Run for faster builds, but nobody reads documentation! So how could we disable Instant Run for our team and save everyone from slow builds?

Here is a Groovy snippet for Gradle which does exactly that:

def instantRun = project.getProperties().get("android.optional.compilation")?.contains("INSTANT_DEV")
if (instantRun) {
 throw new IllegalStateException("Disable Instant Run: Android Studio -> Preferences -> Build, Execution, Deployment -> Instant Run")
}

The snippet checks the project for a specific property, which Android Studio injects into Gradle whenever Instant Run is enabled. When a build is run with Instant Run, the build fails and a helpful error message is displayed guiding the developer to disable Instant Run. This might be slightly annoying to colleagues, but it is all for the team’s benefit: A 10-second settings change is rewarded many times over with faster builds.

Learn more about how injected properties in Android Studio work here.

Instant Run was removed in Android Gradle plugin 3.5.0.

Empty Directories

Another slow path we had at SoundCloud was that of empty directories. Empty directories in Gradle create a slow path compilation, which we should disallow. But how and why does this happen? Well, let’s suppose you have the following directory structure on your local machine:

src/main/java/com/soundcloud/Player.java
src/main/java/com/soundcloud/Artwork.java
src/main/java/com/soundcloud/audio/

And let’s assume you have the following on CI:

src/main/java/com/soundcloud/Player.java
src/main/java/com/soundcloud/Artwork.java

If audio is an empty folder, are the inputs to the task the same or not? The Java compiler itself ignores empty source directories. Gradle, however, does not. Empty directories are treated as different input properties. Well, that is, until this issue is resolved.

How can we push out a change to delete all empty directories on all our local project workspaces? If we take a cue from the previous section, we can fail the build if we find any empty source directories. This ensures that the directory structure, as Gradle sees it, is the same locally as it is on CI:

allprojects {
 tasks.withType(SourceTask).configureEach { t ->
  t.doFirst {
    t.source.visit { FileVisitDetails d ->
      if (d.file.directory && d.file.listFiles().size() == 0) {
        throw new IllegalStateException("Found an empty source directory. Remove it: \nrmdir " + d.file.absolutePath)
      }
    }
  }
 }
}

The above snippet, which should be placed in the root build.gradle, looks at all SourceTasks in the project. It traverses the source directory structure to find any empty directories. When it finds a source directory, it prints a helpful error message to the console telling you how to delete that directory. It performs this check before any source task is run.

You can also remove all empty directories by running the following command:

find . -empty -type d -delete

One thing to note is that if the root directory is empty, this does not fail the build.

So, before adding this to our codebase, we needed to be sure this didn’t impact our build times. In this case, every single module in our project took less than 5 milliseconds to run this check, except one particularly large legacy module, which took 89 milliseconds to run.

Is it worth it? We decided it was. Almost all of our developers were experiencing build cache misses while having a clean local Git state compared to master. After pushing the change, many developers started getting a little frustrated when they would have several build failures in a row because of nested empty directories. It is quite common to have these lying around after refactoring code or changing branches, since Git ignores empty directories. However, after the initial change, these failures became quite rare and we eventually increased our remote build cache hits.

CompileDebugAIDL

Next, we saw a build cache miss for a generated class called InAppBillingService.java. This class is generated by the compileDebugAidl task that is part of the Android Gradle plugin. It looks for any source files ending in .aidl, and for each one, it generates a Java file before the JavaCompile task runs in your AIDL source directory. It then adds these generated Java files to the source directory of the JavaCompile task. We opened the generated Java source file to find this comment:

/**
 * This file is auto-generated. DO NOT MODIFY.
 * Original file /Users/no/workspace/soundcloud/android/libs/src/main/java/com/soundcloud/android/InAppBillingService.aidl
 */

This file is always going to generate a remote cache miss, because the absolute path is different on every machine. We used the power of Gradle to solve this one as well:

tasks.named("compileDebugAidl").configure {
  doLast {
    outputs.files.forEach { directory ->
      directory.traverse(type: FILES) { file ->
        file.setText((file as String[]).findAll {
          !it.contains('Original file:')
        }.join(‚\n'), 'utf-8')
      }
    }
  }
}

The above code snippet finds the compileDebugAidl task and adds a doLast action, which iterates through each of its lines and rewrites it as is, skipping the line that contains “Original file.” The file is now the same on every machine. That solved the build cache miss for us, but we want to make the world a better place, so we filed a bug against Google.

While examining this task, we also noticed that the CompileAidl task would take 1.3 seconds to perform an up-to-date check — something which should typically be done in less than 10 milliseconds. The reason it took longer is because the check filters the entire source tree. A Gradle engineer opened an issue about this for us, but in the meantime, as a workaround, we moved this AIDL file into its own module, reducing the run time to .003 seconds.

The Result

There are, of course, a couple more build cache misses we fixed here and there that aren’t mentioned in this article. One fun thing to note: Make sure your annotation processors generate the same code locally as on CI. And when annotation processors generate lists, make sure they are ordered or sorted, as code may be processed in different orders on CI.

The end result of all this work was quite noticeable when looking at build times because speed gains from modularization are multiplied by improving cache hits. In the week before this work started, our build times were 49.55 seconds on average.

Even accounting for four months of increased codebase size, after the majority of this work was done, our average build speed dropped to 36.84 seconds on average. You can also note the increased remote and local build cache usage.

From 49.55 to 36.84 seconds per build on average is a huge change in the day-to-day lives of developers, as our team averages around 40 builds per developer per day. It doesn’t just mean that developers are faster at writing code. Having faster build speeds means that developers have more time to explore the realm of possibilities when building new features. More explored possibilities leads to more maintainable code, increased app stability, and fewer bugs.

If you’d like to learn more, check out “Remote Build Cache Misses” from BerlinDroid Gradle Night. You can watch the talk or browse the slides.

Gradle Remote Build Cache Misses

Fri, 30 Aug 2019 00:00:00 GMT

Until recently, one of the top technical risks facing SoundCloud’s Android team was increasing build times. Our engineering leadership was well aware of the problem, and it was highlighted in our company’s quarterly goals and objectives as modularization. Faster build times means more productive developers. More productive developers are happier and can iterate on products more quickly.

Modularization is key to decreasing build times, but avoiding work is another important part of the puzzle, and build caching is one way to avoid that work. Gradle, our tool for building Android, has a local file system cache that reuses outputs of previously performed tasks. We have been using the Gradle remote build cache in order to save our developers’ time. It helps us avoid redoing work that other teammates have already done or switching to old branches. However, to get the full benefits of caching, you have to go beyond simply setting it up.

This is the story about how we solved some cache misses with our remote build cache and, in turn, saved our developers’ time. Maybe some of these tips can help you and your team save some time as well. This is part one of two. More cache misses and results will come in part two.

Tasks and Caching

A task is an amount of work with inputs and outputs, and the remote build cache works by storing the outputs of Gradle tasks. If all the inputs are the same, we can skip the work and just reuse the outputs.

For example, in the JavaCompile task, the inputs are Java files such as Audio.java and Waveform.java, and the outputs of the compile task are the compiled class files such as Audio.class and Waveform.class. If we haven’t added any new Java files or changed any of the contents of the Java files, we can reuse the output of the previous JavaCompile task.

To determine that none of the inputs have changed, Gradle computes a build cache key, which uniquely identifies a task. It is computed based on the list of inputs and outputs for that task. Gradle then packs and stores the outputs of cacheable tasks in the Gradle build cache. A cache is simply a store of the build cache keys and the outputs. The cache can be local (file system) or remote (http).

Comparing Builds

A cache miss occurs when Gradle can’t find a corresponding entry for a task that should have been cacheable. To identify build cache misses, we use Gradle Enterprise build scans. Gradle Enterprise identifies cache misses by recording the inputs to a Gradle task, which can then be used to compare two builds. Here is an example of the same exact build on the same computer but with a whitespace added to a specific file:

From this, we can gather that whitespace in Java source files is treated as part of the input’s cache key. Gradle must re-execute this task if any of the whitespace in any of the inputs changes, which is unnecessary.

BuildConfig Misses

We can also use Gradle Enterprise to compare the same task running locally vs. in our build seed cache. In order for the remote cache to work, the inputs to the tasks between the local machine and the remote machine must match.

The BuildConfig is a file in Android builds used to keep environment and configuration constants. Here is a sample BuildConfig class, and as you can see, all the values are constants:

public final class BuildConfig {
    public static final boolean DEBUG = Boolean.parseBoolean("true");
    public static final String APPLICATION_ID = "com.soundcloud.android";
    public static final String BUILD_TYPE = "debug";
    public static final int VERSION_CODE = -1;
    public static final int TEST_RETRY_COUNT = 0;
}

However, the constants can be set by functions, which may be different depending on the environment:

afterEvaluate {
  android.libraryVariants.all { variant ->
    variant.resValue "bool", "analytics_enabled", "true"
    variant.resValue "bool", "verbose_logging", "false"
    variant.buildConfigField "int", "TEST_RETRY_COUNT", "${getRetryCount()}"
  }
}

In our case, the TEST_RETRY_COUNT class was being set by a function, getRetryCount(). This function was very simple, but it led to different values locally vs. on the CI server:

private static int getRetryCount() {
  if (isCI) {
    return 1
  } else {
    return 0
  }
}

Since the value was always 0 locally and 1 on the CI server, we could not reuse the outputs of this task with CI for local builds. Unfortunately, this was in one of the largest modules in our codebase, which meant a large portion of our code would have had to be recompiled, even if it didn’t need to be.

Our solution was to remove this check from our build configuration and instead check it at runtime. For this case, we could determine at runtime if we were running on the CI server or not:

class ActivityTest<T : Activity> constructor(activityClass: Class<T>) {

  val isRunningOnTestLab = Settings.System.getString(contentResolver, "firebase.test.lab") == "true"

  @JvmField
  val retryRule = RetryRule(if (isRunningOnTestLab) 1 else 0)
}

Kotlin Compiler Flags

We also noticed that all of our Kotlin compile tasks were not pulling anything from the cache. Since we use Kotlin in nearly every module, this meant we were pulling almost nothing from the Gradle cache.

We looked at the source of KotlinCompile to understand what incremental (shown above) was and what it did. incremental is true by default, but it took us a while to realize that we had disabled incremental compilation for all our CI tasks.

We turned incremental compilation back on for our CI nodes that seed our build cache and immediately saw a big boost in cache hits. More cache hits meant less time spent building something that was already built.

Conclusion

In the end, we learned that simply turning on a remote build cache is not very effective in improving build speeds. In fact, blindly turning it on might not get any cache hits at all. It is important to keep in mind how tasks work, how they behave on different systems, and how to use the right tools to analyze differences in task inputs across different systems. That’s how we solved some cache misses with our remote build cache and, in turn, saved our developers’ time. Maybe some of these tips can help you and your team save some time as well.

In part two, we’ll dive into more complicated cache misses and talk more about the impact of solving them.

In the meantime, if you’d like to learn more, check out “Remote Build Cache Misses” from BerlinDroid Gradle Night. You can watch the talk or browse the slides.

SoundCloud Is Playing the Oboe

Fri, 21 Jun 2019 00:00:00 GMT

Media and playback are at the core of SoundCloud’s experience. For that reason, we have established and grown an engineering team that is specialized in providing the best possible streaming experience to our users across multiple platforms.

To do this, we combine the industry’s best-fitting solutions with our own custom technologies, libraries, and tools. In this article, let’s dive into how we improved latency in our Android application by leveraging a new engine for our player’s audio sink.

Maestro powers our web applications, but for the ones that are compiled with native code, we created Flipper, a lightweight cross-platform audio streaming engine written in C++. Flipper drives tens of millions of plays per day in our Android and iOS mobile apps, and it is designed with SoundCloud’s use cases in mind, which provides the flexibility we need for innovation and fast delivery.

Flipper

As is the case with many other media players, Flipper is designed as a pipeline of components, each of which is responsible for one step of the streaming technology. Having multiple discrete steps — or stages — allows for parallel processing of a continuously flowing stream of data. Think of it like a car manufacturing plant: While one group of workers is focused on constructing the engine of the car, someone else is assembling the tires.

For the scope of this article, let’s focus on one single component: the sink. This is the last stage in the pipeline, and the sink’s responsibility is to offload the decoded audio samples to the underlying hardware abstraction layer (HAL) of the operating system in which the application is running. It is critical for this to happen in a timely manner, so as to avoid unwanted audio artifacts (crackling or popping).

Having the sink as a component abstraction is very useful when building Flipper, since it enables portability of the code. As a result, we can unify the exposed API to the rest of the stages in the pipeline, regardless of the specific platform it runs on. On iOS, these specificities are in Core Audio’s APIs, while on Android, the samples are processed with other bindings.

Android Audio APIs

The Android OS has evolved over the years, and along with it, so has its audio functionality. As a result, there are now a few sets of audio APIs. In Android Gingerbread, the Open Sound Library for Embedded Systems (OpenSL ES), a C library built with portability (and mobile) in mind, graduated to be the recommended interface to the audio hardware. Both the application-level player (MediaPlayer) and third-party developers have access to OpenSL ES APIs via the Android NDK.

Even though OpenSL was able to support Android’s use cases for a considerable amount of time, it started showing signs of being outdated as applications evolved to require higher performance. Beginning with Android Oreo, however, Google integrated a new C library with a simpler API surface to target these low-latency scenarios: AAudio.

Since then, Google has been advising developers to use AAudio for building audio applications, but the API availability becomes an issue when targeting a large audience of users, as fragmentation becomes more evident. Developers still have to maintain code for OpenSL ES bindings if they want to provide audio capabilities on older versions of the operating system. On top of that, manufacturers also apply changes to the behavior of the APIs, deviating from the official documentation Google provides and forcing all sorts of workarounds and device-specific hacks.

Enter Oboe, a C++ wrapper built on top of OpenSL ES and AAudio, that exposes an “AAudio-like” interface, includes fixes (or workarounds) for known manufacturer-specific issues, and automatically picks the most suitable audio API based on a device’s capabilities. If you’re already familiar with Android development, this is the same idea behind the Support Libraries, in that it provides a compatibility layer between the application and platform code.

SoundCloud and Oboe

The cost of maintaining two implementations for our Flipper sink on Android and the risk of replacing a critical part of our already battle-tested core library for playback were among the reasons we postponed the integration of our player with AAudio, but the release, and later, stable graduation, of Oboe motivated the work on a prototype integration.

Following our usual rollout process, we started by deploying Oboe to a controlled subgroup of our userbase by using a remote feature toggle, and as we got more confident with the performance in production, we extended its availability.

It’s been a few months since we began measuring Oboe’s integration with Flipper in production, and while it is still a rather new tool, it has proven to be beneficial to us in many ways.

For one, the global community united around improving Oboe introduces new concepts and capabilities faster than what we could do internally. It also provides us with access to the experts behind the platform APIs, along with the possibility to contribute back to the upstream repository, due to its open-source nature. Finally, the combination of Oboe and AAudio gives us richer insights into the system’s HAL and its performance — for example, the number of buffer underruns, which allows for additional optimizations.

Most importantly, our users also gained: We have been able to provide a better experience for playback, latency-wise, as our telemetry systems show that AAudio contributed to an approximate 10 percent decrease in overall latency in comparison to the same sampling size with OpenSL ES.

All in all, we are happy with the results of what started as a prototype implementation, and we’re aiming to make the new audio sink available to our entire userbase in the near future, while still keeping an eye open for further improvements to latency and the playback experience in general.

If you’re interested in media streaming, Android, large-scale challenges, and what we do at SoundCloud, make sure to check out our jobs page!

Alerting on SLOs like Pros

Tue, 04 Jun 2019 00:00:00 GMT

If there is anything like a silver bullet for creating meaningful and actionable alerts with a high signal-to-noise ratio, it is alerting based on service-level objectives (SLOs). Fulfilling a well-defined SLO is the very definition of meeting your users’ expectations. Conversely, a certain level of service errors is OK as long as you stay within the SLO — in other words, if the SLO grants you an error budget. Burning through this error budget too quickly is the ultimate signal that some rectifying action is needed. The faster the budget is burned, the more urgent it is that engineers get involved.

This post describes how we implemented this concept at SoundCloud, enabling us to fulfill our SLOs without flooding our engineers on call with an unsustainable amount of pages.

Let’s Talk SRE

SLOs, alerting, and error budgets are central concepts of Site Reliability Engineering (SRE), a discipline invented by Google in the early 2000s. Although it was initially proprietary to Google, over the last few years, SRE has become a widely discussed topic and is now applied in various flavors throughout the industry. An important milestone in this development was Google’s publication of the bestselling book Site Reliability Engineering: How Google Runs Production Systems. The book is available online for free. If you run anything even remotely at scale, there is really no excuse as to why you shouldn’t read this book in its entirety. Among all of its important chapters, there are three of particular importance for this blog post:

Chapter 4 — Service Level Objectives explains everything you need to know about SLOs (and SLIs and SLAs, in case you’ve ever wondered about the difference).
Chapter 6 — Monitoring Distributed Systems lays the foundation of “alerting philosophy” and creates an understanding of when it makes sense to page engineers (and when it does not).
Chapter 10 — Practical Alerting mostly describes Borgmon, Google’s traditional internal monitoring system. While Borgmon is proprietary to Google, the author points out that you can try out the described concepts at home (or at work, even) because many current open-source monitoring systems follow similar semantics, with Prometheus receiving a special shoutout. Conveniently, we use Prometheus for the large majority of our monitoring (which is not surprising, as most of Prometheus’ initial development took place here at SoundCloud).

Keep in mind, though, that you are not Google. We all can learn from the giants, but blindly copying them could go horribly wrong. On that note, how different organizations ventured into SRE territory is the main topic of yet another SRE-themed book: Seeking SRE, which was curated and edited by David Blank-Edelman. We at SoundCloud contributed a chapter, too, in which we talked about our somewhat bumpy road to applying SRE principles in an organization that is totally not Google.

The Google SREs themselves also felt the pain of us “mere mortals” trying to figure out how to make use of the lessons from the SRE book in our day-to-day work, so they published a companion to the original SRE book, called [The Site Reliability Workbook], which they again made available online for free. There are a bunch of cookbook-style chapters that are directly applicable in practice. One of them, Chapter 5 — Alerting on SLOs, describes perfectly what we had been doing already, albeit in a less sophisticated way. With the inspiration from that chapter, we were able to refine our SLO-based alerting with great success. In fact, the solution we ended up using is so similar to the one described in the chapter that you should just read it together with this blog post to get the most out of it.

Symptoms vs. Causes

But let’s first take a step back and think about alerting on symptoms vs. causes. This is an important topic in Chapter 6 from the original SRE book (where you can read up all the details if you feel so inclined).

Back in the days when your entire site ran on one LAMP server, there wasn’t much of a need to think about the difference between symptoms and causes. If your one web server went down (cause), your site was down (symptom). Waking somebody up the moment your web server stopped replying to ping probes was the right thing to do.

Nowadays, even mid-size sites run on hundreds, if not thousands of servers, and one or two of them are almost guaranteed to be down at any time. And that’s fine, because the software is designed to tolerate some failures. As such, it wouldn’t make any sense to wake somebody up whenever a single machine goes down.

You can apply the same thought to many other — actual or potential — causes of an outage. Paging alerts, those that wake you up in the night, should be based on symptoms, i.e. something that actively or imminently impacts the user experience. In addition, the alerting condition should also be urgent and actionable. If any of these requirements are not met, the alert should not be a page.

However, there could still be a meaningful alert on a non-paging level. Let’s call these kinds of alerts tickets, as somebody will have to look at them eventually, e.g. a dead server has to be dealt with at a suitable time. A user-visible but not urgent problem can easily be addressed during work hours rather than in the middle of the night, but even during an actual outage, it is helpful to get informed (via some kind of dashboard rather than via your pager) about possible causes of an outage that are detected by the monitoring system.

Ideally, your SLO is a precise definition of an unimpacted user experience. In other words, symptom-based alerts are usually alerts on (active or imminent) violations of your SLO.

Black-Box vs. White-Box Monitoring

With the lessons of symptoms vs. causes in mind, you might think now that black-box monitoring is the obvious choice for detecting symptoms, while white-box monitoring is ideal for causes. That’s a reasonable thought, but in reality, our practice at SoundCloud is a perfect match of what Google tells us in Chapter 6 of the SRE book: “We combine heavy use of white-box monitoring with modest but critical uses of black-box monitoring.”

We mostly use black-box probes as a workaround for legacy software that isn’t yet instrumented for white-box monitoring. And we run fairly complex Catchpoint probes to simulate typical usage of our site. Interestingly, these probes are what we use to determine if we have hit our monthly, quarterly, and yearly availability goals, but we don’t use them to page anybody. Why is that so? The lazy answer is: We need white-box monitoring anyway (to investigate causes). Why should we invest in additional black-box monitoring except for in the most critical cases? But there are more reasons (quotes are again from the SRE book):

Ideally, we want to detect user-impacting outages before they do noticeable harm. However, “for not-yet-occurring but imminent problems, black-box monitoring is fairly useless.”
Probes are not real user traffic. For example, the test track that our Catchpoint probes attempt to play might be perfectly available, while, for some complex reasons, the rap battle that has just gone viral is not.
Measuring long-tail latency and detecting low error rates take a lot of probes and thus a long time. However, tail latency is quite crucial in distributed systems, and our availability goals are usually measured in many nines, and so even a small error rate might violate our SLO. This could be the main reason why we don’t page anybody based on our Catchpoint probe. The probes are too slow to detect low but significant error rates, despite already limiting them to probing a very specific business KPI. The latter is a problem on its own, as we want our paging alerts to cover all relevant features of the site. However, the long-term results of the Catchpoint probes are great as a general quality calibration for our other monitoring. If they detect something, we can revisit historical alerting records to see if the near-real-time monitoring caught the issue, too.
In a multilayered system, like a microservice architecture, “one person’s symptom is another person’s cause.” A frontend service sees a failing backend service as a cause, while for the backend service, its own failure is a symptom, as it is now violating the (internal service-to-service) SLO. Adopting this view, we can use the white-box monitoring of the instrumented frontend service to track requests issued to the backend service. This is essentially black-box probing of the backend with the complete and real user traffic.

The most rewarding place to apply the pattern described in the last point above is our HAProxy-based terminator layer at the edge of our infrastructure. An earlier Backstage Blog post described it in detail.

Recently, HAProxy announced the introduction of a native Prometheus metrics endpoint, but the versions we have been working with are not yet instrumented for Prometheus. This gap is bridged by the Prometheus HAProxy exporter, a small glue binary that can be scraped by the Prometheus server in the usual way, whereupon it talks to HAProxy in its own fashion to retrieve metrics and convert them to the Prometheus format and then sends them back as the scrape result. By running such an HAProxy exporter next to each HAProxy instance, we get all the metrics we need. In particular, there is a counter for all HTTP requests HAProxy has sent to the various backends, partitioned not only by the backend, but also by the HTTP status code. With Prometheus, we can easily calculate relative rates of 5xx responses over arbitrary time spans.

Multiwindow, Multi-Burn-Rate Alerts

With the error rate as seen by the HAProxy terminator layer, we can now apply the alerting technique described in Chapter 5 of the SRE workbook. If you haven’t yet read that chapter, now is probably really a good time to do so. It progresses quite nicely, approaching the ultimate solution iteratively and solving one problem after another. Spoiler alert: Here, we will only implement the ultimate solution, the sixth iteration in the chapter, which is called Multiwindow, Multi-Burn-Rate Alerts. This features a set of alerts, each of which has a different alerting threshold that corresponds to different rates of burning through the monthly error budget. Depending on the burn rate, each alert uses a combination of two different time windows over which the error rate is calculated. Fast burning is detected quickly. Slow burning needs a longer time window.

First, we need to actually set the SLO. The chapter from the SRE workbook assumes an error rate of 0.1% (or a 99.9% success rate) as a typical value. That’s also generally true for SoundCloud, so the various numbers calculated in the chapter make sense for our scenario. However, some backends might have a stricter or more relaxed SLO. We still keep the various burn rates and window sizes the same for them, but we allow the target error rate to be configured by backend. This configuration takes the form of rules in PromQL (the Prometheus expression language, which is also used in the SRE workbook). They look like the following:

- name: slos_by_backend
  rules:
  - record: backend:error_slo:percent
    labels:
      backend: "api-v4"
    expr: 0.1
  - record: backend:error_slo:percent
    labels:
      backend: "api-v3"
    expr: 0.2
  # ... Many more backends.

Next, we need the actual error rate averaged over various time windows: 5m, 30m, 1h, 2h, 6h, 1d, 3d. These are also calculated as recording rules, but they are now based on actual live data instead of a constant number. This is the rule for the 5m case:

- name: multiwindow_recording_rules
  rules:
  - record: backend:haproxy_backend_http_errors_per_response:ratio_rate5m
    expr: |2
        sum by (backend)(rate(haproxy_backend_http_responses_total{job="haproxy", code="5xx"}[5m]))
      /
        sum by (backend)(rate(haproxy_backend_http_responses_total{job="haproxy"}[5m]))
  # Other rules accordingly with 5m replaced by 30m, 1h, 2h, 6h, 1d, 3d.

The rate with the [5m] range selector calculates requests per second, averaged over the specified timeframe, from the ever-increasing counters. The sum aggregator in both enumerator and denominator separately sums up the rate from all HAProxy instances for each backend. Each backend ends up as a label on the elements in the resulting vector. The denominator includes all requests, while the enumerator only selects those with code="5xx — which results in an error rate.

With all these recording rules in place, we can now assemble the multiwindow, multi-burn-rate alerts. We have chosen to use four different alerts, each combining a long window, a short window, and a burn rate factor, as listed in the following table.

Alert	Long Window	Short Window	`for` Duration	Burn Rate Factor	Error Budget Consumed
Page	1h	5m	2m	14.4	2%
Page	6h	30m	15m	6	5%
Ticket	1d	2h	1h	3	10%
Ticket	3d	6h	1h	1	10%

Each alert fires if the SLO error rate times the burn rate factor is exceeded when averaged over both the long window and the short window.

Example: The SLO error rate for api-v4 is 0.1% (to achieve “three 9s” of availability). The owners of api-v4 get paged if their backend has returned more than 1.44% 5xx responses over the last 1h and over the last 5m. They also get paged if their backend has returned more than 0.6% 5xx responses over the last 6d and over the last 30m.

The short window ensures a short reset time of the alert, i.e. an alert should stop firing soon after the problem has been solved (which is not only convenient but also allows for paging to happen again if the problem comes back).

Why the different burn rate factors? They enable pages for fast burning of the monthly error budget and tickets for slow burning of the budget. (Slow burning of the error budget must not go undetected and must be addressed eventually, but it’s no reason to wake somebody up in the middle of the night.) The Error Budget Consumed column lists the percentage of the monthly error budget consumed at the time the alert triggers. The math is easiest to check in the last row, with a burn rate factor of 1: If the error rate happens to be exactly at the target of 0.1%, and that happens for 3 days, then 10% of the monthly error budget is burned because 3 days is 10% of a 30-day-long month.

Note that an alert can fire much more quickly than within the length of the long window. A backend that suddenly starts to return 100% 5xx will cross the 1.44% hourly error rate after only 52s (1.44% of 3,600s).

This leaves us with the for duration column, which is interestingly missing in the SRE workbook. We’ll talk about it in a minute. But let’s first see how the alerting rules actually appear:

- name: multiwindow_alerts
  rules:
  - alert: ErrorBudgetBurn
    expr: |2
        (
          100 * backend:haproxy_backend_http_errors_per_response:ratio_rate1h
        > on (backend)
          14.4 * backend:error_slo:percent
        )
      and
        (
          100 * backend:haproxy_backend_http_errors_per_response:ratio_rate5m
        > on (backend)
          14.4 * backend:error_slo:percent
        )
    for: 2m
    labels:
      system: "{{$labels.backend}}"
      severity: "page"
      long_window: "1h"
    annotations:
      summary: "a backend burns its error budget very fast"
      description: "Backend {{$labels.backend}} has returned {{ $value | printf `%.2f` }}% 5xx over the last hour."
      runbook: "https://runbooks.soundcloud.com/terminator/#errorbudgetburn"
  # Followed by the other three alerting rules.

For the sake of brevity, this only lists the first of the four alerting rules. By replacing the relevant recording rules, the factors, the for duration, and the applicable label values with the values from the table above, you can easily deduce how the other three alerting rules appear.

The SRE workbook also lists the alerting rules in PromQL, but in a more compact form. Our example here is more verbose because we have added a few things we needed for our particular use case:

As explained above, the SLO error rate might be different per backend and the threshold is configured as a recording rule named backend:error_slo:percent. You can find it in the expression above. Note that the > comparison operator is used with a modifier, on (backend), to tell Prometheus which label to use for the label matching.
annotations which follow internal conventions, are added to facilitate handling the alerts. For example, each alert should contain a link to the relevant runbook in our runbook repository.
We also add a system label for alert routing. The Prometheus Alertmanager allows us to configure alert routing based on labels in a very powerful way. Somewhere else in our codebase, we have a registry of which team owns which systems. From there, we autogenerate an Alertmanager config that routes alerts to their owners based on the system label. Everything that can be labeled (for example, pods on Kubernetes) also gets a system label, and that label is propagated into Prometheus. HAProxy only knows the backend, and thus the HAProxy exporter doesn’t expose a system label. However, by convention, we use the system name as the backend name, and thus we can simply set the system label to the value of the backend label. (In harsh reality, the backend label is a multi-component string with other components, too. Prometheus has you covered there as well, with the label_replace function used to cut out the system part. This little complication is left out above to keep things a bit shorter.)
Finally, there is the for clause, with the duration from the table above. An alert with a for clause only fires once the alerting condition has been fulfilled continuously for the configured duration. Intriguingly, the SRE workbook explicitly advises against for clauses because they inevitably increase the detection time for the alerting condition. We still decided to use a for clause with a duration that is short compared to expected response times (of the alert itself and of the engineers getting alerted). The for clause is essentially a simple but effective defense against statistical outliers. For example, if a service has just started to receive traffic after a launch or an undrain operation, the error rate over any time window will be dominated by the very short time it has received traffic at all. Without the for clause, even a very short and modest initial error spike (maybe caused by cold caches) will trigger all the alerts.

Conclusion

With the alerts described above, we can reliably detect breaches of configurable SLOs with an appropriate detection time (the quicker the error budget is burned, the sooner the alert will fire) and a sufficiently short reset time. The resulting alerts are strictly symptom-based and therefore always relevant. To reach a person able to act on a particular alert, it is routed to the on-call rotation owning the affected backend.

Playback on Web at SoundCloud

Fri, 03 May 2019 00:00:00 GMT

Maestro is a library we have developed to handle all playback across SoundCloud web applications. It successfully handles tens of millions of plays per day across soundcloud.com, our mobile site, our widget, Chromecast, and our Xbox application. We are considering open sourcing it, and this blog post is a technical overview of what we’ve achieved thus far with Maestro.

What We Support

At SoundCloud, we aim to support all modern web browsers, mobile browsers, and IE 11. Our goal is to provide the best possible playback experience using the functionality provided by your browser.

What We Stream

We currently stream three codecs:

mp3
opus
aac

Our primary protocol is HLS (HTTP Live Streaming). This means the contents of a track are split up into short segments, and we have a separate file (playlist), which contains URLs to all of the segments, along with their corresponding times in the track. You can find more information about HLS here.

What the Browser Provides

We make use of the <audio /> element, Media Source Extensions (MSE), and the Web Audio API.

The minimum support we require is the <audio /> element and the ability to play one of our codecs. MSE and the Web Audio API are required for the best experience.

We are able to gracefully degrade when either the Web Audio API or MSE is missing or there are errors during playback.

We’ll cover what we use MSE and the Web Audio API for in a bit, but first, let’s see what the <audio /> element alone does for us.

`<audio />`

This is able to take a URL of an audio file and play it back if the codec is supported by the browser. It is informed of the codec in the content-type header on the response, and it provides an API, which can be used to control playback and to determine if the browser supports the codec:

const audio = document.createElement('audio');
audio.src = 'http://example.invalid/something.mp3';

audio.play();

Media Source Extensions

With the <audio /> element alone, the browser does all the work behind the scenes, and you don’t have access to the underlying buffer.

With MSE, we are able to create a buffer for a codec that is supported by the browser. We are then able to handle both downloading the media ourselves and appending it to the buffer. This means we can make optimizations, such as preloading, which is where we download the first few seconds of tracks we think you will play before you click the play button and store them in memory. And then when you click play, we can append this data straight from memory into the buffer instead of having to go to the network:

const audio = document.createElement('audio');
const mse = new MediaSource();
const url = URL.createObjectURL(mse);
audio.src = url;
audio.play();

mse.addEventListener('sourceopen', () => {
  // 'audio/mpeg' for mp3
  const buffer = mse.addSourceBuffer('audio/mpeg');
  buffer.mode = 'sequence';
  const request = new Request('http://example.invalid/segment0.mp3');
  fetch(request)
    .then(response => response.arrayBuffer())
    .then(data => {
      buffer.appendBuffer(data);
    });
});

Web Audio API

The Web Audio API is the newest of the APIs mentioned here. We use a small portion of this API to perform a quick fade in and out when you play, pause, or seek. This makes seeking feel snappier, and playing/pausing is less abrupt:

const audio = document.createElement('audio');
const context = new AudioContext();
const sourceNode = context.createMediaElementSource(audio);
const gainNode = context.createGain();
sourceNode.connect(gainNode);
gainNode.connect(context.destination);

audio.src = 'http://example.invalid/something.mp3';
audio.play();

// Schedule fade out.
gainNode.gain.linearRampToValueAtTime(0, context.currentTime + 1);

Goals of Maestro

Simple API
Plugin architecture
Easy feature detection
Type safety
Support all major browsers
Handle bugs/differences in browser implementations
Great performance
- Ability to preload
- As responsive as possible
Configurable buffer length and cache size
- Should be able to work on devices with constrained memory, such as Chromecast
Metrics
- Provide error and performance data, which can be monitored to detect bugs and make more improvements

Tech Stack

The API

Maestro consists of many packages. The core package provides an abstract BasePlayer class, which provides the player API. It delegates tasks to specific implementations, but external communication happens through BasePlayer. Up-to-date state can be retrieved through player methods, and the user is also notified of any changes.

For example, the play() method returns a Promise which may resolve or reject. The BasePlayer will inform the implementation when it should play or pause, and the implementation will inform the BasePlayer when it is actually playing. Each player implementation is decoupled from the actual play() method. This also means the isPlaying() method and corresponding updates can be handled completely by BasePlayer. Another example is getPosition(), which will normally ask the implementation for the current time, except for when a seek is in progress, in which case BasePlayer will return the requested position. This means the time from getPosition() always makes sense and users don’t need to override it when seeking to ensure it doesn’t jump around.

Player implementations are contained in separate packages, and they all extend BasePlayer. We currently have the following players:

HTML5Player — This is the simplest player. It takes a URL and a MIME type, which are passed directly to a media element.
HLSMSEPlayer — This extends HTML5Player, and it takes a Playlist object, which is responsible for providing segment data. This player uses MSE.
ChromecastPlayer — This player is a proxy to controlling a Chromecast.
ProxyPlayer — This player can control another player, which can be switched on the fly. It also has some configuration related to the direction to sync when a new player is provided. One of the benefits of this player is that it can be provided to apps synchronously, even when the real player isn’t available yet. Then, once the real player is available, its state will be synced to match the proxy. Some other use cases of this are switching between playback on a Chromecast and locally, or switching qualities. The app only has to interact with one player, and the switch can happen behind the scenes.

State Management and Events

There is a lot of playback state to manage, and in Maestro, most of this is contained inside BasePlayer. Users also want to know when parts of the state change and will sometimes react to changes by performing other player actions. This introduces some complexities when we are running on a single thread. Sometimes we also want to update several parts of the state atomically (across multiple functions). An example is: If the user seeks to the end of the media, we also want to update the ended flag to true. The logic concerned with updating the ended flag is not tied to the seeking logic in code, but the update of the seeking state and the ended state should happen together in the API.

To achieve this, we built a component called StateManager, which enables us to:

Update multiple parts of the state across functions before calling out to inform the user of the changes.
Notify the user of state changes at the end of the player call stack so that any future interactions they have with the player are not interleaved in the call stack as a result. (For example, do the work and then fire the event, instead of firing the event and then doing the work.)

`StateManager`

The state manager maintains a state object. All changes to this object are made using an update() method, and a callback can be provided, which is then notified of any state changes that happened inside the last update(). These calls can be nested:

type ChangesCallback<State> = (
  changes: Readonly<Partial<State>>,
  state: Readonly<State>
) => void;
type Control = {
  remove: () => boolean;
};
type Subscriber<State> = {
  callback: ChangesCallback<State>;
  localState: State;
};

class StateManager<State extends { [key: string]: Object | null }> {
  private _state: State;
  private _subscribers: Array<Subscriber<State>> = [];
  private _updating = false;

  constructor(initialState: State) {
    this._state = clone(initialState);
    // ...
  }

  public update(callback: (state: State) => void): void {
    const wasUpdating = this._updating;
    this._updating = true;

    try {
      callback(this._state);
    } catch (e) {
      // error handling...
    }

    if (!wasUpdating) {
      this._updating = false;
      this._afterUpdate();
    }
  }

  public subscribe(callback: ChangesCallback<State>, skipPast = true): Control {
    // ...
  }

  private _afterUpdate(): void {
    this._subscribers.slice().forEach(subscriber => {
      const diff = this._calculateDiff(subscriber.localState);
      // We always recalculate the diff just before calling a subscriber,
      // which means that the state is always up to date at the point when
      // the subscriber is called.
      if (Object.keys(diff).length) {
        subscriber.localState = clone(this._state);
        deferException(() => subscriber.callback(diff, subscriber.localState));
      }
    });
  }

  private _calculateDiff(compare: State): Readonly<Partial<State>> {
    // ...
  }
}

Example Usage

type OurState = { a: number; b: string; c: boolean; d: number };
const stateManager = new StateManager<OurState>({
  a: 1,
  b: 'something',
  c: true,
  d: 2
});

stateManager.subscribe(({ a, b, c, d }) => {
  // On first execution:
  // a === 2
  // b === 'something else'
  // c === false
  // d === undefined

  // On second execution:
  // a === undefined
  // b === undefined
  // c === undefined
  // d === 3
  updateD();
});

stateManager.subscribe(({ a, b, c, d }) => {
  // a === 2
  // b === 'something else'
  // c === false
  // d === 3
});

doSomething();

function doSomething() {
  stateManager.update(state => {
    state.a = 2;
    updateB();
    state.c = false;
  });
}

function updateB() {
  stateManager.update(state => {
    state.b = 'something else';
  });
}

function updateD() {
  stateManager.update(state => {
    state.d = 3;
  });
}

Note that the first subscribe callback will be executed twice, but the second will only be executed once, and only with the latest state (i.e. d === 3).

Also note that we don’t get nested call stacks, because the callbacks are only executed once the work is finished.

Browser Limitations

Unfortunately, different browsers have different codec support (which can also depend on the OS) and different container requirements.

Chrome, for example, supports a raw MP3 file in MSE, but Firefox requires the MP3 to be in an MP4 container. This means that in Firefox, we need to package the MP3 that we download into an MP4 in the browser. Other codecs have similar complexities.

It’s also inevitable that there are bugs. Supporting a media processing pipeline that can handle a huge variety of media, without breaking backward compatibility, in a secure way, and in a web browser, is a huge task! Luckily, Maestro is able to handle workarounds for various bugs in different browsers, some which differ under the hood between versions.

Autoplay policies also differ between browsers, and this means we currently have to share media elements between players. This adds complexity because when the source of an element is changed, there are still events emitted for the previous source for a short time after, meaning we have to wait for the “emptied” event before attempting to use it, and we have to keep track of everything that was requested in the meantime. Maestro’s HTML5Player makes this simple with provideMediaElement(mediaEl) and revokeMediaElement(). This allows you to move a media element between players at runtime. When a player doesn’t have a media element, the player is simply stalled.

Testing

The BasePlayer and player implementations are covered by unit tests and integration tests: We use Mocha, Sinon, and Karma, along with mocha-screencast-reporter. The latter is great for viewing the progress of tests running remotely.

The BasePlayer alone currently has more than 700 tests, which ensures that the API behaves correctly. One test, for example, checks that the play() promise is resolved when the implementation reports it is playing. Another checks that play() is rejected with the correct error if the player is killed before the play request completes. There are also tests that check that the player errors if an inconsistency is detected — such as a player implementation reporting that a seek request could not be completed when the BasePlayer never requested a seek operation.

We also run all the tests on a variety of browsers and browser versions (including Chrome and Firefox beta) using SauceLabs. This takes several hours to complete, and so we test the major browsers for pull requests, and then we test everything just before a release. We also run all the tests weekly to ensure that there are no issues arising with new browser versions. Doing so once highlighted a bug with Web Audio in Firefox beta which would cause playback to freeze after the first few seconds.

Progressive Streaming (with the `fetch()` API)

We recently added support for progressive streaming (in supported browsers). This means that instead of having to wait for an entire segment to be downloaded before we process it and append it to the buffer, we’re able to process the data as it arrives, meaning we’re able to start playback before the segment download has finished.

This was made possible with the fetch() API (and moz-chunked-arraybuffer in Firefox), which provides small parts of the data while it is still being downloaded:

fetch(new Request(url)).then(({ body }) => {
  return body.pipeTo(
    new WritableStream({
      write: chunk => {
        console.log('Got part', chunk);
      },
      abort: () => {
        console.log('Aborted');
      },
      close: () => {
        console.log('Got everything');
      }
    })
  );
});

Before we added progressive streaming, if a download failed, we would just retry it, and this logic was pretty self-contained. With progressive streaming, it is more complex because, if a download fails part-way through, the entire pipeline has already started working on the data. We decided to retry the request on an error and discard all bytes that we have already seen. If the retry fails, then we are able to signal the error down the pipeline.

This also brings about more complexity. Before, we knew that each segment contained a complete number of valid units of audio, meaning the different parts of the pipeline could make certain assumptions. Now each part of data can contain fractions of units of audio, so we need to be able to detect when this happens and keep a buffer that waits for a complete unit to arrive.

What’s Next?

We have run Maestro in production since June 2017, and we’ve have had minimal reports of playback issues. We are able to monitor the performance and errors in real time, and in cases where errors occur, we are able to retrieve playback logs, which help with debugging.

We are looking for where to take Maestro next, and that’s where you come in: Let us know how you would use it and what you would like to see :D

If you have any questions regarding this post, or you notice any playback problems on soundcloud.com ;), please get in touch!

Taking Our Bug Bounty Program Public

Thu, 18 Apr 2019 00:00:00 GMT

We’re excited to announce the launch of our public bug bounty program with Bugcrowd — the #1 crowdsourced security platform. This public program is open to Bugcrowd’s full crowd of top, trusted whitehat hackers, and we will award up to $1,500 per vulnerability identified on our website, API, and mobile apps.

Security is a top priority at SoundCloud, and we’re committed to keeping the community and its content safe. And, as a leading audio streaming platform, we’re prepared to handle an extremely unique set of security issues. These span from processing, transcoding, and formatting user-generated content without risking remote code executions, to detecting and blocking malware distribution and preventing illegitimate downloads and streaming access. Additionally, since the platform offers a highly social streaming experience with user-generated content and integration, we have to be mindful of potential XSS and CSRF attacks.

As part of our commitment to our users, we’re focused on building state-of-the-art security monitoring and protection solutions for our platform. In order to balance that focus with the team’s operational work, we’re always looking for ways to improve our efficiency. And one of those ways is to have additional support for handling top-of-funnel security work for vulnerability reports. Examples of this work include triaging, reproducing, prioritizing, and resolving duplicates.

This is where Bugcrowd comes in. Bugcrowd’s community-driven vulnerability testing is a key tool for us to receive external testing on our services and platform, along with explicit pentesting by security agencies and our various internal automated tests and peer reviews. With Bugcrowd, the quantity and quality of vulnerability reports is higher than ever before. Many of Bugcrowd’s security testers follow the same news and read the same forums as malicious users, so they help us react to new attack vectors much faster.

Since using Bugcrowd, we’ve seen several benefits, including:

A significantly lowered barrier to reporting security vulnerabilities and increased quality in security vulnerability reports
Additional dedicated time to focus on building services specific to our needs
Having a known platform with clear processes, taxonomy, and rules that attracts more professional researchers with more expertise
Increased confidence that critical issues are continuously being probed, identified, and addressed

We’re excited to take this next step in our crowdsourced security journey: making our bug bounty program public. To engage in our program, take a look at our program brief: https://bugcrowd.com/soundcloud.

Release Quality and Mobile Trains

Wed, 03 Apr 2019 00:00:00 GMT

Once every two weeks, we prepare new versions of our mobile apps to be published to the app stores. Being confident about releasing software at that scale — with as many features and code contributions as we have and while targeting a wide range of devices like we do at SoundCloud — is no easy task. So, over the last few years, we have introduced many tools and practices in our release process to aid us.

In this blog post, I’ll cover some of the techniques we use to guarantee we’re always releasing quality Android applications at SoundCloud.

All Aboard!

Maintaining high quality is about more than just hitting deadlines; it’s a continuous job. As such, we’re constantly keeping an eye on our code and asset changes to make sure we get the best out of our engineering efforts. In addition to human code reviews, our repositories’ pull requests go through an automated pipeline of tests for every single changeset. This pipeline makes use of industry standard tools for both static analysis and unit, integration, and UI testing.

An automatically generated GitHub comment with statistics about the code changes

We’ve also built tools that highlight other potential issues early on during the code review cycle. By integrating them with other pieces of our day-to-day routine, such as GitHub (above) and Slack (below), we ensure our developers have a more transparent understanding of their applied diffs.

A Slack message sent to the author of the pull request after tests are run

Even so, it’s still a challenge to foresee how an application with our scale will behave in the wild, with the variety of devices, locales, and use cases we support. For that reason, we are also big believers in having a remote feature-toggling system. Not only does it give us a way of preventing issues from spreading once they are detected, but we also use these toggles to enable early internal testing of our new features via our alpha and beta builds and to decouple app releases from feature releases.

By using continuous integration tools, we ensure that our master branch is always deployable. This is accomplished by scheduling time-based jobs to be run — either nightly, weekly, or bi-weekly — or via triggered jobs (for example, when a pull request is merged into our master or release branches). With these CI jobs, we can guarantee our obfuscated and shrinked release builds are all set after the passes of tools like ProGuard or R8; update to the latest localized translations for all the languages we support; and automate other repetitive tasks.

Release Train Model

We have a two-week interval between each of our releases because we follow the Release Train model, in which, similarly to a train that departs a station on a specified schedule, we have specified a day for what we refer to as Code Freeze, which is when two members of the engineering collective (previously selected from a rotation of all members) “cut the master branch” into the release branch. We refer to them as release captains, but people familiar with release trains call them Release Train Engineers (RTE).

Pragmatically, this means that once master gets cut, no new code gets into the changeset that the release captains are responsible for publishing (with the exception of emergency hotfixes, which are then merged back to the master branch).

Following a release train not only helps in limiting the ownership of each pair of release captains, but it also gives everyone a sense of predictability over when changes will be rolled out to the public.

The dates for each code freeze are displayed for the entire company in a shared calendar

As a result, our design and data analysis teams can estimate when their help will be needed for defining and delivering specifications for feature development; product managers know when their experiments and product launches will go live; and the backend and frontend engineering teams are able to plan their work to aim for certain code freeze dates.

Code freeze is just one of the steps described in a detailed checklist for the captains (generated via our CI jobs). Additional steps include naming the new build for internal reference (we follow alphabetical order and always pick from a pop culture character set — we’re currently using Marvel superheroes as inspiration!) and updating documents, links, and Slack channels for the visibility of the outgoing train.

The release captains are then responsible for performing a set of manual regression tests that cover the main use cases of the application and some of the scenarios we could not automate (yet). Once all is verified and the captains feel confident with their release, they are free to start the staged rollout process by deploying the build to the beta channel and subsequently rolling out to percentages of the production user base.

Continuous integration pipeline with staged rollout steps

The Train Has Departed the Station

During the two-week period of staged rollout, it’s the release captains’ job to monitor data, metrics, and dashboards for user reports, reviews, and crashes that might have ended up in production. After evaluating the impact of each of these issues, it is their call whether to request new changes to be applied in their train, which is done by pinging the involved engineers and working toward fixes.

We aim to always be attentive to feedback, but the sooner issues get reported, the sooner we can act to fix them. On that note, I’d like to encourage you to join the beta to get early access to new exciting features, in addition to supporting us by being part of this direct channel of communication.

Just before the end of their two-week “shift,” the captains might decide to hold a “post-mortem meeting,” in which incidents are discussed and their root causes understood and documented so that the entire platform collective can come up with preventive actions to reduce the likelihood of reoccurrence. The idea is that these are blameless, with the sole goal being that of improving systems and processes and spreading the knowledge across the entire team.

End of the Line

Overall, ensuring a good standard of what we publish to our users is a product of many factors. Not only do we need, use, and build tooling and automation to facilitate our daily work, but more importantly, at SoundCloud, we are always attentive to supporting the processes the team puts around building a quality-driven culture.

How to Reindex One Billion Documents in One Hour at SoundCloud

Thu, 21 Mar 2019 00:00:00 GMT

In the past, the Search Team at SoundCloud had high lead times for making updates to Elasticsearch clusters, either during the implementation of a new feature or simply while fixing a bug. This was because both tasks require us to reindex our catalog from scratch, which means reindexing more than 720 million users, tracks, playlists, and albums. Altogether, this process took up to one week, though there was even one scenario where it almost took one month to roll out a bug fix.

In this post, I would like to share the concrete Elasticsearch tweaks we made so that we can now reindex our entire catalog in one hour.

Search at SoundCloud

At SoundCloud, we use what is canonically known as Kappa Architecture for our search indexing pipeline. The main idea here is: have an immutable data store and stream data to auxiliary stores.

So, how does this work? Once a user, for example, uploads a track or a playlist on SoundCloud, we receive a live update¹ in the eventfeeder. Then the indexer fetches the uploaded data² from our primary data source — MySQL — and creates a JSON document, which is published to Kafka. The Kafka topics have log compaction enabled, which means we will only retain the latest version of the documents. Finally, the shipper reads the JSON documents from Kafka and sends them to Elasticsearch via the bulk API.

Since we use Kafka to back the document storage, we can reindex a new cluster from scratch by simply resetting the Kafka consumer offset of the shipper component, i.e. replaying all stored messages in Kafka.

Elasticsearch Cluster Setup

As you can see in the diagram above, in order to have high availability and isolation of failures, we have three clusters running in production, all with the same dataset. Each cluster consists of 30 Elasticsearch nodes running on 30 boxes. As a general rule of thumb, we try to scale Elasticsearch horizontally instead of vertically, i.e. adding more boxes instead of running multiple Elasticsearch nodes on the same box. This Elastic talk provides a good explanation of why this is the case.

Coming back to our setup, each box consists of:

150GB SSD³
64GB RAM
32 cores

So in total, we have approximately 2TB RAM and 1,000 cores available to serve more than 1,000 search queries per second at peak time.

Our Secret Sauce for Fast Indexing

Now that you’re somewhat familiar with SoundCloud’s indexing pipeline, in this section, I’ll share the concrete Elasticsearch tweaks we implemented that contributed to our performance improvements.

Ingredients

Concurrent bulk API calls and large bulk size
Optimal primary shards count
Async translog setting
Turned off Elasticsearch refresh
Merging of Lucene segments
Faster replication settings

Directions

Bulk API

Make sure that the sending part (in our case, shipper) utilizes the maximum possible combination of bulk size and concurrent requests against the Elasticsearch bulk API. To find this exact sweet spot, try to incrementally increase the batch size until the indexing throughput no longer improves. Since finding the exact sweet spot is a process of trial and error, rinse and repeat!

Primary Shards

Note: The primary shard configuration is a static setting and must be specified during the index creation time.

As a rule of thumb, the count of primary shards defines how well the indexing load is distributed on the available nodes.⁴ Before we started our spike, we had 10 primary shards distributed on 30 Elasticsearch nodes. Yes, this doesn’t sound like a configuration, because effectively, only 10 machines would be utilized and the rest would be idling. But after changing the primary shard size to 30, we saw lots of improvement in the indexing throughput.

…And do not forget to turn off shard replication during the actual reindex!

Translog

The translog in Elasticsearch is a write-ahead/action log which is needed because the actual internal Lucene commit operation (writing to disk) is an expensive operation due to it occurring synchronously after every write request. The log stores all indexing and delete requests made to Elasticsearch. Furthermore, it also helps with recovery from hardware crashes by replaying all recent acknowledged operations that were not part of the previous Lucene commit.

By default, there is a synchronous fsync that takes place for each indexing request, and the client receives an acknowledgment afterward. But during the reindexing, we usually don’t need this guarantee and could simply restart the process in the case of a hardware failure.

By setting index.translog.durability to async, both the fsync and the Lucene commit will now occur in the background, thereby making the reindexing a lot faster. Remember to set the durability back to the default value(request) once the bulk indexing is done. You can also enhance the flush threshold by changing translog.flush_threshold_size. In our setup, we’re using 2GB. The default value is 512MB.

Refresh Interval

As you may know, the refresh interval in Elasticsearch directly controls when an indexed document will be searchable. Since there are no searches during the reindex, you can turn the refresh process off completely by setting the refresh_interval to -1.

Note: The default refresh interval is one second.

Segment Merging

After the primary shards have been indexed, you may want to turn on shard replication next, but before doing so, allow Elasticsearch to merge Lucene segments via _forcemerge?max_num_segments. In doing so, the index will be compactified before it is replicated, i.e. saving disk space. Currently, we’re investigating the ideal merging strategy in our setup. If you are curious how Lucene’s segment merging works, you can find a great visualization of it here.

Replication

Now there’s only one last thing to do: Turn on shard replication! But before doing so, consider increasing the concurrent shard recoveries per node (default: 2) and the recovering network throttle (default: 20MB) by tweaking cluster.routing.allocation.node_concurrent_recoveries and indices.recovery.max_bytes_per_sec.

In our setup, we are allowing 100 concurrent shard recoveries, and the indices.recovery.max_bytes_per_sec rate is set to 2GB. Make sure you revert it to the previous value before going live!

Nutrition Facts

Indexing Throughput

Before

After

Including the long tail, the primary shard indexing takes around 30 minutes for more than 720 million documents, whereas it previously took three hours.

Search Latency

Our search latency for 95 percent of all requests dropped to roughly 90ms⁵ after rolling out the 30 primary shards configuration.

Note that the listed tweaks above were used in Elasticsearch 5.6.10. Depending on your used version of Elasticsearch, the shown tweaks might not work or could end with slightly different results.

Summary

When we first made the above improvements, we only had two engineers working on the Search Team, and as such, the new lead time for bringing up new clusters enabled us to be much more productive: We could now start an A/B test, prepare a second one, mitigate a recent incident, and reindex a development cluster in one sprint, whereas previously we hardly had room for more than two indexing runs in the same time span!

The shown performance improvements helped to cut down the reindexing time for new clusters from one week to one hour, thereby enabling the Search Team to perform more effectively. In addition, we simultaneously improved the search latency by four times for 95 percent of all search requests.

If our search problems sound interesting to you, then feel free to apply for the currently vacant backend engineer position on the Search Team! We would also love to hear your stories about managing and working with Elasticsearch in production or, in general, any other search-related topic. Don’t hesitate to get in touch via @bashlog.

Future Work

Our work doesn’t stop here. There are, of course, more improvement areas beyond what’s detailed above. In the near future, we plan to further investigate the following topics:

60 Primary Shards:
During a small test, we could reach up to even 1 million indexing op/s with 60 primary shards. A load test is currently pending to verify that the search latency won’t be degraded by having 60 primary shards.

Segment Optimization:
As explained in the segment merging section, we want to verify if having only one segment in the indexing process could eventually result in Lucene segments that are too large.

Smaller Clusters:
Currently, we are running clusters with 30 nodes. Here we want to find out the minimum number of nodes necessary that will allow us to handle the same amount of requests, in addition to having the same indexing performance and search latency for users. Smaller clusters would drastically decrease our operational efforts.

Optimized Schema:
We also want to investigate whether our current schema could be further optimized, e.g. removing unused fields or analyzers. This would make the indexing process faster, and it could also save space.

Bonus Material

Identify and Enhance the Capacities That Make Things Go Right

So, the bonus is the simple idea above, which I thought was worth sharing with you. Instead of the usual focus on why do things go wrong?, one should also ask the opposite question: Why do things go right?

By increasing the capacity of things that go right, we can automatically lower the chance that things go wrong in the first place. In terms of our situation described above, we proactively planned time to improve what was already working, and as a result, we made it a lot better.

TL;DR

In the past, the Search Team at SoundCloud had high lead times for reindexing Elasticsearch clusters. Recently, we optimized the time required for this task, bringing it down from one week to one hour.

This post is a follow-up to a Meetup talk that Rene Treffer (Production Engineer at SoundCloud) and I gave last December at the Berlin Elasticsearch Meetup on the topic of the latest improvements in search.

1: Our average live update rate is 50 documents per second.

2: The data that we actually index in Elasticsearch is a small subset of what is available in MySQL, and it consists of basic fields such as title, description, and user.

3: If you are also using SSDs, make sure your I/O scheduler is configured correctly. See here.

4: Likewise, you can increase the search throughput by increasing the replication. This, of course, makes it necessary to add additional nodes to the cluster.

5: During peak time, the latency for the 95th quantile reaches up to 150ms.

Off-Platform Validation

Wed, 06 Mar 2019 00:00:00 GMT

Last month we launched SoundCloud Premier Distribution, which allows creators to distribute their music from SoundCloud to many other streaming platforms and stores. The result? Artists can now get paid for plays on other platforms just like they can on SoundCloud.

For many of our users, this will be their first experience with the conventions and requirements of the music industry supply chain. The barriers to entry to this world are very different than those to a creator uploading to SoundCloud. At SoundCloud, we’re proud of how easy it is and how few clicks it takes to publish your work. In contrast, the process of getting your content onto Spotify, iTunes, or any of the other platforms fed by distributors has many more strict requirements regarding metadata and media.

The aim of SoundCloud Premier Distribution is to make the path from SoundCloud upload to off-platform plays as frictionless as possible. Here we’ll look at how a system of automatic and manual validations allows users to get fast feedback as they prepare a release.

What Are These Extra Requirements?

To upload to SoundCloud, the minimum metadata required is the track title, and even that gets auto-completed based on the name of the file you upload. Easy. There’s a maximum length restriction, but that’s about all.

To send content to other platforms, the title has many more strict requirements, as do other fields. There are syntactic, structural, and semantic rules that have to be observed. Some examples of these are:

The metadata has a fixed character set that, in particular, does not include emojis. 😢
Words in the title should be capitalized, with the exception of a list of short prepositions.
Some words — such as references to physical formats or other streaming platforms — are not allowed.
The artist or label name shouldn’t go in the title, as there are other fields for that.
You shouldn’t specify in the title that it is a live recording or a remastering — there’s another field for that.
The audio format has minimum quality requirements, as does the artwork image format.
The title in the album artwork has to match the title.

There are so many rules and style guides with frightening levels of detail about what will pass quality control for the various streaming services.

Many of these rules are difficult to automate completely. We can’t just block the word “live” in the title, because while “My Hit Song (Live Version)” isn’t allowed, “Live and Let Live” is fine.

Not only are the rules difficult to navigate, but they are also spread out in time. While we can check some rules in real time whenever a user makes a change on soundcloud.com, some take too long to run every time, or they require manual input from our QC team. These checks have to happen asynchronously, with results shown to the user later.

What’s worse is we can also receive validation failures from streaming services much later — in some cases, even after a track has been live for a while.

All that said, how do we help the user through this maze of rules that can fail immediately or even weeks later?

Automatic Checks with Supervision

We’ve built a system of validation rules that can be applied to all of the different fields of metadata. This system checks that data is present and follows the rules, and it reports errors to users. Some rules are easy and can be entirely automatic — for example, image dimensions. Some are almost impossible to check automatically — for example whether or not a track is explicit. And some would be expensive to automate and hard to make correct — for example, genre checking or parsing text from an image. Whenever a change is made to the metadata of a track or a release, it needs to be checked before the update is shipped off-platform.

So we’ve settled on a mixture of rules that can be applied automatically, those that require human eyes, and those that can defer to a human if there’s doubt. Everything can be overridden manually later.

The thing is, we need to cope with a lot of volume, and we only have a finite number of humans checking these releases. And while computer time is cheap, theirs is precious.

As such, there are two lifecycles here that we can model separately:

Fast automatic feedback — We want to know whether each product is valid each time we look at it, and so we run the validations every time a user looks at a page. This is fine for simple validations about the presence of data or many of the rules about text fields.
Asynchronous feedback — There are expensive validations (e.g. manual) that we want to run at most once for each value of the content (whether text, audio, or image), and not repeatedly if it doesn’t change.

Consider that a release might change over time as more metadata is added or changed. Each time there’s a change, the fields that were edited should be checked by the QC team, but the team shouldn’t have to, for example, approve the same artwork every time. Unless the image changes, we should store the pass/fail result and use that whenever we want to check the validity of the release. We call that stored pass/fail result evidence for the validity of the release.

To understand these two lifecycles, we can think of a validation result as being the output and having evidence for the validity of the content as an input. When we check a release, we look for indications that the value satisfies the validation rule — either as stored evidence applicable to that value, or by actually checking the content.

This distinction between slow and fast and synchronous and asynchronous implementations of validation rules has allowed us to run them a lot — every time the user looks at the page, every time we build the rendering of a release that we send to a partner, and every time the release is shown in the QC process. This means that the results are both fast and as up-to-date as possible.

Simple Patterns

Caching values so they don’t have to be recomputed is not a novel idea. Here we’re treating evidence of validity as a cached result of a validation rule. This has allowed us to model changes to data and the various stages of quality control in a way that combines these very different lifecycles. As a result, we can give fast and consistent feedback to users distributing their content, all without overwhelming our QC team.

Radical Candor: An Experience Report

Fri, 15 Feb 2019 00:00:00 GMT

Although it can be easy to know if you’ve messed up badly as a manager, it’s not always as easy to know if you’re doing a good job. In particular, the power dynamics at play can make it hard for people on your team to feel confident letting you know what’s working well and what’s working not so well.

I’ve been managing managers for a while now, and through the years, I’ve tried a few ways to help close this gap. In this article, I’m going to talk about an approach I started using in the last few years that seems to strike the best balance of getting the input managers need while still promoting a healthy culture of direct feedback.

The Approach

Back when I was working at Sky, I came across a post by Kim Scott entitled “Radical Candor — The Surprising Secret to Being a Good Boss.” The title was intriguing enough to get me reading, but what I discovered was an approach to management that was genuinely refreshing. The following diagram is a great summary of the core concept and some of the terms she uses:

I loved the concept, especially since it challenged some of my own tendencies toward “Ruinous Empathy” — caring personally for my coworkers but failing to challenge them directly — and gave me a path to provide more meaningful support for my team.

The book that followed (see radicalcandor.com) is a treasure trove of tools on how to put these ideas into practice, and it contains a section dedicated to the Manager Guidance concept introduced in Kim’s original post.

One aspect of this concept is a skip-level meeting, which is generally defined as a meeting held between a manager’s manager and their direct reports, thereby skipping the level between them. Generally speaking, Kim’s Manager Guidance process follows these steps:

A skip-level meeting is held specifically to ask people how their boss could become a better manager. During this meeting, notes are collected and then shared with the manager after the session.
A follow-up discussion takes place between the manager and their manager to provide additional context about the feedback and answer any questions.
The manager then speaks to their team about what they learned from the feedback and what changes they intend to take in addressing it.

The specific topics people raise in the skip-level meeting are interesting and useful, but most importantly, we want to learn whether people feel good about the feedback culture between themselves and their manager.

My first attempts in using the process at Sky revealed a lot, but for this article, I’m going to focus on how it worked here at SoundCloud, as I was able to put into practice some of what I learned from the first time around. I’ll explain more about how I prepared for the sessions, how I ran them, and how they were received by the teams and their managers. I’ll also cover some of the ways in which they could go wrong and how to address these potential issues.

Preparation

First off, any manager having this process run on their behalf needs to be bought in. They should at least read Kim’s blog post, but better yet, they should read the relevant chapter in Radical Candor, and ideally, the entire book! If, after all of that, they aren’t excited, then it’s better to find a different approach for their guidance.

When teams are invited to a manager guidance session, I provide them with key information. An example invite might read:

This session is to gather feedback to help [your manager] be a better manager. It’s based on a framework Kim Scott talks about in her book Radical Candor, and when I used it (both for me and by me) in a previous company, I found it to be a really helpful exercise.

I recommend reading the entire article here, but the TL;DR about this session can be found in the section starting “Make it easier to speak truth to power.”

There’s no need for you to prepare anything; I’ll capture notes and ask questions to help keep the conversation on track. [xxx] and I want you to feel confident about being completely open.

Running the Session

Structure and Process

In the Radical Candor book, Kim recommends a very specific structure for the meetings, which I mostly followed, as it works well. Here’s how I interpreted it for my use:

Short introduction (c. 5 minutes)
Open discussion (c. 40 minutes)
Reviewing notes together (c. 15 minutes)

As the moderator, I capture notes and I display them on a screen behind me so people can see my terrible note-taking skills in real time and tell me when I capture their points inaccurately.

Introduction

On rereading Kim’s book recently, I realized I had forgotten to apply one piece of advice, which was to get the managers to explain the purpose of the meeting to their teams ahead of time. This meant that a very small percentage of people attending the meeting knew what to expect.

I was anticipating this, so I included time at the top of the session to run through the purpose and set some ground rules. Some of the key points I called out are:

This is about helping the manager learn and grow.
The session shouldn’t focus exclusively on either positive or negative points, as it’s all about getting balanced feedback.
It’s also about checking in on the flow of feedback between the team and their manager.
Everyone’s voice needs to be heard.
The notes are not attributed to individuals, but people still have to be comfortable with them being shared directly.

Topics

I wanted to have some consistency in approach across the sessions (I have five managers reporting to me), so I came up with a loose structure as follows: For Me, For My Team, For the Company. In each section, I explained the overall topic and then used simple starter questions to get the conversation going.

I tend to go with somewhat neutral and uncontroversial questions so as not to push the conversation in a particular direction, but rather see where it gets taken by the group, and I then dig deeper when something interesting emerges. However, sometimes if I know there’s been a particular theme that’s affected a team (e.g. confusing priorities or a visible conflict), I might ask specific questions about how their manager handled it.

The questions you ask can reveal a lot about what you expect from managers on your team, and that’s not a bad thing! At SoundCloud, we recently completed an exercise to define expectations for Engineering Managers, and I expect I’ll start using some of these expectations as a basis for sessions in the future.

For Me

This section gives the attendees the opportunity to talk about how their manager supports their specific needs as an individual. Some example starter questions I used:

How often do you get together 1 on 1? Is that the right cadence for you?
Who controls the agenda for those sessions? Do you get to talk about everything you want to? Do you ever talk about your career plans/aspirations?
How available is your manager to you outside of those sessions?
Does your manager actively seek out opportunities for you that align with your aspirations?
Does your manager ask you for feedback? When did you last give them critical feedback? How did they take it?
Do you feel that you’re learning from your manager? Do you have any specifics?

For My Team

This section is all about the health of the team as a unit and the manager’s contribution to that. Some example questions are:

Do your team meetings run smoothly? E.g. clear purpose, kept on track, actions logged, everyone is heard.
Is collaboration encouraged? Does your manager seek to avoid silo-ing of knowledge/skills?
Do you have social events as a team that everyone can engage with?
Do conflicts within the team get resolved?

For the Company

This section is about how the manager contributes to the success of the relationship between their team and the rest of the business. Here is an example of some of the questions:

Is it clear the role your team plays in the company’s success? Do you think you’re working on things that contribute to that?
Do you feel that the successes of your team are known about in the rest of the company? Are you well represented in key meetings, e.g. All Hands?
Does your manager advocate for you as a team when you raise topics that need assistance from outside of the team?
Do you get clear information about changes to company priorities or policies from your manager? Do they manage to answer all your questions, or at least point you in the right direction?

After the Session

Once the session is complete, I plan a note review session with each manager, and I share the notes with them the evening before. This gives them enough time to digest the information and have questions prepared, but not so long that they stew on any negative points.

The session itself is around 30 minutes long, and I set expectations around the next steps, which start with the manager summarizing the key learnings from the notes and what changes they plan to make. They should then talk through these ideas with their team to reinforce the message that the feedback is being heard and acted upon.

How Did It Go?

How Was It for the Teams?

Probably the most common piece of feedback I received was along the lines of: “That wasn’t as bad as I was expecting!” It can be stressful to think about giving feedback to your manager, so a process that normalizes and encourages it can bring relief to many people.

Here is an explicit piece of written feedback I received:

“…I found that session really really nice. I was very apprehensive going into it, but in the end I think it was one of the most comfortable feedback-collecting formats I’ve been a part of. It was so nice to be able to talk and have someone else write it down, to be able to hear what other people had to say and triangulate, and to frame the whole thing as guidance and direction rather than feedback…”

The other realization I often see dawning on people during the sessions is how difficult it is for a manager to tailor their approach to a group of diverse individuals.

What Feedback Was Given? How Did the Managers React?

Thankfully, there were no major surprises in the feedback for any of the managers in my team, but there were plenty of items they were able to take forward and work with. I had universally positive responses from them, and when we talked about the approach with other managers outside of my group, it was great to hear them advocating for the process and the value it brought them.

For some managers, people on their teams were asking for more critical feedback on their work, as they wanted to understand more deeply how they could grow. Other managers who had focused attention on that already heard that they should also apply the same rigor to giving positive feedback, which is a theme in Kim’s book too.

What Did I Learn?

One piece of advice I didn’t follow from Kim’s book, as mentioned above, is to get the managers to explain the process to their team in advance, which would definitely have saved some time in the sessions. Some people weren’t sure whether their manager would be there and even whether they would be on their own in the session.

The other thing that is clear is the time in the feedback sessions passes very quickly, so it’s important to be very structured about moving through the topics and try to avoid getting stuck on any particular point for too long.

What Could Go Wrong and How to Address It

Gripe Session

If one or more people in the group choose to use the session to moan about their boss, it can be quite detrimental to team morale. Also, as Kim points out in her book, there’s a danger that the moderator can appear to presume that the boss is guilty, or — just as bad — try to defend them against the criticism. Having said that, the complaints are a window into a problem within the team that benefits from being revealed.

The moderator can steer the conversation to avoid getting stuck on a topic that might only have relevance for a small subset of the group, e.g. “[xxx], does this affect you as much as it affects [yyy]?”

No One Talks

One of the reasons I like to be prepared with a set of questions is that sometimes it can be hard to get the feedback flowing with certain teams. I tend to think that’s not a great sign about the feedback culture, but it can also just mean people are reserved and need help exploring the topic. This is also why I like to start with neutral/factual questions that everyone can feel good about answering, like “How often do you have 1:1s?”

Only a Subset of People Talk

Often, teams contain natural leaders aside from their manager, which can lead to interesting dynamics wherein those people can try to steer the conversation on the moderator’s behalf. Reminding the group that the guidance needs to be based on feedback from the whole team can help keep these people from dominating the conversation.

The Group Is Too Big/Too Small

So far at SoundCloud, I’ve run this process for five managers, and the groups attending the sessions have varied in size from three people to seven people. That’s probably a good range, as with only two people, it’s very obvious when collecting the feedback that it’s going to be directly attributable. I would consider 1:1 interviews in such a case.

With a larger group, it’s more difficult to manage the collection of the feedback and make sure every voice is heard. Running multiple sessions to cover the full group is a good alternative in this case. This can happen naturally if the team structure already implies a way to split the group — for example, backend and frontend engineers.

The Process Blocks Direct Feedback

One risk that a previous team member of mine called out is the possibility that people lean on this process as a way to avoid giving direct feedback to their manager. This is valid, and it’s one of the reasons why Kim recommends not running the process too frequently. In the Radical Candor book, she suggests once a year, but I personally think twice a year will be fine for us, since we have rather dynamic teams and the managers on my team are keen to have more frequent check-ins.

I haven’t personally run a follow-up session with a team before, but a key part will be to discuss how successful the manager’s attempts to address the previous feedback have been. Additionally, I would want to find out if there is any change in how confident people feel about giving direct feedback to their manager.

Some People Can’t Make It

It can be difficult to schedule the sessions so everyone can make it along, and there are always going to be situations, like sick days, that mean the whole team can’t get together at the same time. When this has happened for sessions that I’ve run, I have shared the notes with the people who couldn’t make it and asked them if they had anything to add. If they did, I asked them if they either wanted to provide those as written notes, or if we should meet face to face to discuss. I also make sure this is all completed ahead of the review session with the manager.

Bad Notes

The notes are a key output of the process, so the time taken during the session to review them is critical. Whenever I’ve run these sessions, we’ve found ambiguities or confusing language that we’ve been able to correct as a group. It really makes a big difference to how much the manager can get from the process if they can refer back to these notes and get a clear reminder of what they need to address.

Conclusion

A few years ago, Google published some amazing research on the key dynamics that set successful teams apart from other teams at Google. By far, the most important element of this was Psychological Safety, which they defined as: “Team members feel safe to take risks and be vulnerable in front of each other.”

A healthy feedback culture is fundamental to establishing this Psychological Safety, and my experience with using Kim Scott’s Manager Guidance approach is that it can really make a positive difference in that culture. If you’re a manager, why not ask your manager to run this process for you? I’m really looking forward to hearing the feedback that comes from mine!

Garbage Collection in Redux Applications

Thu, 24 Jan 2019 00:00:00 GMT

The SoundCloud Xbox application is a regular React Redux application that runs in a native web view wrapper on Microsoft’s UWP framework. In order to allow users to play music in the background, we also ship a native C++ playback layer with the app. A thin API wrapper allows for communication between the web layer and the native layer.

Background audio is not the only reason for us to ship a native player; we also have to make sure that our application doesn’t use more than 128 MB when put into background mode. This is because applications with more than 128 MB can get killed by the Xbox application scheduler when it notices that the system or a game needs more memory. Our application shares the 128 MB with the web browser process, so the actual amount of memory someone is allowed to use is much lower than 128 MB.

The native player we’re using is the same one used in our mobile applications, which means it’s optimized for low memory consumption. Our initial tests, which we ran before we released the application, showed a low-enough memory usage of the application, and we were confident that we would not hit the maximum memory limit anytime soon.

The Memory Leak That Was Just a Cache

When we ran more memory tests later on, we noticed that our memory consumption for sessions with a use time of more than one hour was steadily growing and never shrinking. As a result, our hunt for a memory leak began. We quickly identified our Redux store as the part of our application that just kept growing, which makes complete sense, since Redux does not define explicit ways to remove parts of the state in order to save memory.

Just to be clear, I’m not blaming Redux here for clogging up our users’ memory! Memory consumption should not be your concern when you are building a web application, unless you have to work in an environment that is enforcing strict memory limits (like the Xbox application scheduler) or you are building an application that users are spending a lot of time in without reloading the page or closing the app (e.g. in the case of Electron apps).

Normalized State

Our Redux store is normalized, which means that we organize incoming data into its different type entities instead of keeping nested structures. Working with nested structures can be very tricky — e.g. when you want to propagate changes of an entity. Let’s look at an example:

{
  "playlists": [
    {
      "id": "p1",
      "tracks": [{ "id": "t1" }, { "id": "t2" }]
    },
    {
      "id": "p2",
      "tracks": [{ "id": "t1" }, { "id": "t3" }]
    }
  ]
}

The above code represents a user’s playlists. They have two playlists (p1 and p2) that each have two tracks. p1 contains tracks t1 and t2, and p2 contains tracks t1 and t3. If the user now wants to update the title of track t1, we have to iterate over all playlists and all tracks in those playlists, because track t1 might be in other playlists as well. This problem becomes even more complicated if you have other collections of tracks in your Redux store (e.g. a user’s liked tracks), because then you will have to iterate over all track collections in your store to make sure you’re updating the track in all places:

{
  "entities": {
    "playlists": {
      "p1": {},
      "p2": {}
    },
    "tracks": {
      "t1": {},
      "t2": {},
      "t3": {}
    }
  }
}

A normalized state like the one above allows for better change propagation because it a) removes duplicates, so you only ever have to change a single object, and b) merges entities from different requests into a single “entities” store so that you don’t have to worry about updating tracks from other collections. In this case, we only have to update the title of the object at entities.tracks.t1 and it will propagate correctly throughout our entire application.

Another great property of normalized state is that it becomes trivial to check if you already have a locally cached instance of, for example, a track you want to render. This can decrease the number of requests your application makes by a lot:

{
  "entities": {
    "playlists": {
      "p1": {},
      "p2": {}
    },
    "tracks": {}
  },
  "playlists": ["p1", "p2"]
}

In the example above, you can see that our playlist representation now only consists of an array of IDs that “point” to playlist entities.

If you are interested in learning more about normalization and how to add it to your application, I recommend checking out normalizr.

Garbage Collection in JavaScript

In the case of our problem, with every request we made, we were increasing the number of objects in our entities store, and we never removed objects, even if they were no longer needed. Usually in JavaScript, your data gets garbage collected once you remove all references to it (e.g. myState = null). In the case where all your data is encapsulated in your React components, garbage collection happens automatically when React unmounts your component. However, when you use Redux, the Redux store will always have a reference to all your data, so it will never get released.

Another way to free memory in JavaScript is to just reload the page every now and then when you detect that the session is already very long or when a lot of objects have accumulated in your entities store. Of course, this requires that your app be able to reload quickly so that users don’t get frustrated with your site’s performance. Additionally, you should make sure your users are not playing any media, because you really don’t want to interrupt them.

For our use case, we wanted to save memory for a background-listening experience, so we were not able to just reload the page. We came to the conclusion that we had to build a garbage collector on top of our Redux store to free memory efficiently. However, this is not the first time we’ve had to build a garbage collection algorithm into one of our applications. The main website at soundcloud.com has had a garbage collection system built into its model layer from the very beginning. But since the main site is built with Backbone.js, we could not reuse the garbage collecting implementation for the Xbox application.

Tracing Garbage Collection

When we looked at the different approaches to garbage collection, we saw a striking similarity between how tracing garbage collection works and the way Redux works in general.

A tracing garbage collector works in two steps (mark and sweep):

Mark — This marks objects that still have references.
Sweep — This scans all objects in memory and removes all the ones that are not marked.

If we were to implement a garbage collector in Redux this way, we would never be able to remove objects, because there would always be references to all objects. In order to make this concept work, we needed to be able to remove references, and as such, find a way to determine which parts of the state were no longer needed. If we could identify unused state, we could remove the state and all its referenced entities.

Redux provides, by design, all the tools necessary to identify used and unused state. Each component’s selector function (mapStateToProps) tells you which part of the state it needs, so we could use those to keep track of state that was no longer used. The only problem is that we didn’t have a list of all selectors currently onscreen, but that’s exactly where our garbage collector library comes in.

Our Redux Garbage Collector

The first thing our library does is keep track of selector functions that are currently onscreen. We do this by wrapping our component with a higher-order component that mimics React Redux’s connect API:

function (mapStateToProps) {
  return compose(
    connect(mapStateToProps),
    lifecycle({
      componentDidMount() {
        // Register onscreen selector
        registerSelector(mapStateToProps, this.props, this)
      },
      componentWillUnmount() {
        // Remove onscreen selector
        removeSelector(this)
      },
    })
  );
}

(I left the other parameters of connect out to shorten the example code.)

We are using Recompose to create a composed component that a) connects to the Redux store, and b) registers and removes the selector when the component mounts/unmounts:

let selectors = [];
function registerSelector(selectorFn, props, instance) {
  selectors.push({ selectorFn, props, instance });
}

In registerSelector, we add the selector to our list of onscreen selectors, along with a reference to the passed props and a reference to the component’s instance:

function removeSelector(instance) {
  selectors = selectors.filter(selector => selector.instance !== instance);
}

We need the reference to the instance of the component because we use it to remove a selector from the list. It’s important to mention that we need to check against the reference of the component and not against a reference of the selector itself. This is because we want to prevent removing a selector of the component that is going to unmount, regardless of the selector function. Another component — one that is still onscreen — might use the same selector function (by reference), but we want to keep its selector function. If we were to filter by the reference of the component’s selector function, we’d run the risk of removing selectors we still wanted to keep around.

Right now, we are keeping track of selectors of components that are onscreen. We can use this information to calculate the part of our component’s state that we can remove, but first we have to change our selectors slightly so that we can make sense of the data they return.

As described before, the bulk of our data is organized in the entities part of our Redux store, which is the part of the store we should remove unused objects from. All other parts of the store are only ID references or arrays of ID references, and they are not consuming that much data, so we can keep them.

However, one problem we’re facing now is that selector functions can return arbitrary shapes of data, and they might aggregate data in a way that does not represent the pure data of the Redux store. One example would be a component that renders the title of a track:

function (store, { trackId }) {
  return {
    title: store.entities.tracks[trackId].title
  };
}

The result of that function would be something like this:

{
  "title": "Redbone"
}

Just looking at this result, the garbage collector will not be able to determine which parts of the state are used to compute the title. In order to make sense of a selector’s result, we need it to somehow return its dependencies in addition to its regular result. This is why all garbage collectable selectors have to return data in the following shape:

{
  "result": {
    "title": "Redbone"
  },
  "dependencies": [
    {
      "entity": "track",
      "id": "t1"
    }
  ]
}

The data returned in dependencies gives us clear instructions on which type of entity and which instance of this entity this selector depends on. Now, returning this extra metadata from all of your components is pretty tedious, which is why we mainly use these selectors in specialized container components that abstract this extra overhead away for us:

<WithTrack trackId="t1">
  (track) => <TrackComponent track={track}>
</WithTrack>

In the above example, WithTrack is connected to the Redux store, and it makes sure to return the proper metadata. This removes a lot of the extra overhead created by the garbage collection library.

Taking Out the Garbage

The actual garbage collection is initiated by a regular Redux action. Let’s call it the sweep action since we are now in the sweep phase of the original algorithm:

function reducer(state, action) {
  if (action.type === SWEEP_ACTION_TYPE) {
    return sweep(state, action.selectors);
  } else {
    return combinedReducers(state, action);
  }
}

The sweep action is “intercepted” in our top-level reducer, which passes the action to our sweep reducer:

function sweep(oldState, selectors) {
  const entities = selectors
    .reduce(
      (dependencies, { selector, props }) =>
        dependencies.concat(selector(props).dependencies),
      []
    )
    .reduce(
      (newState, { entity, id }) =>
        (newState.entities[entity][id] = oldState.entities[entity][id]),
      {}
    );

  return {
    ...oldState,
    entities
  };
}

There are essentially three things happening in our sweep reducer.

We create a list of all dependencies by iterating over and evaluating all selectors. Here we make use of the stored props as well, because we need to pass them to each selector function in order to get the correct dependencies. When we’re done, we end up with an array that contains the dependencies of all components that are onscreen right now.
We start off with an empty object and then copy over each required entity from the oldState to the newState by iterating over the dependencies. The result of this is a fresh entities object that only consists of entities that are required by mounted components. This essentially performs the sweep we know from the algorithm, but instead of removing objects from the state, we actually started with a clean state and then added objects to it. Instead of mark & sweep, we basically execute a mark & keep.
We return the garbage collected state with the slimmed down entities object. Our store now only contains the objects that are necessary to render the current screen.

Determining when is best to start the garbage collection depends on the nature of your application. Some applications might just do it at a certain interval, some might want to check if they are using a lot of memory, and others might just execute every time the user is moved to a different major screen.

Performance

We have not observed a slowdown of our application due to our garbage collection process. Our initial thoughts were that components might rerender after garbage collection, but because the state references in the components are the same before and after garbage collection, components do not rerender. The only thing that differentiates garbage collection from any other Redux action is that it will evaluate all the selector functions on top of all reducers. If you keep your selector functions pure, you should not see a performance hit.

You might be thinking to yourself: “Wow, this would really make my application so much more efficient. I will add it right away!!!” Please, before you consider adding a garbage collection process to your application, check if you actually have problems with too-high memory usage. I would argue that 99 percent of Redux applications don’t require a custom garbage collection process, as they’re either too shortlived or they don’t have strict memory restrictions.

This system is working very well for us and we will keep using it on new applications when needed. We would love to know what you think about our approach if you have dealt with similar problems :)

Integration Testing for Memory Leaks

Fri, 23 Nov 2018 00:00:00 GMT

We write a lot of unit tests while working on the SoundCloud iOS application. Unit tests are obviously great. They’re short, they’re (hopefully) easy to read, and they give us confidence that the code we ship works as expected. But unit tests — as their name suggests — only tend to cover a single unit of code, most often a function or class. So how do we catch the bugs that exist in the interactions between classes — bugs like memory leaks?

Memory Leaks

Memory leaks in Swift can be tricky to catch. There’s the more obvious case of a non-weak delegate reference, but then there are those that are slightly more difficult to spot. Is it obvious, for example, that the following code could contain a memory leak?

final class UseCase {
    weak var delegate: UseCaseDelegate?
    private let service: Service

    init(service: Service) {
        self.service = service
    }

    func run() {
        service.makeRequest(handleResponse)
    }

    private func handleResponse(response: ServiceResponse) {
        // some business logic and then...
        delegate.operationDidComplete()
    }
}

Since the Service is injected, there are no guarantees about how it behaves. By passing in the private handleResponse function, which, although it isn’t necessarily immediately obvious, captures self, we are providing the Service with a strong reference to the UseCase. If the Service chooses to hold that reference — and we have no guarantee that it won’t — then we have a memory leak. But it’s not obvious from scanning the code that this could happen.

There’s already a great post by John Sundell on using unit tests to catch memory leaks in a single class. But with examples like the one above, which are so easy to miss, it’s not always clear that you need to write such a unit test. (We’re certainly not speaking from experience here. 😬)

Moreover, one of the signs of a good test suite is achieving the most valuable amount of coverage with as few – short! it doesn’t count if you put all your asserts in one giant test – tests as possible.

As Guilherme wrote in his recent post, new features in the SoundCloud iOS application are written following “clean architectural patterns” — most commonly some variation of VIPER. Most of these VIPER modules are constructed using what we call a ModuleFactory. This ModuleFactory takes some inputs — injected dependencies and configuration — and produces a UIViewController, which is already hooked up to the rest of the module and can be put onto the navigation stack.

Within a given VIPER module, there can be multiple delegates, observers, and escaping closures, each of which has the potential to cause the entire screen to stick around in memory after it’s been removed from the navigation stack. As this happens, the memory footprint will grow, and the operating system may well decide to terminate the application.

So is it possible to cover as many of these potential leaks as we can while writing as few unit tests as we can? If not, this entire setup has been a huge waste of time.

Integration Tests

The answer, as the title of this post might have suggested already, is yes. And we do this with integration testing. The goal of an integration test is to check how objects within a group interact with each other. Certainly our VIPER modules are groups of objects, and memory leaks are one form of interaction we definitely want to avoid.

Our plan is simple: We’re going to use our ModuleFactory to instantiate the entire VIPER module. We’ll then drop the reference to the UIViewController and make sure that all of the important parts of the module are destroyed alongside it.

The first problem we have is that, by design, we can’t easily access any part of our VIPER module aside from the UIViewController. The only public function on our ModuleFactory is func make() -> UIViewController. But what if we added another entry point just for our tests? This new method would be declared internal, so we’d only be able to access it by @testable import-ing the framework our ModuleFactory lives in. It would return references to all of the most important parts of the VIPER module, which we could then hold weak references to in our test. That ends up looking like this:

public final class ModuleFactory {
    // Some properties & init code, and then...

    public func make() -> UIViewController {
        makeAndExpose().view
    }

    typealias ModuleComponents = (
        view: UIViewController,
        presenter: Presenter,
        Interactor: Interactor
    )
    func makeAndExpose() -> ModuleComponents {
      // Set up code, and then...
      return (
          view: viewController,
          presenter: presenter,
          interactor: interactor
      )
    }
}

This solves the problem of not being able to directly access these objects. Obviously, this might not be perfect, but it suits our needs, so let’s move on to actually writing the test. Something like the following should work:

final class ModuleMemoryLeakTests: XCTestCase {
    // We need a strong reference to the view. Otherwise, the entire
    // module will be destroyed immediately.
    private var view: UIViewController?
    // However, we want to hold weak references to the
    // other parts of the stack, so as to mimic the behavior
    // of UIKit presenting our UIViewController onscreen.
    private weak var presenter: Presenter?
    private weak var interactor: Interactor?

    // In the setUp method, we instantiate the ModuleFactory and
    // call our makeAndExpose method. We need to make sure we
    // don't accidentally hold on to the returned ModuleComponents
    // tuple, as it contains strong references to all parts of the stack.
    // Doing so would clearly interfere negatively with the test.
    func setUp() {
        super.setUp()
        let moduleFactory = ModuleFactory(/* mocked dependencies & config */)
        let components = moduleFactory.makeAndExpose()
        view = components.view
        presenter = components.presenter
        interactor = components.interactor
    }

    // If the test passes, this tearDown step won't actually be necessary,
    // but we should definitely be covered in case the test fails, in order to stop
    // the tests themselves from leaking memory.
    func tearDown() {
        view = nil
        presenter = nil
        interactor = nil
        super.tearDown()
    }

    func test_module_doesNotLeakMemory() {
        // We start by ensuring all of the references are non-nil.
        // This is required to diagnose false positives, e.g.
        // if we failed to correctly assign the variables in the setUp stage.
        XCTAssertNotNil(presenter)
        XCTAssertNotNil(interactor)

        // Now we remove our strong reference to the view.
        // If everything is set up correctly, this is the only
        // strong reference to the entire stack, so everything
        // else should disappear alongside it.
        view = nil

        // Finally, we check that the weak references to the
        // Presenter and Interactor instances are now nil.
        // This ensures that these components, and any of their
        // own subcomponents, are free of memory leaks.
        XCTAssertNil(presenter)
        XCTAssertNil(interactor)
    }
}

So there we have it, a simple way to catch memory leaks throughout an entire VIPER module. It’s by no means perfect, and it requires some custom work for each new module we want to test, but it’s certainly a lot less work than writing individual unit tests for every possible memory leak. It also helps catch memory leaks you might not even realize are possible. In fact, after writing a few of these tests, we found we had one that we just couldn’t make pass, and after some investigation, we discovered a small memory leak in the module. (It bears repeating: 😬.)

This also gives us a starting point for writing a more general suite of integration tests for the module. After all, if we simply keep a strong reference to the Presenter and replace the UIViewController with a mock, then we can fake user input by calling methods on the presenter and make assertions on output by checking the mocked View. Maybe we’ll have more on that in a future post.

For now, let’s all celebrate this step toward a memory-leak-free future. 🙌

(n.b. This post was written while listening to Whatever You Love, You Are by Dirty Three, which you can listen to on SoundCloud.)

A Pragmatic Approach to Tech Debt Reduction

Fri, 23 Nov 2018 00:00:00 GMT

Almost every company accumulates tech debt as time goes on. Tight deadlines, changing requirements, scaling issues, poor or short-sighted system designs, knowledge silos, inconsistent coding practices, turnover of key staff — these things all happen and can contribute to tech debt. So what can be done about it once it’s there?

Background

My SoundCloud team, the Content Team, looks after a large number of legacy systems, along with a number of large systems that are difficult to understand and work with. Topics such as “Tech debt constantly increases development time” and “We have too much tech debt” come up frequently at our team retrospectives.

We have just completed a project that was almost exclusively about tech debt reduction. This post is a series of lessons learned about how to engage in this process (and when to decide not to).

How Much Tech Debt is Too Much?

The Tech Debt Tipping Point

A system with too much tech debt eventually becomes unmaintainable. Changes either take too long, are too difficult to test, or carry an unacceptably high risk of unintended consequences. The only way to extend the system is to add more tech debt. New feature development stops because the value gained is not proportional to the effort required.

Bad experiences with the system breed reluctance to work with the system at all, institutional knowledge of it gradually erodes, upgrades are postponed or simply don’t happen, and eventually someone puts a comment in the system’s README file that says something like “Never change this system again.”

Everyday Productivity Losses

Even for systems that haven’t reached the maintainability tipping point, tech debt can cause constant issues and increase development time, developer stress, and potential bugs.

A classic example of this problem is a test suite with one or more intermittently failing tests. Each individual case of the suite failing but then succeeding on rerun is minor, but the cumulative effect over weeks or months is far from minor. The time lost to reruns is an obvious consequence, but the erosion of confidence in the test suite can be even more damaging. True test failures will be hard to distinguish from the intermittent, “expected” failures, and the temptation to deploy with failing tests, or disable the test suite entirely, will grow.

Paying Down Tech Debt

We’ve seen above why tech debt is undesirable, but deciding what to do about it is not always easy. Here are a few possibilities.

Radical Action

Full Rewrite

Much has been said about “the allure of the full rewrite.” It’s tempting because it solves every problem the current system has — but at the expense of introducing every problem the new system will have. Note that the two lists of problems do not have to be mutually exclusive!

Starting over may well be the only option for systems that have passed the tech debt tipping point or for systems that are no longer fit for purpose because of fundamental design or scaling problems.

Starting over is also a large, all-or-nothing commitment to launching a new system. It will bring no value at all if the project fails, stalls, or is canceled.

Full Deletion

This is a common option of last resort for unreliable tests. Tests that don’t reliably report on correctness and run in a reasonable amount of time are not useful and may in fact be a hindrance. If rewriting the tests is not possible, not having the tests at all may well be preferable.

Needless to say, not having tests is generally not a good state of affairs to be in either.

Radical Inaction

A depressingly common approach is to do nothing and ignore the problem. This can stem from inexperience, a business strategy that demands new feature development at all costs, or simple wishful thinking. Regardless of the reason, persistent inaction tends to lead to the tech debt tipping point.

A Pragmatic Middle Way

First…

You Will Never Remove All the Tech Debt

This is an important thing to accept. No system is perfect, and attempts at perfection eventually reach a point where massive effort is necessary for very little gain. The goal should be a system that’s “good enough.”

But…

You Should Remove the Important Tech Debt

The definition of “good enough” will vary from system to system (see below), but tech debt that prevents the system from fulfilling this definition is important tech debt that needs paying down.

How To Identify the Important Tech Debt

It’s helpful to ask yourself the following questions:

How important is this system? What’s the business and operational impact of bugs, outages, or long development cycles?
Is this system still fundamentally fit for purpose? Do we foresee (or are we currently experiencing) significant scaling problems that this system cannot surmount in its current form?
Is this system easy to understand? Is it well documented?
How often do we need to change or maintain this system? How much staff time do we lose per week/month/quarter because this system is hard to work with?
Is this system reliant on external libraries or underlying technologies that are deprecated?
How many people are comfortable working with this system? Is the company overly reliant on a small number of key people, without whom this system might hit the tech debt tipping point?

Case Study: Converting a Microservice to a Batch Pipeline

One of my team’s services had a long history of tech debt-related problems. When we got a request for several new features, we believed adding these features would push the system perilously close to the tech debt tipping point. The system was designed in a way that put heavy, ever-increasing load on its database. We knew this load would eventually become unmanageable and the proposed new features would only hasten this process. The system needed either a substantial design change or a wholesale replacement. Our ideas for design changes were risky and complex, and we felt more confident of success with a wholesale replacement system. In other words, we felt radical action was justified in this case.

The replacement system is a batch pipeline that we built as a new component in a related system I’ll call Gulper. While the new component was essentially a greenfield project, it still had to integrate with other parts of Gulper. Gulper has not hit the tech debt tipping point, but as one of the team’s oldest and most complex systems, it is tech debt heavy and often hard to work with. In addition to adding the new component, we wanted to pay down some of Gulper’s tech debt in other areas, but we needed to balance that against our project deadlines.

Gulper’s already existing acceptance tests proved to be a particular pain point. They often failed spuriously, they were written in a language different than that in the main code base, they used external resources in a way that meant only one instance of the tests could run at a time, and they ran incredibly slowly. These tests run before every Gulper deploy. At times during our project, there were three people changing and deploying Gulper multiple times a day. The acceptance tests were costing us as much as ten or fifteen staff hours a week. The idea of full deletion of these tests did cross our minds at one point, but instead we committed to heavily refactoring the tests, using a contract-driven approach to uncouple test execution from shared external resources and mitigate spurious failures. Given the huge amounts of staff time the tests’ tech debt was causing, it was clear these tests were nowhere near “good enough.” So spending the staff time to make them “good enough” made perfect sense.

In contrast, we encountered a number of issues with Gulper’s data model that we decided to live with. Refactoring these would have been a major undertaking, and we didn’t believe the time spent would produce sufficient benefit, especially when weighed against our other priorities.

In a nutshell, this project involved some radical action in the form of a full rewrite of one system, some less radical action in the form of a heavy test suite refactor in another system, and some calculated inaction in the form of choosing not to refactor the data model issues.

Conclusions

Identifying important tech debt is not always easy — it takes practice and can be heavily subjective and situation-dependent as well — but paying down important tech debt now may very well keep your systems from hitting the tech debt tipping point later. Conversely, spending too much time paying down unimportant tech debt impedes your ability to deliver features in a timely manner and can be immensely frustrating. As with so many things in life, the key is finding balance.

How We Develop New Features Using Offsites and Clean Architecture

Fri, 02 Nov 2018 00:00:00 GMT

The current SoundCloud iOS App was built back in 2014, resulting in a huge codebase consisting of both Objective-C and Swift, with multiple architectural patterns and lots of legacy code. With time, the team changed and the platform evolved, so for the current iOS Team, maintenance, bug fixing, and implementing new features are all challenges.

In 2017, our team of six engineers wanted to try out a clean architectural pattern and decided to use VIPER.

I first read about VIPER four years ago in a post published by objc.io. This post is probably the most well-known guide to VIPER. At the time, most iOS applications were using Apple’s Model View Controller (MVC) pattern. This pattern is well suited for relatively small teams where there is only one developer working on one of the components (the Model, the View, or the Controller) at a time. VIPER, on the other hand, is good for larger teams and for parallelizing work among them, particularly when an app is big and complex, which the SoundCloud app is.

In May 2017 when I joined SoundCloud, I was excited to improve the massive codebase with my previous learnings. Prior to starting at SoundCloud, I’d used VIPER on a project written in Swift in hopes of avoiding the obstacles of MVC. Overall, I had a really positive experience and I was looking forward to exploring it more, and my work at SoundCloud provided the perfect opportunity to do this.

Because I was the only one on the team familiar with VIPER, we organized a one-day workshop where I explained the VIPER pattern to the team and we rewrote an existing screen (the System Playlist) with it. The feedback from the team was quite positive and everyone was excited to start using it. As a result, we went on to write the new Mini Player feature using VIPER, and because it was successful, we’ve since used VIPER to write additional features.

Offsites

Every time we set out to write a new feature, we begin with a project kickoff, which is where we define what we are doing and why. Only two engineers, the project’s point people, are present for the kickoff. Part of their responsibility is to organize an offsite to show the feature to the team and brainstorm on the implementation.

I always liked how hackathons are done. The fast-paced environment, the chaos, the freedom, the focus on the problem without any interruption, and the brainstorming with the people in the room are all things that enable productivity. We can quickly solve a problem and accomplish a shared goal in a much shorter period, all while having fun at the same time.

Offsites remind me of hackathons because they consist of a full day at another location, without any meetings and with food and drinks. In other words, they’re different than the usual work day. Working toward a goal together, all while having fun, helps with knowledge sharing and team bonding.

The day starts with an explanation of the feature in question and the UX, followed up by questions to clarify the functionalities. The intention here is to have everyone on the same page without any misunderstandings. Once we have enough information, we start on the Contract for the new feature.

For us, the Contract is the most important file, and it’s where all protocols of the VIPER module are defined. This provides a quick and easy overview of the behavior of the feature — as you can see here — and helps us organize our ideas.

Example

As an example, here is how we might build a generic feature. It assumes some familiarity with VIPER, but for more details on the way we at SoundCloud use VIPER, see here.

We start with a simple application that shows a list of tracks, as seen in the image below.

As we can see:

The user sees a list of tracks.
The user can add a track.
The user can delete a track.

Contract

We tend to follow a specific order to create the Contract (Protocols). This doesn’t mean that a different order would be worse; this is just the way the conversation usually flows.

Presenting

We start with the Presenting protocol, which handles UI events as follows.

This will probably call the Interactor to fetch the data:

viewDidLoad()

This will probably go to the Router to show another screen or call the Interactor to immediately add a new value:

didTapAdd(title: String, artist: String)

This will receive the index of the row and find which track should be deleted. It will probably call the Interactor to delete the track:

didSwipeToDelete(at row: Int)

Interacting

Next, we develop the Interacting protocol because the UI interactions are already defined in the Presenting protocol.

We found that using delegate over blocks between the components is clearer and more organized, given we often call the same delegate in multiple methods.

This is how the Interacting protocol will look:

fetchTracks()
addTrack(title: String, artist: String)
delete(track: Track)

InteractorDelegate

It is time to go back to the View. So next is the InteractorDelegate, which is implemented by the Presenter. You can expect something like this:

didFetch(tracks: [Track])
handle(error: ServiceError)

Viewing

The Viewing protocol comes next, with the following:

update(viewModels: [TrackViewModel])

The above receives an array of ViewModels to be displayed in the TableView. Usually, when we delete something, we call the same method with the items updated. If necessary, we can create a method to update a specific item at the index.

show(error: String)

The above receives a ViewModel or a String to be shown.

showLoadingIndicator()
hideLoadingIndicator()

Routing

Let’s assume that tapping on a track will direct to another screen with more detail.

The Routing protocol will just present the detail screen, like so:

presentDetailScreen()

It will get another Builder to retrieve the module and to use the weak ViewController to show the next screen.

Hands On

Before we start coding, we select the most challenging parts to work on. Depending on the project, it could be a UI interaction, or it could be the decoupling of a class, a service, or even some of the VIPER components.

We split the team in groups, with each group trying to solve the problem that is given to them. Sometimes it is just a spike that will be performed on a different project, but sometimes we will work on the main application together.

If the feature is straightforward, each group gets one of the VIPER parts. Then we start branching from a feature branch in which the Contract will be updated.

As time goes on, we continue to communicate with one another, helping with the most difficult parts and explaining architectural decisions, which makes the team more aware of the entire development process. Thanks to a different environment without interruptions, we can have these discussions without disturbing other colleagues.

By the end of the day, we have a rough idea of how our feature is going to behave. Through this process, we are essentially creating a bridge, wherein each team is constructing from a different place and hoping that in the end, all the pieces fit together.

What we want to achieve on this day is to identify the possible problems we might have and try to solve or propose solutions while we are together and focused. As a bonus, we may have some of the components done by the end of the day, like a Presenter or Interactor.

After the Offsite

After having an offsite, it’s easier to create user stories and estimate and define what needs to be done. To date, the iOS team at SoundCloud has had four offsites. Each one had different outputs and sizes, but all of them helped us share knowledge and brainstorm problems that could have been difficult to solve later.

Having a clean architecture has given us the possibility of splitting the work, which makes it easier to have six people working on the same feature. It also helps improve awareness of the areas that needed to be covered, thereby reducing unexpected issues.

As a bonus, we discovered that having everybody in a different working environment with one goal in mind helped us discover problems earlier, improved our shared knowledge and ownership of a feature, and bonded the team.

We are looking forward to building more features in this format and seeing what we’ll learn from it. We’ve found that every time we do it, our process evolves, we learn new things, and we’re able to refine our workflow. We hope that over time we can decrease assumptions, have more consistent estimations, reduce project risk, and be able to ship features with more confidence.

Using Kubernetes Pod Metadata to Improve Zipkin Traces

Wed, 19 Sep 2018 00:00:00 GMT

SoundCloud is built on hundreds of microservices. This creates many challenges, among them debugging latency issues across the services engaged in the completion of a single request. As such, it’s natural that we were early adopters of Zipkin, a distributed tracing tool that emerged from Twitter, which was built in part to help engineers tackle these sorts of problems.

In recent years, we’ve switched to using Kubernetes as our container deployment platform. Kubernetes makes it easy to break a microservice down into narrowly focused operational responsibilities, which often correspond to different workloads the service must handle and are typically expressed in Kubernetes as distinct pod templates.

Because each Kubernetes pod is assigned its own IP address, and because each pod has a well-defined responsibility, it’s possible to make higher-level assumptions about an IP address given the easy access to data that Kubernetes accumulates across deploys. This is in contrast to the pre-Kubernetes era at SoundCloud, when an IP address referring to a “bare-metal” machine would likely have hosted several very different services.

The BEEP team¹ here at SoundCloud recently revisited our Zipkin setup, primarily to make two improvements: to increase the quality and accuracy of service names in Zipkin, and to enhance Zipkin span data.

Updating Zipkin service names across our hundreds of services in concert with a constantly evolving backend architecture has been a particular challenge. This is because Zipkin uses service names to represent how one service relates to another, and without good, consistent service names, the value of Zipkin as a visualization or debugging tool is greatly reduced.

As microservices combine to process a single overall request, they may (on a sampling basis) also report trace data to Zipkin. A trace identifier is established for an overall request and propagated as one microservice calls out to others. The caller microservice sends the remote service name, endpoint name, call timing, and trace ID to Zipkin, which Zipkin turns into a span. Spans with the same trace ID are woven together to form a Zipkin trace. See also Zipkin: Instrumenting a library.

Defining Zipkin Service Names in One Central Place

In a service-to-service interaction, the caller service reports the IP address of the callee service’s pod as part of the span data it sends to Zipkin. After brainstorming, we realized we could customize the Zipkin collector to transform the span on the fly, performing a quick reverse IP lookup of the Kubernetes pod metadata and replacing the caller’s version of the Zipkin service name with one assembled from the callee’s pod labeling:

Assembling a Zipkin service name based on Kubernetes pod metadata.

The result was an across-the-board improvement in both Zipkin service names and dependency graph quality.

As a side effect of this approach, control over how a node is represented in Zipkin switched from the caller services to the callee service:

When callers control naming, the same callee may end up represented as several different services.

An alternative: all names come from reverse IP lookup against a centralized index.

The team that owns the locale-info service makes the decision, for all teams, how that service is named, thus removing the need for multiple teams to coordinate on establishing or changing the service name. And the original Zipkin service name given by the caller is not lost — we add it as a span annotation.

Useful Span Annotations from Pod Metadata

Zipkin span collection generates significant reverse-lookup load, but our integration hits a simple, very fast Kubernetes metadata caching service written by the SoundCloud Production Engineering team. Responses from this service contain a rich set of pod metadata that goes beyond what’s merely necessary to assemble a good Zipkin service name, and we add all of this data as extra annotations on the Zipkin span.

The result is that when looking into a latency problem, we can not only see basic Zipkin information about how long a service took to respond, but we also know details like the Git revision of the service code that was running on the particular pod, because we consistently set that as a Kubernetes label on all pods. This is a level up in an engineer’s ability to efficiently pinpoint production problems.

An example Zipkin span with Kubernetes-derived annotations.

Our approach involves inserting a small amount of logic into the Zipkin collector and then leveraging our internal Kubernetes pod labeling scheme to improve span data. The result is normalized and modernized service names, along with greatly enhanced details embedded in Zipkin traces that engineers can use to troubleshoot production issues.

Value-added span transformation ideas are getting more attention in the Zipkin community. The OpenZipkin team has a collector extension framework in the works, and we’re excited to see what other Zipkin “mashups” come out of the Zipkin community. If you’re interested, please join the OpenZipkin gitter discussion.

Thank You

Benjamin Debeerst, SoundCloud Backend Engineering Productivity Team
Kristof Adriaenssens, Engineering Manager, SoundCloud Content Team
René Treffer, SoundCloud Production Engineering Team
Tobias Schmidt, SoundCloud Production Engineering Team
Adrian Cole, Pivotal Spring Cloud Team and prominent Zipkin contributor

Additional Reading

1: The Backend Engineering Productivity team’s mission is to help solve common, cross-cutting engineering problems that our product-focused engineers face. We also build and maintain many of SoundCloud’s “core” services.

Hands-Off Deployment with Canary

Wed, 29 Aug 2018 00:00:00 GMT

At SoundCloud, we follow best practices around continuous delivery, i.e. deploying small incremental changes often (many times a day). In order to do this confidently and reliably, we verify these changes before releasing, using different measures including, among others, unit, integration, and contract tests.

This process is automated as part of a continuous delivery (CD) pipeline. Historically, a typical CD pipeline here has contained at least the following steps:

After building and packaging the code, a battery of automated tests was run in order to verify the expected behavior. Once all the tests passed, deployment to production could be manually triggered.

This automated process, together with code reviews and other internal measures, greatly reduced the risk of introducing regressions.

However, even with precautions like those mentioned above in place, things eventually can (and will) go wrong. In order to detect production issues, we use monitoring and alerting, so when a problem is found in production, we can react by reverting to a well-known previous state. Even so, since the change has already been fully rolled out, the user experience has already been severely impacted.

In order to improve the user experience, we have been exploring different ways of reducing the impact and the Mean Time to Recovery (MTTR) of faulty deployments. Enter canary releases…

Iteration 1: Manual Canary Release

The term “canary” originates from a mining tradition from the early 20th century, consisting of canaries being brought down to coal mines in order to detect toxic gases. When a canary died, it alerted workers to carbon monoxide or other poisons in the air, allowing them to evacuate before they were affected.

In the context of software development, canary release is a technique that consists of making release candidates available to a small subset of users in order to detect regressions before the changes are available to all users.

At SoundCloud, we initially implemented this technique by building tooling to allow packaging and publishing release candidates into a canary environment and configuring load balancers to divert a small amount of production traffic to it. By comparing metrics between production and canary environments, our developers are able to identify faulty release candidates before they make it to production.

This was very convenient, but it came with some disadvantages. As it was a manual process and not integrated in the CD pipelines, it was easy for developers to forget to turn off old release candidates. Multiple versions receiving production traffic sometimes caused confusion and hard-to-debug issues. And although the process was documented, not everyone in the company was aware of the possibility of using canary environments. As a tedious manual process needed to be followed, it was only used for a very small number of releases. Finally, both the lack of traceability and allowing any developer to put arbitrary code into production without following standard quality checks came with a set of security and safety concerns.

Iteration 2: Canary Pipeline Integration

A natural next step was to integrate canaries into our deployment pipeline, thereby automating the process and ensuring it was used for every release. This also ensured that any code receiving production traffic had passed our standard quality processes and that the release candidate running in the canary environment was up to date.

With these changes in effect, the pipeline looked like this:

Every change went through a canary step in which developers could manually compare metrics and identify issues before proceeding to a full production deploy.

This was a clear improvement over the previous situation, but it was still a manual process that relied on developers carefully looking at relevant graphs and spotting potential problems, which was time-consuming and error prone.

Iteration 3: Automated Canary Verification

In order to fully automate our deployments, we had to solve the problem of automatic canary verification. We soon realized we could leverage our existing monitoring and alerting infrastructure to accomplish just that.

Every service at SoundCloud has a set of defined rules used to detect anomalies and undesired behaviors. With small modifications to the rule expressions, we were able to detect these conditions in each of our defined environments.

The missing piece was to query our monitoring system from the CD pipelines to automatically detect and roll back any change introducing regressions. This allowed us to remove manual intervention while ensuring the same standards of quality used for our production environments.

The resulting pipeline looked like this:

Conclusion

The addition of automatic canary deployments and verifications at SoundCloud was a massive success and has greatly improved our productivity and reliability. The confidence we have in automatically detecting any regression has removed the manual toil around deployment and enables our developers to focus on adding value without spending time babysitting deployments. When things inevitably go wrong, we now have yet another safety net that protects us from large-scale outages. Using automatic processes and removing human interaction to verify and roll back faulty deployments has greatly reduced our Mean Time to Detect (MTTD) and Mean Time to Recovery (MTTR).

Creating Readable Spark Jobs

Mon, 06 Aug 2018 00:00:00 GMT

Nowadays, it’s rather common to encounter Apache Spark being utilized in a lot of companies that need to process huge amounts of data, and things aren’t any different here at SoundCloud — as one can imagine, we have lots of data to process all the time.

While Spark does offer a streaming API, along with other tools like Kafka Streams and Apache Flink, we mainly (but not exclusively) use batch processing with Spark to solve a variety of problems, such as:

The creation of some partners’ reports
Machine learning
Datasets
Data (re)processing pipeline

While Spark’s Scala API could be more idiomatic and type safe, it does offer a nice DSL that allows one to create expressive pipelines. However, it’s also not uncommon to see these jobs getting a little bit out of control in terms of readability, with many nested operations. This can lead to situations where one needs to spend some time reasoning about what’s going on and what types are being passed through all of those intermediate operators.

In this article, we’ll see how to break down the aforementioned types of jobs into smaller chunks and how to compose them using functional programming techniques. In order to illustrate this approach, let’s go through a simplified use case of a real system our team is responsible for. Even though the example illustrated here is implemented using Spark, these techniques could be employed with other technologies as well, like Scrunch.

Practical Example

One of the things the Content Team here at SoundCloud is responsible for is receiving content from labels and making it available to users. As part of this process, our systems need to decide which SoundCloud profile each track should be uploaded to.

The way this decision process works is based on rules that are matched against specific parts of the metadata of each track. For our simplified use case, we have two rule types:

One matching both artist and label names
One based only on the artist name

We’ll assume there are no conflicting rules, i.e. there will be no two rules matching the same label/artist or artist, and the rules matching both artist and label names have higher priority than the ones matching only the artist name. Of course, things are more complex in the real world, but this should be good enough for our example. Before getting into the job itself, let’s take a look at the model:

case class Label(id: Long, name: String)

case class Artist(id: Long, name: String)

case class Track(id: Long, title: String, artistId: Long, labelId: Long)

case class RuleType(name: String)

object RuleType {
  val Artist = RuleType("artist")
  val ArtistAndLabel = RuleType("artist-and-label")
}

trait MappingRule {
  def id: Long
  def ruleType: RuleType
  def userId: Long
}

case class MappingRuleArtistAndLabel(id: Long, artist: String, label: String, userId: Long) extends MappingRule {
  override def ruleType: RuleType = RuleType.ArtistAndLabel
}

case class MappingRuleArtist(id: Long, artist: String, userId: Long) extends MappingRule {
  override def ruleType: RuleType = RuleType.Artist
}

case class AppliedRule(id: Long, ruleType: RuleType, userId: Long)

case class TrackWithAppliedRule(id: Long, title: String, artist: Artist, label: Label, maybeAppliedRule: Option[AppliedRule])

A few things could be better in this model:

We could use value classes to wrap the primitive types like IDs and names.
The RuleType could be defined as a sealed trait with case objects implementing it.
The TrackWithAppliedRule could have a field of type MappingRule instead, thereby removing the need for the AppliedRule class.

Unfortunately, some of these ideas are rather difficult to implement with vanilla Spark. One approach to overcome these issues is by using Frameless, but that would be a topic for another blog post, so let’s leave the model as it is.

Moving forward, the input to the job will be five datasets:

One with the tracks
One with the artists
One with the labels
One with the artists’/labels’ rules
Another one with the artists’ rules

The job basically needs to:

Join tracks with artists and labels
Apply the artists’/labels’ rules
Apply the artists’ rules
Select the best rule and return a dataset of TrackWithAppliedRule

Starting with the Tests

Below is a single test case that can help with better understanding the logic and that will also help with the development and refactoring of the job:

import org.apache.spark.sql.SparkSession
import org.specs2.mutable.Specification
import org.specs2.specification.Scope

class RuleApplierJobSpec extends Specification {

  trait Context extends Scope {
  val spark = SparkSession
    .builder()
    .appName("tracks-rules-applier")
    .master("local")
    .config("spark.sql.shuffle.partitions", "1")
    .getOrCreate()
  val job = new RuleApplierJob(spark)

  val label1 = Label(id = 1, name = "Label 1")

  val artist1 = Artist(id = 1, name = "Artist 1")
  val artist2 = Artist(id = 2, name = "Artist 2")
  val artist3 = Artist(id = 3, name = "Artist 3")

  val ruleArtist1AndLabel1 = MappingRuleArtistAndLabel(id = 1, artist = "Artist 1", label = "Label 1", userId = 1)
  val ruleArtist1 = MappingRuleArtist(id = 2, artist = "Artist 1", userId = 2)
  val ruleArtist2 = MappingRuleArtist(id = 3, artist = "Artist 2", userId = 3)

  val track1 = Track(id = 1, title = "Title Track 1", artistId = 1, labelId = 1)
  val track2 = Track(id = 2, title = "Title Track 2", artistId = 2, labelId = 1)
  val track3 = Track(id = 3, title = "Title Track 3", artistId = 3, labelId = 1)
  }

  "it should match rules against artist and label of each track" in new Context {
  import spark.implicits._

  val labels = Seq(label1).toDS()
  val artists = Seq(artist1, artist2, artist3).toDS()
  val artistsAndLabelsRules = Seq(ruleArtist1AndLabel1).toDS()
  val artistsRules = Seq(ruleArtist1, ruleArtist2).toDS()
  val tracks = Seq(track1, track2, track3).toDS()

  val result = job.run(tracks, artists, labels, artistsAndLabelsRules, artistsRules).collect().toSeq

  result should containTheSameElementsAs(Seq(
    TrackWithAppliedRule(1, "Title Track 1", artist1, label1, Some(AppliedRule(1, RuleType.ArtistAndLabel, 1))),
      TrackWithAppliedRule(2, "Title Track 2", artist2, label1, Some(AppliedRule(3, RuleType.Artist, 3))),
      TrackWithAppliedRule(3, "Title Track 3", artist3, label1, None)
  ))
  }
}

Working and Iterating on the Job

The first version of the job works and gets the tests passing, but it’s not that readable — especially if you consider people joining a team and having to understand and maintain the job:

import org.apache.spark.sql.{Dataset, SparkSession}

class RuleApplierJobV1(spark: SparkSession) {

  import spark.implicits._

  def run(tracks: Dataset[Track],
          artists: Dataset[Artist],
          labels: Dataset[Label],
          artistsAndLabelsRules: Dataset[MappingRuleArtistAndLabel],
          artistRules: Dataset[MappingRuleArtist]): Dataset[TrackWithAppliedRule] =
  tracks
    .joinWith(artists, tracks("artistId") === artists("id"), "inner")
    .joinWith(labels, $"_1.labelId" === labels("id"), "inner")
    .map { case ((track, artist), label) => (track, artist, label) }
    .joinWith(artistsAndLabelsRules, $"_2.name" === artistsAndLabelsRules("artist") &&
      $"_3.name" === artistsAndLabelsRules("label"), "left_outer")
      .map { case ((track, artist, label), artistAndLabelRule) => (track, artist, label, Option(artistAndLabelRule)) }
    .joinWith(artistRules, $"_2.name" === artistRules("artist"), "left_outer")
    .map { case ((track, artist, label, maybeArtistAndLabelRule), ruleArtist) => (track, artist, label, maybeArtistAndLabelRule, Option(ruleArtist)) }
    .map { case (track, artist, label, maybeArtistAndLabelRule, maybeArtistRule) =>
      TrackWithAppliedRule(
        track.id,
          track.title,
          artist,
          label,
          maybeArtistAndLabelRule.orElse(maybeArtistRule).map(r => AppliedRule(r.id, r.ruleType, r.userId)))
    }
}

A first approach to make it better is a very basic technique in programming: breaking a big problem into smaller pieces. We can do that by creating smaller functions, each one doing one part of the work:

import org.apache.spark.sql.{Dataset, SparkSession}

class RuleApplierJob(spark: SparkSession) {

  import RuleApplierJob._

  def run(tracks: Dataset[Track],
          artists: Dataset[Artist],
          labels: Dataset[Label],
          artistsAndLabelsRules: Dataset[MappingRuleArtistAndLabel],
          artistRules: Dataset[MappingRuleArtist]): Dataset[TrackWithAppliedRule] = {
  val tracksWithArtists = joinTracksWithArtists(tracks, artists)
  val tracksWithArtistsAndLabels = joinTracksAndArtistsWithLabels(tracksWithArtists, labels)(spark)
  val withArtistsAndLabelsRules = applyArtistsAndLabelsRules(tracksWithArtistsAndLabels, artistsAndLabelsRules)(spark)
  val withArtistsRules = applyArtistRules(withArtistsAndLabelsRules, artistRules)(spark)
  convertToTrackWithAppliedRule(withArtistsRules)(spark)
  }
}

object RuleApplierJob {

  private def joinTracksWithArtists(tracks: Dataset[Track], artists: Dataset[Artist]) =
  tracks.joinWith(artists, tracks("artistId") === artists("id"), "inner")

  private def joinTracksAndArtistsWithLabels(tracksWithArtists: Dataset[(Track, Artist)], labels: Dataset[Label])(spark: SparkSession) = {
  import spark.implicits._
  tracksWithArtists.joinWith(labels, $"_1.labelId" === labels("id"), "inner")
    .map { case ((track, artist), label) => (track, artist, label) }
  }

  private def applyArtistsAndLabelsRules(tracksWithArtistsAndLabels: Dataset[(Track, Artist, Label)], artistAndLabelRules: Dataset[MappingRuleArtistAndLabel])(spark: SparkSession) = {
  import spark.implicits._
  tracksWithArtistsAndLabels
    .joinWith(artistAndLabelRules, tracksWithArtistsAndLabels("_2.name") === artistAndLabelRules("artist") &&
      tracksWithArtistsAndLabels("_3.name") === artistAndLabelRules("label"), "left_outer")
    .map { case ((track, artist, label), artistAndLabelRule) => (track, artist, label, Option(artistAndLabelRule), Option.empty[MappingRuleArtist]) }
  }

  private def applyArtistRules(tracksWithRules: Dataset[(Track, Artist, Label, Option[MappingRuleArtistAndLabel], Option[MappingRuleArtist])], artistRules: Dataset[MappingRuleArtist])(spark: SparkSession) = {
  import spark.implicits._
  tracksWithRules
    .joinWith(artistRules, tracksWithRules("_2.name") === artistRules("artist"), "left_outer")
    .map { case ((track, artist, label, maybeArtistAndLabelRule, _), ruleArtist) => (track, artist, label, maybeArtistAndLabelRule, Option(ruleArtist)) }
  }

  private def convertToTrackWithAppliedRule(tracksWithRules: Dataset[(Track, Artist, Label, Option[MappingRuleArtistAndLabel], Option[MappingRuleArtist])])(spark: SparkSession) = {
  import spark.implicits._
  tracksWithRules.map { case (track, artist, label, maybeArtistAndLabelRule, maybeArtistRule) =>
    TrackWithAppliedRule(
      track.id,
        track.title,
        artist,
        label,
        maybeArtistAndLabelRule.orElse(maybeArtistRule).map(r => AppliedRule(r.id, r.ruleType, r.userId)))
  }
  }
}

If you want to avoid passing around these gigantic dataset types with lots of tuples, there are basically two approaches to deal with these types of intermediate state:

Defining basic alias types
Wrapping them in case classes

In our case, let’s go with the first approach and define some alias types to pass around:

type TracksWithArtists = Dataset[(Track, Artist)]
type TracksWithArtistsAndLabels = Dataset[(Track, Artist, Label)]
type TracksWithRules = Dataset[(Track, Artist, Label, Option[MappingRuleArtistAndLabel], Option[MappingRuleArtist])]

There are still some nuances in this code that are worth elaborating on.

All of these small functions need to be defined in the companion object instead of in the class itself. The reason is simple: If they are defined in the class, it will cause Spark to try to serialize the entire class, and that will lead to the infamous TaskNotSerializableException once the SparkSession object is not serializable.
Because Spark needs default encoders provided as implicit parameters by the SparkSession instance, the SparkSession needs to be passed to pretty much all of the internal functions.

Another point that plays on our side to help with the refactoring is the fact that the types of each one of the functions align perfectly (with exception of the SparkSession), i.e. the output of one function is the input of the next one:

So we have a scenario that function composition is almost 100 percent a good fit, except for the annoyance of also having to pass the SparkSession around to almost all the functions. Having said that, it looks like this situation would allow the creation of a pipeline of functions if we could somehow inject the SparkSession dependency when we want to execute it instead of providing it to each function call. This seems to be a good fit for the Reader Monad pattern:

  "The Reader monad represents a computation, which can read values from a
  shared environment, pass values from function to function, and execute
  sub-computations in a modified environment."

This is also a perfect fit for our scenario, as each function needs to read a value (SparkSession) from a shared environment in order to perform its computation and return its value. Both Scalaz and Cats provide implementations of the Reader Monad. In our case, we’ll be using Cats, as it’s already part of the project we were working on, but the implementation would be quite similar with Scalaz. Here’s how the final version would look:

import cats.data.Reader
import org.apache.spark.sql.{Dataset, SparkSession}

class RuleApplierJob(spark: SparkSession) {

  import RuleApplierJob._

  def run(tracks: Dataset[Track],
          artists: Dataset[Artist],
          labels: Dataset[Label],
          artistsAndLabelsRules: Dataset[MappingRuleArtistAndLabel],
          artistRules: Dataset[MappingRuleArtist]): Dataset[TrackWithAppliedRule] =
  joinTracksWithArtists(tracks, artists)
    .flatMap(joinTracksAndArtistsWithLabels(_, labels))
    .flatMap(applyArtistsAndLabelsRules(_, artistsAndLabelsRules))
    .flatMap(applyArtistRules(_, artistRules))
    .flatMap(convertToTrackWithAppliedRule)
    .run(spark)
}

object RuleApplierJob {

  type TracksWithArtists = Dataset[(Track, Artist)]
  type TracksWithArtistsAndLabels = Dataset[(Track, Artist, Label)]
  type TracksWithRules = Dataset[(Track, Artist, Label, Option[MappingRuleArtistAndLabel], Option[MappingRuleArtist])]

  private def joinTracksWithArtists(tracks: Dataset[Track], artists: Dataset[Artist]) = Reader[SparkSession, TracksWithArtists] { _ =>
    // same as previous listing
  }

  private def joinTracksAndArtistsWithLabels(tracksWithArtists: TracksWithArtists, labels: Dataset[Label]) = Reader[SparkSession, TracksWithArtistsAndLabels] { spark =>
    // same as previous listing
  }

  private def applyArtistsAndLabelsRules(tracksWithArtistsAndLabels: TracksWithArtistsAndLabels, artistAndLabelRules: Dataset[MappingRuleArtistAndLabel]) = Reader[SparkSession, TracksWithRules] { spark =>
    // same as previous listing
  }

  private def applyArtistRules(tracksWithRules: TracksWithRules, artistRules: Dataset[MappingRuleArtist]) = Reader[SparkSession, TracksWithRules] { spark =>
    // same as previous listing
  }

  private def convertToTrackWithAppliedRule(tracksWithRules: TracksWithRules) = Reader[SparkSession, Dataset[TrackWithAppliedRule]] { spark =>
    // same as previous listing
  }
}

As seen above, instead of each function returning only its desired type, it returns a Reader that produces a type given a SparkSession. Reader Monads are composable, so the functions can be aligned after each other, thereby creating a pipeline of functions, and only when this pipeline is triggered does the external dependency (SparkSession in this case) need to be provided. The code above is much cleaner and easier to reason about! Even a newcomer to this codebase would be able to easily understand what’s going on here.

Another approach to compose the functions would be using a for comprehension instead of flatMap. Both approaches work and are quite similar, with the disadvantage that in this specific use case, one will need to give names to the intermediate values in the for comprehension solution.

For completeness, it’s worth mentioning that implicit parameters could also be used to pass the SparkSession around, but I find that less readable and a little bit harder to reason about, as the dependencies are not very clear. Having said that, it might be a better approach in some scenarios, such as that of an external library, and it’s also a matter of preference/style.

For the sake of simplicity, I decided to keep all the functions inside the same companion object, but you could also create a dedicated object for each of them instead, and the solution would end up being pretty much the same.

Conclusion

As companies write more and more Spark code in Scala, it’s important that we strive to bring more functional aspects to this codebase. In addition to looking for opportunities to apply these concepts as shown in this article, frameworks like Frameless are also working toward creating more expressive DSLs to work with Spark, so it’s worth keeping an eye on those as well.

Getting a Team Back on Track

Wed, 25 Jul 2018 00:00:00 GMT

Sometimes, an important team that’s part of an otherwise healthy company culture starts tanking and the people on the team get frustrated and even quit.

In this article, I want to share what I learned when I started to manage a team — referred to as the R Team from here on out — that had huge problems when I took over as Engineering Manager, as well as explain how I got it back on track.

I hope this post raises your awareness of some of the issues that can affect a team and that you will be able to apply them too — whether you’re a manager, an individual contributor, or just observing a team from the outside.

The TL;DR

Operating at the throughput limit of a team will tear it apart.
Technical challenges and business pressure are not the problem. The problem is a lack of focus and collaboration.
Limiting parallel work streams will increase productivity, team happiness, and stakeholder trust.

The Beginning

The R Team was always under a lot of pressure but somehow coped with it. But then, all of a sudden, the team seriously tanked. In a very short time, one of the engineers quit and the team’s Engineering Manager went on a break and later left the company. In the meantime, two of the team’s five engineers handed in their notices. This left the team with only two engineers, and one of them was also entertaining the thought of moving on.

On the product management side, it was similar. One Product Manager left, and the next one stayed for only one month. The team was behind on everything, and the backlog was infinite. Work was no fun.

What Was Broken?

Most of the time, issues piling up is what can turn a team of great and motivated engineers into a frustrated bunch of individuals. But in order to understand the full picture and to come up with a plan to recover, it is important to put everything on the table. Within the R Team, there were a few very obvious problems that we identified right away:

A huge amount of tech debt
A large group of stakeholders, each of them asking the team to work on equally high-priority tasks
A high amount of domain complexity

To put this into perspective, the amount of tech debt and domain complexity made onboarding new team members — a process that should normally take no longer than a month — a long and intense three-month journey. And because of being pressed for time, there was never time to clean up tech debt. For instance, the oldest of the team’s data pipelines was still in production alongside two generations of rewrites. The same was true for other systems that had multiple generations of rewrites in production. The backlog stretched for years.

It was a lot to deal with, but even under such arduous circumstances, a motivated team could take all this as a challenge and work hard to catch up. So why didn’t this team do this? Why did the members instead feel like giving up? To find out the answer to this, I had to do some digging.

After continued talks and investigation, mostly in one-on-ones, more and more issues became clear. One was that in order to cope with the competing high-priority deliverables, the team parallelized workstreams, which made the members function more like a group of specialists working on separate things as opposed to a team with a shared goal. For example, the team often had more than one epic per person in the sprint as a way of keeping all high-priority tasks going. As a side effect, this pretty much prevented effective knowledge sharing. Indeed, in order to get a complete overview of the tech stack, all team members, each with individual special knowledge, were needed to give their input.

The parallelization also delayed shipping, which then eroded trust from stakeholders. To make matters worse, the business expert who was needed to clarify technical specifications during implementation was overwhelmed by repetitive, manual work. This not only made his job tedious and labor-intensive, but it also made the feedback cycle on technical questions very long.

In the middle of all of this, there was a big change in the organizational structure sitting above the team, and the manager of the team’s new group started up a parallel workstream to review alternative approaches to reducing the tech debt, ultimately replacing the team’s existing technical vision.

Although this was done in collaboration with the team and one member of the team was seconded into the workstream, it couldn’t help but feel like the team wasn’t trusted to solve its own problems.

All these issues considered, it’s fair to say that the team members had the collective feeling of not being heard, understood, or supported by the rest of the company.

Fixing Things

Call for Support

The R Team is absolutely critical for our business, but the team was falling apart. Engineers wanting to leave a team or even the company is a very clear signal that a fundamental change is needed and things can’t continue in the same way. It was also a signal that we failed to take action on earlier signs. To deal with this, we knew the situation needed to be escalated through the whole company. And we escalated the hell out of it: within stakeholders meetings, in executive updates, and with whoever asked about the team.

For starters, we began to look for people internally, and thankfully, we found great people to join the team. Both senior and junior engineers stepped up and wanted to help. And in an effort to bring the team back to its old capacity, we opened a job ad.

We also got first-hand support from a Senior Principal Engineer who joined the team for more than a quarter. He kick-started migrating to a new tech stack, helped with forming a new tech vision, and provided guidance. All of this aided in relieving some of the pressure on the team. In addition to that, we found an amazing Product Manager whose tasks were to clearly prioritize projects and chase down simplifications. The situation of the team was being addressed, and the members felt like they were being listened to by the rest of the company. Support had arrived.

When scaling the team, we also made it more diverse. We now have a team in which three out of seven engineers are women and the ratio of junior engineers to senior engineers is also balanced. This change automatically brought in new, differing, and fresh perspectives. It also helped the team in discussing problems more holistically; no questions remained unasked. This increased knowledge sharing and collaboration, and this alone already began shifting the team’s mood.

New Trust

In my opinion, the most important asset of a team is a shared goal in form of team vision, and the vision must come from within the team. The team members are the experts, and only the experts will be able to come up with the right plan. And as a bonus, because the entire team is involved in forming the plan, each team member naturally buys into it from the very beginning.

But before a team can start discussing great plans, it needs be able to discuss effectively. So let’s get back to forming the vision later and instead focus on communication.

The foundation of team discussions in which everyone speaks up and all options are being brought up is trust. This is not only a must in order to come up with a long-term team vision, but it’s also absolutely necessary to effectively discuss small design decisions ad hoc. The fascinating simple mechanic here is: Get people to open up and share their story, and see how mutual understanding and trust grows. There are plenty of exercises to foster trust within a group, and a lot of them work. For us, two exercises did the trick: “Paint a picture about yourself,” in which we explained where we’re coming from; and “Ways of working,” in which we discussed how we worked — more specifically when we’re the most productive, what’s most important to us, and what’s most frustrating to us individually.

And of course, offsites were instrumental in building togetherness within the team. Taking time away from project work to have full-day offsite meetings, which are primarily focused on vague trust building, might first seem like a big investment, but these paid off immediately. Everyone got to know their teammates better, and as a result, both communication and collaboration improved almost instantaneously.

Defining a Vision

In a couple of workshops, we discussed the biggest pain points the team was facing. We started to put together a “Most Wanted” list of the worst kind of tech debt. In separate sessions, we discussed an ideal solution to our domain challenges and came to similar conclusions, albeit from differing angles. We decided we would weigh work in planning sessions to determine if it would bring us closer to our tech vision or help eliminate painful tech debt like slow, flaky, or tedious manual processes.

But how do we cover what we actually need to deliver to our business? It turned out that just two metrics needed to be visualized to measure our progress: First, a view of how well we were doing in hitting monthly deadlines to generate data outputs (pictured). And second, the number of completed deliverables versus the total number of deliverables. This made our progress and our targets very clear.

Another tool that turned out to be super useful was a complete system diagram, which functioned not only as a snapshot in time, but also as a living document we would actively maintain. We would even color components we wanted to get rid of in red. And in order to visualize progress we made, we cloned snapshots of the drawing every quarter. We also linked all the components to repositories or other resources. If anything was unclear or missing, we often discussed this in the comments of the diagram.

Now we had a go-to visualization that everyone could use to help in understanding the big picture. And we still use this diagram today — for example, during sprint planning, in order to discuss upcoming epics and their dependencies and changes in relation to the rest of our infrastructure.

Automate All the Things!

The productivity of the R Team relies upon a responsive business expert to clarify all kinds of questions, but as mentioned earlier, the expert was often too busy with manual tasks, like creating reports, to respond to requests. He was even trying to help the team by manually delivering tasks, but this almost led to a deadlock situation. He was a constraint in the team’s value chain.

Automating the tasks on his plate unblocked us and freed up more than 30 hours of his time each month. In addition, it also increased quality and decreased the total time of the related business processes. Computers were invented for the purpose of automating repetitive tasks, and doing so in this situation made work on the business side much more meaningful. Together with business, we now tightly collaborate on the epic at hand. With the shorter feedback cycle, we’re much faster. Additionally, the business side now has more bandwidth to talk to external partners directly and manage their relationships.

Ruthless Prioritization

Even if determining top priorities may seem impossible at times, it can always be done. It must always be done. Depending upon your team size, you also need to limit the amount of work streams. As a rule of thumb, the number of team members divided by two is the maximum number of work streams that should be in progress.

There is a long list of reasons why this is absolutely necessary:

For individuals, working sequentially is faster because it reduces context switching.
Collaboration is the most effective knowledge sharing. Ignoring that leads to self-imposed fires because any individual team member may eventually go on vacation, go on parental leave or sick leave, or leave the team altogether. Ideally, you actually want at least three engineers to have enough context to work on each task.
It has a direct impact on team happiness. It allows people to support each other by discussing and solving problems together and to ultimately feel like they’re part of a great team.
Wrapping up work-in-progress as quickly as possible helps with managing stakeholder expectations and gives the team a sense of accomplishment and progress.

Of course, this all may sound like common sense. However, it’s really easy to fall into the trap of committing to too many things when we are under pressure, both individually and as teams. Taking on more work when there’s already too much is neither healthy nor effective.

Stakeholder conversations can be tough because you need to push for the exact order of the top priorities and the team must only work on a few of them at a time. This might be quite a shock for the stakeholders. But by doing so, the team will actually start to ship efficiently again. And by showing you can complete work efficiently, you regain trust with your stakeholders. In short, building trust with stakeholders is fascinatingly simple: Be very clear with the progress of all your projects and deliver them one by one.

Conclusion

With the right approach, the R Team’s morale was recovered in a very short time. And after only four months, the team was back on track delivering on business requirements while chipping away at tech debt. And none of this was magic.

As stated in the beginning, operating at the throughput limit of a team will tear it apart. A team with 99 percent utilization will actually be slower than a team with some breathing room. And as tech debt piles up and knowledge silos develop further, the team will start to struggle.

Technical challenges and business pressure are also not the problem. We all want to work on important and challenging problems and make a great impact. The problem is a lack of focus and collaboration.

Defining a Role with a Retrospective Activity

Fri, 22 Jun 2018 00:00:00 GMT

Agile retrospectives are a widely used practice within engineering teams. They provide teams with a way to reflect on how they work and become better at what they do. One of the main benefits of retrospectives is that they empower teams to define and make changes by analyzing what happened in an iteration and by determining what can be improved moving forward. Here at SoundCloud, we hold retrospectives at the end of every iteration (every two weeks), and we often do them at the end of projects as well.

In this blog post, I’ll share how I used a well-known retrospective activity with my team to define the responsibility of a role.

A few months ago, our iOS Tech Lead took on a new role, which left the team with an open position to fill. Usually, the person in a particular role is the one to define the tasks and expectations of that role as part of a handover. This can be done by writing down what they did and what they were supposed to do, but this process can be somewhat subjective and error-prone.

In this case, because the iOS Tech Lead was involved in many things, some of which weren’t directly related to the team or the platform, there weren’t clear boundaries for the role. That said, both the team and the hiring manager agreed that the definition of the role needed to be updated.

As the Engineering Manager of the Consumer iOS Team, I was asked to come up with a proposal on how to rewrite this role definition together with the iOS Collective. The iOS Collective consists of engineers from the Consumer Team, the Media Streaming Team, and the Core Clients Team. What a great challenge!

I began by asking myself what a good format was for creating a safe space for gathering peoples’ opinions, facilitating discussion, and finding out what the team needed from a Tech Lead. I knew it was important to have the opportunity to bring up new ideas and responsibilities as well, since the team has changed and evolved over time.

I invited everyone from the iOS Collective to come prepared for a one-hour meeting and give me their answers to the following questions:

What is good in the role, and do you see a value in it?
Is there something new you want to see in this role?
Is there something that is being done that you would rather see less of?
Is there something that is being done that you think would bring more value if it is done more?

This is a modified version of the KALM — Keep, Add, More and Less retrospective format, which is used for having a conversation about current activities and their values as perceived by team members.

For our meeting, we used Post-it Notes to give answers to each of the four questions. Everyone came with their notes, organized by color, and added them to the whiteboard. Since we had a limited amount of time and there were nine of us, each giving around 10 answers for the questions, we knew we couldn’t discuss everything. Instead, we focused on the answers that people did not agree on or that were not clear.

We managed to discuss and clarify the topics that stood out as things people didn’t agree on or were confused about, and we ended up with the following, not counting the duplicates:

Keep: 26 topics
Add: 12 topics
Less: 5 topics
More: 11 topics

The fact that there were so many responsibilities to be kept and almost the same number to be added and done more justified the need of the role definition. It felt really good to being able to do this important work together and have the assurance that the next Tech Lead would be set up for a good start.

In addition to using this information to define the role of the new Tech Lead, the results were given to the former Tech Lead as feedback on the work they did in that role. It was also a chance to realize what the team was happy with, what was not necessary, what they were missing, and what they wished was done more often.

Using the output of this session, together with the hiring manager and the former Tech Lead, we defined the responsibilities of what is now called the iOS Platform Lead.

Involving the team in the process resulted in a renewed sense of trust, which reinforced morale. It also gave everyone confidence that the new iOS Platform Lead can work in a way that meets the needs of everyone. This is in addition to having well-defined expectations for their role, which is a necessity for performing well and growing.

Roles of the iOS Platform Lead

Project Necromancy

Tue, 05 Jun 2018 00:00:00 GMT

Or, how to raise a project from the dead with tools you probably have lying around at home.

Background

In a perfect world, every project would flow smoothly — from requirement gathering, to implementation, to acceptance by stakeholders, to launch. Sadly, we don’t live in a perfect world. Sometimes projects hit bumps in the road due to changing business requirements, market conditions, or a host of other factors. This post describes our experiences managing a project that was deprioritized during implementation and lay dormant for an extended period of time before resuming in a different form.

The Life and Death of Project Mark I

The original incarnation of the project in question would have added a feature to an internal tool with significant use across different areas of SoundCloud.

For this project, we added a new microservice to our backend architecture. We deployed and integrated this service without any functionality to begin with, and then we set about building (though not yet deploying) the service’s feature set.

We suffered a minor setback in feature development when a crucial, overlooked edge case came to light and forced a partial redesign of one service component. This resulted in our having to throw away the code we’d written for the component and start again. It was not a significant development in and of itself, but it later informed our thinking about this component. We felt we’d learned an important lesson from the initial mistake and any further changes to the component would risk reintroducing the design flaw.

Otherwise, the project was going fine… Until the day it was suddenly canceled. Our product roadmap had shifted. The functionality we were building was no longer important. The team’s energy needed to focus elsewhere.

We discussed removing our new, currently non-functional service altogether, but we decided to leave it in place. After all, it wasn’t doing any harm, and perhaps the project would resume one day. We also preserved our partially finished feature code in a git branch, just in case.

Then we moved on and largely forgot about our seemingly doomed project.

Interlude

The next several months saw our team merge with another team and several individual team members move to unrelated areas within SoundCloud. Meanwhile, our product roadmap continued to evolve, and the functionality we’d started to deliver with Project Mark I, previously unimportant, eventually became important again. Or rather, certain aspects of the functionality became important again.

Four months after the original project’s cancellation, we were ready to resume developing a subset of its features with a team that contained almost none of the original engineers.

The Project Reanimated

We had to make a decision. Should Project Mark II build on the work we’d done for Project Mark I or not? Project Mark II contained only some of the planned Mark I features. Was there enough commonality to resume the old project, or would it be easier to start afresh?

We decided to reuse as much of the Mark I code and documentation as possible. And after an abbreviated requirements phase, we set to work building Mark II on top of the existing Mark I codebase and data model.

Picking up unfinished code written by engineers who were no longer on the team presented challenges, but not the typical challenges of working with legacy code. We had no problems with documentation, technical debt, or understanding the existing code. Instead, we developed a cognitive anchor effect around choices the previous engineers had made. We were reluctant to question design decisions because: “They must have done that for good reason.” This was especially the case given that we knew an earlier design mistake had necessitated a partial code rework. We thought we were incorporating earlier learnings by leaving things as they were and that we risked repeating the earlier mistake if we changed things.

The engineers who wrote the initial code were all still at SoundCloud and would have been happy to answer questions, but they were also busy with their new teams, and we didn’t want to bother them. As it turned out, we should have bothered them more.

Another group we should have bothered more was our set of stakeholders in SoundCloud’s product organization. This became apparent when we demonstrated Mark II’s functionality to them only near the end of the project, and at that demo, learned a crucial fact: a major feature we’d built for them was not needed. Unfortunately, the concept behind this feature was “baked into” the data model we’d reused, unchanged from the original Mark I project where it had been a hard requirement. As we’d failed to question the original data model, we’d also failed to question the need for this complicated feature that we had spent a considerable amount of time building.

Oops.

It didn’t make sense to support this feature if it wasn’t needed, especially as it complicated the user interface. We therefore made the somewhat painful decision to push back our launch while we removed the feature and simplified the data model. While the delay was frustrating, we felt it was important to deliver the right project, even if that meant delivering later. Today, Project Mark II has many happy users, so we believe we made the right decision.

Lessons Learned

Difficult or frustrating situations nevertheless yield valuable knowledge for the future. That said, here are some of the things we discovered over the lifetime of these projects.

Reanimated Projects Are Changed Projects

We thought of Project Mark II as a direct continuation of Project Mark I. This was only partially accurate. The decision to reuse the Mark I code and data model introduced an implicit assumption into Mark II: “When in doubt, old decisions are right.” While reusing the existing work certainly saved us substantial time and effort, the cognitive bias against questioning the existing work also cost us substantial time and effort.

Requirements Are Even More Important Than Usual

Getting requirements right is important in any project. It’s also hard, and it’s even harder when the requirements have a high chance of becoming outdated or being misunderstood.

When we started Project Mark II, we revalidated some of the Mark I requirements, but we should have rerun the project’s entire requirements phase afresh. This would have forced us to examine requirements that had already become code. It would have also forced us to engage with our stakeholders, who were scattered across multiple teams and multiple geographic locations, and to have conversations we didn’t know we needed to have.

Show Your Work

Finally, given the project’s unusual lifecycle, we should have demoed our progress much sooner than we did. We don’t generally do sprint demos in our team, but in this case, it would have saved us a lot of effort.

TL;DR

When resurrecting an unfinished, dormant project:

Revisit your requirements. All of them.
Revisit your existing code. All of it.
Be conscious of cognitive biases that say, “But someone decided that earlier.”
Adopt a flexible mindset. Be prepared to change course.
Communicate with stakeholders even more than usual.

Engineering Levels at SoundCloud

Thu, 24 May 2018 00:00:00 GMT

An absolutely crucial part of the experience of being an engineer at SoundCloud is learning and growing as a person. Pretty much everyone we hire mentions this aspect as one of their main motivations for joining the company. And while retaining highly talented and motivated people and helping them develop is naturally valuable for SoundCloud as a company, it’s also profoundly beneficial for the employees themselves.

Internally, we have programs that exist to help our engineers grow. These include providing a budget for conferences and courses, organizing tech talks and meetups, and coordinating with external experts to host special workshops. We put a lot of effort into creating an environment where people are encouraged to improve. It’s a mindset — part of our culture. We keep leveling up everything: ourselves, our teams, our systems, our processes. Every day is an opportunity to make something work a bit better.

The other side of this story is that SoundCloud also has to recognize this growth. We want to embrace people who are making great progress, and we also want to make sure we keep paying them fairly as their market value grows alongside their skills and expertise.

Why Do We Have Levels?

To provide clarity on what our understanding of growth is, we have a level system (as many other companies do). This system defines our engineering culture by giving guidance to people on how to increase their impact within SoundCloud, facilitating career discussions between engineers and their managers, providing a framework for setting salaries, and last but not least, encouraging and rewarding certain behaviors.

Transparency around everyone’s level at a company helps people identify their role models. Additionally, an explicit level structure is beneficial for people who are underrepresented, as non-explicit hierarchies tend to favor the people who form the majority (which, most of the time, is white men).

How Do We Set Levels and How Do We Use Them?

Our leveling process begins as early as a prospective new employee’s interviews. There are three key factors that help us assess the proper level someone enters the company at. We start with their past experience — how long they have been in the industry and what kind of companies they have worked for. Second, we review the interview panel’s feedback, which helps us understand how the candidate could contribute to the team. Finally, we take into consideration a candidate’s personal assessment of which level they belong at.

With every candidate, we actively work against any unconscious biases to ensure that we make a fair and accurate decision. For example, research shows that employers in science tend to underestimate women’s seniority and market value. This is the kind of bias that we are committed to acting against.

Once someone has joined the company, we set expectations to match their level. It’s important to emphasize that even though being on a higher level comes with a higher salary, it also sets higher expectations.

When someone has clearly grown, picked up new responsibilities, and through this, increased their impact, they should be promoted to the next level. Currently, we have a process where someone who wants to be promoted puts together a promotion case accompanied by feedback from peers with whom they work closely. People giving feedback are asked to use the document that defines each level to reason about skills and performance. Then a promotion panel, made up of a diverse group of SoundClouders, convenes twice a year to make decisions about each individual case.

Regardless of an employee’s level, managers regularly meet with engineers in private one-to-one meetings to discuss how they are doing. The content of these discussions is based on the engineering level definitions, and the meetings themselves provide guidance about which areas an engineer is doing well in and where they need to improve. Conversations like this help them to be successful at the company and reach the next level.

What Is Expected at Each Level?

We have five levels for engineers at SoundCloud. Here is a short summary of them and what they entail:

Level 1 — The individual scale
Focusing on themselves and improving their own skills. Successfully delivering as part of a team with the help of other team members.
Level 2 — The team scale
Focusing on their team and improving the team’s efficiency and productivity. Successfully driving a team project.
Level 3 — The area scale
Focusing on their own team and some neighboring teams and improving collaboration between these teams, e.g. successfully driving a cross-team project.
Level 4 — The organization scale
Having an impact on the entire engineering organization. Improving efficiency and collaboration across the organization.
Level 5 — The company scale
Having an impact on the entire company. Driving cross-organization initiatives, and leveling up SoundCloud as a business.

To create the official descriptions of expectation for an engineer at each level, we collected all the behaviors we desire in an engineer and assigned them to five dimensions:

Results/Delivery
Behavior/Mindset
Tech Skills/Mastery
Influence/Visibility
Communication/Collaboration

It’s important that this list of behaviors is not used as a checklist; one does not need to tick all the boxes to be eligible for a promotion. Additionally, many of them are not explicit expectations, but rather indicators, and we are often convinced that someone has reached a particular level if we see most of these indicators in practice.

Here is the detailed view of these levels:

We are proud of our engineering culture here at SoundCloud and the many steps we have taken to grow it, and we’ve found that the definitions of our engineering levels are the best way to capture and document it. As our culture evolves, we will continue to update the level definitions in order to reflect these changes. This ensures that we can keep providing our engineers with clear definitions of how to be successful at their jobs.

Keeping Counts In Sync

Fri, 11 May 2018 00:00:00 GMT

Track play counts are essential for providing a good creator experience on the SoundCloud platform. They not only help creators keep track of their most popular songs, but they also give creators a better understanding of their fanbase and global impact. This post is a continuation of an earlier post that discussed what we do at SoundCloud to ensure creators get their play stats (along with their other stats), both reliably and in real time.

A long time has passed since we shared about the challenges of and our approach to real-time stats. Since then, we’ve had to expand our infrastructure to cope with the continuously growing stream of events representing plays, likes, downloads, and other important social signals that we’d like to count and present to creators. As a result, what might seem like a simple counting system is actually a cornucopia of services orchestrated to power features like creator statistics, reporting, search, and recommendations. The list goes on and on.

What We Talk about When We Talk about Counts and Statistics

The main challenge for a statistics powering system is that we not only need to present summed up totals for arbitrary time ranges, but we also need to allow creators to dig into these summed time ranges with dynamic granularity. For instance, when someone is interested in a week of play counts, aside from displaying the totals for that week, they might like to see the daily performance as well. Ideally, the total of those days should add up to the total shown for the week, and when zooming in to a specific day, all the hourly counts would be shown as well.

Another challenge is keeping the total counts of plays, likes, reposts, and other signals we would like to display for a certain track updated. These counters offer an aggregated view of the entire time a track is present on our platform, and they are ubiquitous in that they are visible everywhere a track is displayed. Because, most of the time, these counters are public, they are requested from the service several orders of magnitude more times than the actual stats that are only visible to the individual creators.

Current Solution: Stitching Counts

As highlighted in the aforementioned post, our system Stitch is at the heart of the stats and counters. Stitch pre-aggregates events over several dimensions to answer queries efficiently. The main dimension is time, and the buckets consist of hourly, daily, and monthly partitions. The second dimension is the different type of events, of which the play events are the most abundant, making them the most challenging to keep updated. On top of all this, we are doing this pre-aggregation for every track, along with a summation of every creator’s overall stats and counters.

Stitch architecture is an incremental Lambda Architecture. Instead of always regenerating the serving layer from the beginning of time, we only generate the changed buckets of hours, days, and months. Batch counts for the current day — including hourly batches — are handled with a daily job, and batch counts for the current month are handled with a monthly scheduled job.

We’ve chosen this way of regenerating the serving layer in batches from the end of the last batch calculation performed because play counts do not generally come from the past (with the exception of things like spam and backfills from bugs in the event pipeline). Creators also tend to expect monotonically increasing play counts over time, although this property is not valid for counts of “likes” or “reposts” that could actually go down as well.

With this setup (shown and described above), we have the possibility of swapping or recalculating certain days or months of data, but usually we don’t need to, so long as everything goes fine (as in no bugs and no network failures or other outages). The Crunch batch jobs run in our Cloudera-flavored Hadoop cluster over data ingested and archived by our event pipeline based on Kafka. Both the batch and the real-time layer convert the Protobuf event messages into increments and sum them up in their specific buckets.

The batch-generated “serving” layer and the real-time or “speed” layer, as coined in the Lambda Architecture, both use Cassandra as their primary data store. In the batch layer, we use the Sorted String Table (SSTable) loading feature of Cassandra to effectively ingest even several months of data. For the speed layer, we took a different approach of leveraging the distributed counters feature, issuing every increment (or decrement for comment, repost, or like counts) as a database update.

The Stitch API constitutes the query layer that consolidates (or should we say stitches) the real-time view and the batch-aggregated numbers seamlessly as a unified response to the clients of the service.

Total Counts

Given that the system aggregates counts by the creator, track, and time, answering total counts on a given track for all time requires fetching by all track and time buckets. Given this constraint, we added a caching layer to store total counts for a given track. To lower the pressure on our Cassandra cluster, we added a Memcached cluster to serve approximately the top 90 percent of the most requested total counters.

We increment both the source-of-truth counters in Cassandra and the cache counters in real time, but in order to limit the divergence of the two views, we invalidate the entire cache on a daily basis with a randomized time to live (TTL) parameter. The randomization is needed to lower the probability of highly accessed track counters all invalidating at the same time, which would cause periodic spikes in our cluster usage. The invalidation is needed because our event pipeline only supports at-least-once delivery and the updates are not transactional.

Top Lists

As part of our stats product for creators, we offer various top lists, thereby allowing them to gain more insight into their audiences and track performances. This system is responsible for generating and serving these lists in a way similar to how Stitch provides counts. By connecting to the same event pipeline, the system aggregates the events and creates partitioned top lists for every dimension we’re interested in, such as the most played tracks for a creator for a day.

One of the challenges we’ve faced in splitting these two facets of the same product into two different systems is keeping the two views in sync by scheduling them at the same cadence and always triggering recalculation on all the systems affected by a problem. Additionally, it’s important to run spam filtering and throttling as part of a common processing unit upstream in the event pipeline.

There is another system that provides global top lists. It relies on the same event stream and is a good example of the versatility of having a consistent pipeline that can be tapped into by several decoupled systems. That said, it faces challenges similar to those of the aforementioned stat product for creators in terms of consistency of the numbers and ordering.

Alternatives

Our internal engineering culture is driven by RFC-based decision making, and one of the key points in these collaborative documents is finding alternatives to the presented solution.

Additionally, while growing and optimizing a system, it’s important to properly assess whether it’s actually the right thing to do or else consider that maybe it’s time for a radical change. Every time we have to grow our Cassandra cluster or fine-tune the batch jobs to cope with more and more data, we get a tingling feeling that maybe this is not the way forward. The inherent tradeoffs we’ve made around using more storage for query efficiency, or dealing with caching and invalidation, or having to reconcile different views of the same dataset, were all predicated on older knowledge of the problem and solution space.

By exploring the alternatives (sometimes over and over again) we are able to substantiate our choices. In the case of Stitch, we tried out most of the following approaches, either during SAT (self-allocated time), or as scheduled spikes to improve our systems, but none resulted in a breakthrough that would justify fundamentally changing our current setup (yet).

Always Count

Historically, we started with a huge database of all events. Every time a statistic or a total count was needed, we issued a query to this database to actually count all the events in real time. This quickly became infeasible and we started extracting this functionality into dedicated databases, and finally into our current Stitch system.

With “Fast Analytics“ databases “buzzwording” around, it might be time to revisit this approach. By storing these increments in columnar stores that provide efficient analytical queries, we could use less storage than the pre-aggregation approach does. Analytical databases like Amazon Redshift, Google BigQuery, Apache Kudu, or even Vertica provide results in seconds, but not milliseconds (as our current approach does). Maybe a hybrid approach like FiloDB, implementing columnar storage on top of the Cassandra store, could be a valid alternative.

Pre-Aggregate Differently

Our current solution relies heavily on pre-aggregation, so another approach would be to use a different pre-aggregation scheme by leveraging some funky data structures like Fenwick trees. The main challenges would be in creating efficient distributed versions of these prefix sum data structures.

One viable way could be by leveraging the partitions already present in our Kafka-based event pipeline and building stateful services by continuously consuming events, similar to Kafka Streams, and providing a query API on top of these mini stores.

Another challenge is augmenting algorithms to not only result in total counts for a range, but to efficiently calculate all the individual data points for a specific range and granularity. When we tried out this alternative, we represented the trees as sparse arrays, utilizing fast integer-packing algorithms implemented in JavaFastPFOR to lower the serialized footprint of the data. Nonetheless, the overall latencies were not as performant as the Cassandra store approach, and the complexity of the code was significantly higher. Essentially, we were reimplementing a database.

Another alternative would be to only aggregate data that will be queried. This would trade some initial latency for a queried period for a substantial amount of storage and potentially unused computation time. We could take user requests and pipe this into an adaptive caching system.

Back to the Drawing Board

As already stated above, none of these approaches has resulted in us changing our setup, because of extra operational/engineering overhead and the unacceptable increase in latency they would require. However, it has taught us valuable lessons about engineering tradeoffs and the challenges of keeping things synchronized and consistent, and we remain open to exploring new ways of keeping track of counts.

Special thanks to Anastasiia Basha, Jeff Kao, and Natalye Childress for all the help on this post, and of course to other past and present SoundClouders who have contributed to continuously evolving these systems. If you are interested in solving these kinds of challenges, please check out our jobs page.

Running Android UI Test Suites on Firebase Test Lab

Fri, 04 May 2018 00:00:00 GMT

Testing mobile applications is not always an easy feat. In addition to defining what to test and determining how to write those tests, actually running tests can also be problematic — in particular, UI test suites running on real mobile devices or emulators sometimes run for an extensive amount of time.

This post talks about both how we at SoundCloud initially handled our Android UI tests with internal tooling and our migration to Firebase Test Lab for running all of our UI tests.

In the CI builds running on our Android listener application, the time it took to run the UI tests was always a bottleneck for the overall runtime of a build. So in order to not just rely on one single device or emulator, we developed tooling and services to allow us to run our UI tests on an internal device farm using 30 Nexus 5 devices in parallel. This made it so we could run our tests in about 15 minutes, as opposed to the three hours it would take if we ran them all on a single device.

Unfortunately, after years of working successfully, this system started showing signs of its age, causing multiple problems in our day-to-day development. Initially, it had a great feature set with rich UI test results and reports, video recordings of failed tests, reports of the most frequently failing tests over time, and more. But eventually, it started to slow us down. We frequently saw long build queues and failures due to infrastructure issues, and soon it became clear that we had to either fully commit to maintaining and updating the system, or else start thinking about alternatives.

We began investigating different platforms and cloud providers offering real devices and emulators in different configurations, and in the process, we realized two problems we would face.

Our test suite was tailored to running on physical Nexus 5 devices, and due to shared state, many tests had to be run in isolation to work correctly. Additionally, our custom test services made sure every test ran in isolation without sharing any state with other tests. However, cloud providers for running Android tests offer a variety of devices in different sizes and usually run all tests one after another, often with shared state between tests. This difference, along with the age of our test suite, resulted in multiple issues for us.
The size and runtime of our test suite meant that running our UI tests using a cloud provider to spin up a single device or emulator would degrade our test feedback times to multiple hours.

Making Our Test Suite More Robust

It was clear that in order to run our tests on other device/emulator farm setups, we would have to make our testing more robust. For example, some tests expected to run on the exact screen dimensions of a Nexus 5 and started failing if the screen size differed slightly. Others couldn’t handle the state other tests left behind.

Around the same time that we were looking at different solutions and experimenting with ways to improve the stability of our test suite, Google released the first versions of the Android Test Orchestrator, which wraps the test instrumentation to allow each single test to run in its own Instrumentation. The result is that tests do not share a lot of state, and each one runs more in their own sandbox. Additionally, if one test happens to crash, the other tests can still run and report their correct states; the test suite will not be interrupted immediately due to a crash.

Using the Android Test Orchestrator is as easy as adding a few lines to your project’s build.gradle file, and doing so can help stabilize a test suite quite a bit. This, of course, depends greatly on the test suite itself:

android {
  defaultConfig {
   ...
   testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
 }

  testOptions {
    execution 'ANDROID_TEST_ORCHESTRATOR'
  }
}

dependencies {
  androidTestImplementation "com.android.support.test:runner:<latest_version>"
  androidTestUtil "com.android.support.test:orchestrator:<latest_version>"
}

After adding this piece of setup every time tests from the androidTest configuration are run they will run with the Android Test Orchestrator.

Firebase Test Lab

Firebase is Google’s suite of many services for building great mobile applications. One part of it is the Firebase Test Lab — this enables developers to easily create and launch a test run on a real device or Android emulator hosted in a Google data center and run a set suite of tests on it. After the tests have finished running, the Firebase web UI will then display the results of each test — in addition to information such as a video recording of the test run, the full Logcat, and screenshots taken with the help of the Firebase Test Lab Screenshotter Library.

Tests can also be run on Firebase Test Lab through Android Studio. Simply sign into Android Studio with the Google account linked to your application’s Firebase project. When you run a test, select the “Cloud Testing” tab, choose the Firebase project, and configure the devices/emulators you want to run your tests on.

For our use case, which was wanting to test our Android application, the choice to go with Firebase — which will always allow us to test on the latest Android versions and use the newest test tooling such as the Android Test Orchestrator — was an easy one.

However, the main use case of Firebase Test Lab is one that allows users to upload the application APK and a test APK. Firebase Test Lab then runs all the tests contained in the test APK — or a subset of them depending on the configuration — on one device/emulator or a matrix of different ones. However, every device and/or emulator will run the same set of tests. For a test suite of our size, this would result in huge test durations.

Massively Sharding Tests

Firebase Test Lab currently does not offer splitting a test suite into multiple parts and executing smaller parts of the suite on multiple emulators in parallel to bring down testing times. However, the open source community (namely Walmart Labs) has come up with a solution for this.

Flank is a tool that allows for massively sharding an Android UI Test suite across many devices/emulators on Firebase Test Lab, at the same time, in parallel. In other words, instead of running all tests on a single emulator, it allows for scheduling a smaller amount of tests on multiple devices/emulators. Each of these shards will then execute in parallel, massively reducing the time it takes to run all tests in the process.

Flank is configured with a small file defining the device or emulator that should be used, the Android OS version, and how long each shard should run for. Flank keeps a record of the duration of each test so that it can group them together to ensure a fairly constant runtime. Additionally, as Firebase bills Test Lab usage by the minute, Flank tries to intelligently group tests together so that they use the minimal overall billable time on the devices, thereby saving costs.

By using Flank, we were able to reduce our overall test suite runtimes when compared to our previous internal systems. And in case our test suite grows larger, Flank will help us schedule our tests on more emulators, enabling us to continue to have our current test durations.

Our previous in-house testing solution gave us many features we got used to and loved, which we lost when we started using Flank. For example, after a test run, we would receive a test report in rich HTML with links to Logcat and video recordings of the test run for failing tests. Additionally, filtering tests by annotations was a way for us to easily configure different suites of tests to run, depending on the current task. However, as part of our work to migrate our UI test suite to Firebase Test Lab, we built these features into Flank and contributed them back into the main repository so that other users of Flank can enjoy them as well.

Overall, running our tests in Firebase Test Lab with Flank has helped us modernize our UI testing stack, stabilize our test suite, and even speed up test runtimes in the process, which leads to happier and more productive developers.

Managing Unplanned and Support Tasks

Mon, 26 Mar 2018 00:00:00 GMT

One challenge engineering teams often face is dealing with work that doesn’t revolve around developing new features but that still requires the team’s attention and time. The Content Engineering Team here at SoundCloud is no exception, so we iterated on a process to deal with unplanned and support tasks to end up with fewer interruptions and more time to spend on implementing planned features.

We define unplanned and support tasks in the following way:

Unplanned tasks — work that unexpectedly appears and needs immediate action. An example of this is a severe bug that impacts our users.
Support tasks — work that doesn’t need immediate action and can be planned but still needs engineering support. Reasons for these tasks can be lack of automation, lack of visibility into the systems we build, work caused by infrastructure changes, or non-critical bugs we introduced ourselves.

There are several reasons why it is important that we come up with an approach to dealing with unplanned and support tasks. For one, these tasks take up time we would normally spend on delivering features, so naturally we want to minimize the number of them. Secondly, we want to make this kind of work visible; it is often invisible outside of the team, which can give the impression the team is slow and dysfunctional. Finally, these unexpected and often urgent tasks cause extra stress and context switching for the team — in addition to the fact that support tasks are often not very interesting, due to their manual and repetitive nature.

The Request Process

Initially, my team had a very open way of dealing with requests from stakeholders that resulted in unplanned and support tasks: People who had requests could leave them in our Slack channel, and a team member in a single point of contact (SPOC) role was supposed to address them.

Some team members liked taking on this role because they liked helping out our stakeholders. Others didn’t like it as much because they didn’t want to be interrupted, and as a result, were slower in responding to requests.

And even though this idea resulted in a designated person responsible for addressing incoming requests at any given time, all the other team members still saw them. As a result, they were often distracted, and some people started discussing the requests even when they were not urgent. It got to the point where the distractions and unclear priority of the requests began popping up as a recurring topic in our team retrospectives.

This led us to try another approach. We got rid of the SPOC role and instead tried asking stakeholders to reach out using our team Slack channel or via direct messages, but only for urgent requests. We defined an urgent request as something that impacts our users, disrupts a key feature, or blocks employees from doing their work. For all other requests, we specified an email address that reached the team Engineering Manager (EM) and Product Manager (PM).

We had no difficulties convincing stakeholders to try this approach, and as a result, the number of requests in Slack dropped almost immediately.

This is the support request process we still use today.

Planning the Work

We have two-week iterations in our team, and if a non-urgent support request comes up, we add it to the next iteration. The next iteration rule is important because we want to avoid building up a backlog of issues that never get worked on.

Initially, we went over the support task tickets during our planning session so that everyone was aware of them. Then we let the team self-organize with the goal of finishing the tasks by the end of the iteration. This resulted in a situation where it was often the same team members tackling the support tickets while other team members focused on feature work the entire length of the iteration. However, this wasn’t necessarily fair, so after the issue came up in one-on-one meetings and retrospectives, we addressed it and found a new solution.

Our new way of handling these tasks was to assign support tickets to team members in a round-robin fashion, thereby ensuring everyone got a fair share of support tickets. This not only improved team morale, but it made it so that the work was divided more equally and fairly, which also resulted in increased knowledge sharing.

Another variant we started trying out recently is having one day in the iteration when the team works together to finish the support tickets. This has the additional advantage that more pairing takes place, which is better for knowledge sharing. It is also nice that everyone shares the work together at the same time as a team effort.

Tracking the Work

As part of the overall process, we also think it’s important to track the work done on these tasks. We do this by creating a JIRA issue for every request that comes in, even when the request can be answered by the Engineering Manager or Product Manager. Once an issue is created, we also share it with the stakeholder who initiated the work so they can follow up on the progress.

Every request JIRA issue is labeled. Urgent requests get the “unplanned” label, and non-urgent requests get the “support” label. Then, once we are done working on an issue, we log the estimated amount of time we spent working on the issue using JIRA’s Log Work functionality.

Measure, Learn, and Act

Keeping track of the work in this way allows us to measure it, learn from it, and act on it.

More specifically, about once every two months, we query all the finished unplanned and support issues of the past iterations in JIRA. We then group them in categories and add up the time spent on them. An example of this from earlier this year is shown below.

Based on the categories and the time spent, we come up with actions. Some examples of actions we’ve come up with so far include:

We found out that we spent quite a lot of time supporting our operations team because we were missing some automation to perform a specific task. This led us to prioritize the work required to automate the missing step, which not only got rid of the repetitive and boring tasks for our engineers, but also removed the dependency and waiting time for our operations team.
We spent a considerable amount of time working on fixing our deployment pipelines because of infrastructural changes that were outside of our control (e.g. updated base Docker images). This led to the decision to inform the infrastructure team and ask them to consider if the benefits of their changes outweighed the additional workload for the engineering teams. We also improved our deployment pipelines so that they get triggered automatically when a new dependent Docker image has been updated. By doing this, we get notified of breaking changes immediately instead of being caught by surprise when we make a code change.
When we introduced a bug in one of our systems, it required a lot of time to recover from the bug. We found out it took so much time because the system generated derived data / intermediate state. And because we couldn’t reprocess all the input data, it meant that all the derived data needed to get fixed/patched manually. This motivated us to come up with a plan to improve the system design.
We found out that some of the bugs we introduced were caused by not having good enough validation criteria for our stories/epics. This resulted in us becoming better at defining validation criteria together with our PM.

Measuring, learning, and coming up with action points is important, because having tangible numbers about the time spent on unplanned and support work makes it easier to prioritize work for fixing structural issues. This is good for both the team and for SoundCloud, as engineers are interrupted less and can spend more time on impactful work.

If you are interested in learning more about our processes, you should also read the blog post Deliver Software Faster by Managing Work in Progress, Not by Adding Overtime.

Automatic Stubbing of Network Requests to Deflakify Automation Testing

Fri, 26 Jan 2018 00:00:00 GMT

Apple introduced automated UI testing in Xcode 7. This was a great addition for developers because this native support promised, among other things, an improvement in the flakiness notoriously associated with automation tests. As many of us developers have experienced, tests can sometimes fail even when there has been no modification to the test or underlying feature code.

Flakiness can often come from external factors. One such source of flakiness is live data, e.g. data loaded from a web service API. Data from the web service often drives the state of the UI. A test might assume the UI is in a certain state, e.g. the like button shows as “liked” or that a playlist has 10 songs. But when the data from the web service changes, the test fails.

Such occurrences are relatively common, and there are many other scenarios, such as network connectivity issues, in which these false negatives can occur. In these situations, the functionality of the app is not broken; it’s just that our assumptions about the data are no longer valid. What’s more, debugging the cause of these failures can be a time-consuming affair.

In this blog post, I’ll discuss how we at SoundCloud implemented a simple system for automatically recording and replaying the network responses that occur during the course of an Xcode UI automation test in order to remove any flakiness that may be introduced from the existence of live data coming from a web service.

It should be noted that third-party options such as DVR provide similar functionality. However, given that automatic stubbing of network requests can be achieved with a handful of classes and extensions, this post aims to showcase a lightweight approach that doesn’t require adding yet another dependency to your project.

An example project showcasing the approach taken can be found here. Open ArtistsFeature.swift for instructions.

Overview

The first step is to create an AutomationTestURLSession object that is capable of stubbing the app’s network responses with the contents of a JSON file, since this is the data format we expect from our web service.

The next step is to inject this session whenever tests are running.

Finally, we will implement a system for automatically recording and replaying network responses in order to create an easy-to-use API.

The figure above shows how an AutomationTestURLSession is injected into the app in order to stub network responses using JSON files.

Session Stubbing

Let’s say we have a Client object our app uses to make all network requests. A simplified version might look like this:

class Client {
    let session: URLSession

    init(session: URLSession) {
        self.session = session
    }

    func makeRequest(_ request: URLRequest, completion: @escaping (Data?, Error?) -> Void) -> URLSessionDataTask {
        let task = session.dataTask(with: request) { data, response, error in
            // ...
        }
        task.resume()
        return task
    }
}

In order to replace/stub network responses with the contents of JSON files on disk, what we do is replace the app’s URLSession with our own AutomationTestsURLSession when tests are running. The AutomationTestsURLSession will then be responsible for stubbing the responses. To facilitate this, we start with a couple of protocols that our Client’s URLSession will conform to:

protocol URLSessionManaging: class {
    func dataTaskFromRequest(_ request: URLRequest, completionHandler: @escaping (Data?, URLResponse?, Error?) -> Void) -> URLSessionDataTasking
}

protocol URLSessionDataTasking: class {
    func resume()
    func suspend()
    func cancel()
}

With these protocols in place, we can easily make URLSession and URLSessionDataTask conform to it:

extension URLSession: URLSessionManaging {
    func dataTaskFromRequest(_ request: URLRequest, completionHandler: @escaping (Data?, URLResponse?, Error?) -> Void) -> URLSessionDataTasking {
        return dataTask(with: request, completionHandler: completionHandler) as URLSessionDataTasking
    }
}

extension URLSessionDataTask: URLSessionDataTasking {}

Now that we have URLSession and URLSessionDataTask conforming to our protocols, we can depend on the protocols from the Client instead of the concrete object types:

class Client {
    let session: URLSessionManaging

    init(session: URLSessionManaging = URLSession.shared) {
        self.session = session
    }

    func makeRequest(_ request: URLRequest, completion: @escaping (Result) -> Void) -> URLSessionDataTasking {
        let task = session.dataTaskFromRequest(request) { data, response, error in
			//...
        }
        task.resume()
        return task
    }
}

Now we can create our AutomationTestsURLSession to stub network responses with the contents of JSON files by making it conform to the same URLSessionManaging protocol:

class AutomationTestsURLSession: URLSessionManaging {

    func dataTaskFromRequest(_ request: URLRequest, completionHandler: @escaping (Data?, URLResponse?, Error?) -> Void) -> URLSessionDataTasking {
        return AutomationTestsDataTask(url: request.url!, completion: completionHandler)
    }
}

In the code above, AutomationTestsURLSession returns an instance of AutomationTestsDataTask, which conforms to our URLSessionDataTasking protocol instead of the standard URLSessionDataTask.

typealias DataCompletion = (Data?, URLResponse?, Error?) -> Void

class AutomationTestsDataTask: URLSessionDataTasking {
    private let request: URLRequest
    private let completion: DataCompletion

    init(request: URLRequest, completion: @escaping DataCompletion) {
        self.request = request
        self.completion = completion
    }

    func resume() {
        if let json = ProcessInfo.processInfo.environment[url.absoluteString] {
            let response = HTTPURLResponse(url: url, statusCode: 200, httpVersion: nil, headerFields: nil)
            let data = json.data(using: .utf8)
            completion(data, response, nil)
        }
    }

    func cancel() { }
    func suspend() { }
}

The resume method of our AutomationTestsDataTask is where the stubbing actually occurs.

In order to explain the first line of the resume method, it’s useful to know that Xcode automation tests run to the app in a separate process, which limits the ways in which the app and the tests can share data. One common way to pass data between the tests and the app is by using the launchEnvironment property on XCUIApplication. This is basically a dictionary of type [String: String]. Keys and values you set in tests will be available to the app at runtime via the environment property of ProcessInfo.processInfo.

Here we check to see if there is some JSON in ProcessInfo’s environment dictionary for the data task’s request, as identified by its URL. If we have some JSON for a given request, then we convert it to data and call completion immediately. This results in the stubbed data being returned to our application instead of the live data from the API that would usually be returned!

Now all we need is a way of detecting that automation tests are running so that we can swap out the session object the Client uses.

Session Injection during Automation Tests

At SoundCloud, we create a separate app delegate object to use when running automation tests, e.g. AutomationTestsApplicationDelegate. This can be a good place to set up application state that your tests rely on and/or clean up that state between test runs.

We can hook into this system to detect whether or not tests are running. First we need to set an environment variable whenever tests are running. We can use XCUIApplication’s launchEnvironment property to set a flag. Say we’re writing a test for a sign-in flow in our app. We could alter the implementation of setUp, like so:

class SignInFeature: XCTestCase {

    private var app: XCUIApplication!

    override func setUp() {
        super.setUp()
        app = XCUIApplication()
        app.launchEnvironment = ["IsAutomationTestsRunning": "YES"]
        app.launch()
    }

    func test_signIn() {

        // ... test code

    }
}

We can dynamically set an app delegate class by deleting the standard @UIApplicationDelegate annotation that is added to the default app delegate class that Xcode creates and adding a main.swift file instead:

import UIKit

let isAcceptanceTestsRunning = ProcessInfo.processInfo.environment["IsAutomationTestsRunning"] != nil
let appDelegateClass: AnyClass = isAcceptanceTestsRunning ? AutomationTestsApplicationDelegate.self : AppDelegate.self

let args = UnsafeMutableRawPointer(CommandLine.unsafeArgv)
.bindMemory(to: UnsafeMutablePointer<Int8>.self, capacity: Int(CommandLine.argc))
UIApplicationMain(CommandLine.argc, args, nil, NSStringFromClass(appDelegateClass))

The main thing we do here is use the isAutomationTestsRunning flag in order to decide which app delegate object to use.

We can use this to create a simple helper method on UIApplication to tell if tests are running:

extension UIApplication {

    static var isAutomationTest: Bool {
        guard let delegate = UIApplication.shared.delegate, delegate is AutomationTestsApplicationDelegate else {
            return false
        }
        return true
    }
}

Then all we need to do is create the correct session depending upon whether or not tests are running and use it wherever we usually create the client:

func makeClient() -> Client {
    let session: URLSessionManaging = UIApplication.isAutomationTest ? AutomationTestsURLSession() : URLSession.shared
    let client = Client(session: session)
    return client
}

With this, we already have the core of what we need to stub requests in our tests. However, using it as is would place a certain burden on developers; in order to stub requests, they first need to know what requests are made during a test, something which could potentially be a lot. Then they would need to manually make the requests, e.g. in Postman; save the responses to a file; and manually specify which stub should replace which request. The implementation currently lacks the API that would enable this final requirement, but since I’ve promised you automatic recording/replay functionality, I’ll skip straight to the good stuff!

Recording Stubs

What we’re aiming for is a simple API that takes the burden off developers and enables them to do network stubbing quickly. Our goal is to be able to call one API for recording, run our test, and if it succeeds, then switch to another API for replaying the recorded stubs — something like the record/replay functionality that is built into Xcode automation tests themselves.

To achieve this, we rely on an important mapping throughout: URLRequest -> Stubbed Response. We will basically use the URLRequest to construct a path on disk to the stubbed response. We can then use this path to save responses to disk as JSON files when we’re in “record mode” and load them back when we’re in “replay mode.”

The gif above shows stubs being recorded for the test_singIn test.

Environment Setup

Before we get started with the automation and application side setup, it’s worth spending a little time on the part that binds the two together. We will need to pass data between the two processes, and because the environment and launchEnvironment dictionaries of ProcessInfo and XCUIApplication, respectively, are just “stringly typed,” we can add some syntactic sugar on top to make the whole thing a bit more type-safe:

public enum EnvironmentKey: String {
    case stubbedTestName
    case networkStubsDir
    case recordMode
}

public enum RecordMode: String {
    case recording
    case replaying
}

final class Environment {

    private let processInfo = ProcessInfo.processInfo

    subscript(key: EnvironmentKey) -> String? {
        return processInfo.environment[key.rawValue]
    }
}

extension XCUIApplication {
    func setEnvironmentValue(_ environmentValue: String, forKey key: EnvironmentKey) {
        var launchEnv = launchEnvironment
        launchEnv[key.rawValue] = environmentValue
        launchEnvironment = launchEnv
    }
}

Here we basically define an EnvironmentKey enum that allows us to wrap the environment and launchEnvironment dictionaries of ProcessInfo and XCUIApplication in order to be able to pass data between the app and tests in a type-safe way. Along the way, we’ll discover what the various environment keys are used for, but most should be fairly self-explanatory.

Automation Tests Setup

We need to pass the name of the running test through to the app so that our AutomationTestsURLSession can construct the directory in which to record/replay stubs. We can benefit from the fact that each test in a given test case is essentially a new instance of the test case. This is how XCTest helps ensure that individual tests don’t interfere with each other. In fact, for an example test case called, say, SignInFeature, if you print self.description during a test, you will see something similar to the following:

-[SignInFeature test_signIn]

The output above reflects the test case name (SignInFeature) and the actual running test (test_signIn). This is perfect for our needs because it provides us with an easy way to associate a test name with the requests that are made during the test:

class SignInFeature: XCTestCase {

    private var app: XCUIApplication!

    override func setUp() {
        super.setUp()
        app = XCUIApplication()
        app.launchEnvironment = ["IsAutomationTestsRunning": "YES"]
        app.startRecordingStubs(for: self)
        app.launch()
    }

    override func tearDown() {
        app = nil
        super.tearDown()
    }


    func test_signIn() {
	    //...
    }
}

We implement startRecordingStubs as an extension on XCUIApplication, passing self so that we can use self.description to get a name for the test. Although relying on the output from self.description may seem brittle, its convenience was too good for this developer to ignore:

extension XCUIApplication {

    func startRecordingStubs(for testCase: XCTestCase, file: StaticString = #file, line: UInt = #line) {
        setupFileStubs(for: testCase, recordingMode: .recording, file: file, line: line)
    }

    private func setupFileStubs(for testCase: XCTestCase, recordingMode: RecordMode, file: StaticString = #file, line: UInt = #line) {
        let testName = String(describing: testCase)
        setEnvironmentValue(recordingMode.rawValue, forKey: .recordMode)
        setEnvironmentValue(testName, forKey: .stubbedTestName)
        if let networkStubsDir = networkStubsDirURL(file: file, line: line) {
            setEnvironmentValue(networkStubsDir.path, forKey: .networkStubsDir)
        }
    }
}

The code above basically sets test_signIn into “record mode” so that the app knows to start recording network requests during its execution. We’ll later use stubbedTestName and networkStubsDir EnvironmentKey cases from the app to construct the path to the stubs on disk. The .networkStubsDir key will point to the root directory where all stubs are stored. The .stubbedTestName key will point to the name of the currently running test, used to create the path to the directory for all stubbed requests for this test.

For the implementation of networkStubsDirURL, we take advantage of the default #file argument passed into every method, which gives the absolute path to the current file on disk. Using this, we can construct a new path for where we will record our network stubs, relative to some directory that we know will always be there, like AutomationTests/ in the example below:

private func networkStubsDirURL(file: StaticString, line: UInt = #line) -> URL? {
    let filePath = file.description
    guard let range = filePath.range(of: "AutomationTests/", options: [.literal]) else {
        XCTFail("AutomationTests directory does not exist!", file: file, line: line)
        return nil
    }
    let testsDirIndex = filePath.index(before: range.upperBound)
    let automationTestsPath = filePath[...testsDirIndex]
    return URL(fileURLWithPath: String(automationTestsPath)).appendingPathComponent("NetworkStubs", isDirectory: true)
}

App-Side Setup

In order to record the network responses in the app when the tests are running, we need to add a bit of functionality to our AutomationTestsURLSession:

class AutomationTestsURLSession: NSObject, URLSessionManaging {

    private let environment = Environment()
    private let _session = URLSession(configuration: .default)
    private let testStubsRecorder = TestStubsRecorder()

    func dataTaskFromRequest(_ request: URLRequest, completionHandler: @escaping CompletionBlock) -> URLSessionDataTasking {

        if environment.isRecordingStubs {
            let completion = testStubsRecorder.recordResult(of: request, with: completionHandler)
            return _session.dataTask(with: request, completionHandler: completion)
        }

        if shouldStub(request) {
            return AutomationTestsDataTask(request: request, completion: completionHandler)
        }

        return _session.dataTask(with: request, completionHandler: completionHandler)
    }


    private func shouldStub(_ request: URLRequest) -> Bool {
        return NetworkResponseStubStorage().stubExists(for: request)
    }

}

The logic of dataTaskFromRequest(_:completionHandler:) should be easy enough to read:

If we’re recording, record the stub and return the standard URLSessionDataTask to the application.
Otherwise, if we should stub the response, return an instance of our AutomationTestsDataTask, which will do the stubbing.
Otherwise, bypass the AutomationTestsURLSession by immediately returning a standard URLSessionDataTask to the application.

We’ve added three new private properties. The first is an instance of our Environment class. It has a new property, isRecordingStubs, which informs the AutomationTestsURLSession of whether or not it should record the responses from the network. This is based on the call we made back in setUp of our test case, i.e. app.startRecordingStubs(for: self):

var isRecordingStubs: Bool {
	guard let recordModelValue = self[.recordMode] else { return false }
	return recordModelValue == RecordMode.recording.rawValue
}

The second property, _session, is an internal URLSession property. We use this when we want the network response to complete as normal, i.e. in points 1 and 3 above.

The final property, testStubsRecorder, is an instance of a new class, TestStubsRecorder, which actually saves the responses to disk:

typealias NetworkResponse = (data: Data?, response: URLResponse?, error: Error?)
typealias CompletionBlock = (Data?, URLResponse?, Error?) -> Void

final class TestStubsRecorder {

    private let environment = Environment()
    private let fileManager = FileManager.default

    func recordResult(of request: URLRequest, with completionHandler: @escaping CompletionBlock) -> CompletionBlock {

        func completionWrapper(networkResponse: NetworkResponse) {
            record(networkResponse: networkResponse, for: request)
            completionHandler(networkResponse.data, networkResponse.response, networkResponse.error)
        }

        return completionWrapper
    }

    private func record(networkResponse: NetworkResponse, for request: URLRequest) {

        // handle errors

        guard let testName = environment[.stubbedTestName] else { return }

        guard let data = networkResponse.data else { return }

        // create dir for test case if it doesn't already exist
        createTestCaseDirIfRequired(forTestName: testName)

        // create and return path where stub JSON file will be saved (inside the test case dir)
        let stubPath = makeTestStubPath(forTestName: testName, request: request)

        fileManager.createFile(atPath: stubPath.path, contents: data, attributes: nil)
    }
}

The main method we’re interested in here is recordResult(of:with:), which our AutomationTestsURLSession calls whenever “record mode” is enabled. It uses a local function which wraps the passed completion block. This is the completion block that will return network responses to the application. We actually return this wrapped function and pass it as the completion parameter of the internal URLSession’s URLSessionDataTask that our AutomationTestsURLSession holds so that when the data task’s completion block gets called, our recordResult(of:with:) method runs first, thereby writing the results to disk. This method then internally calls the URLSessionDataTask’s completion block so that the data can also be returned to the app as normal. The implementations of createTestCaseDirIfRequired and makeTestStubPath are omitted for brevity.

One thing not explained so far is the AutomationTestsURLSession’s shouldStub method. This makes use of a NetworkResponseStubStorage object, which is a simple object that makes use of the Environment object and FileManager in order to construct the file paths to the stubbed responses on disk. If shouldStub returns true, we return an instance of our AutomationTestsDataTask, which actually does the stubbing:

final class NetworkResponseStubStorage {

    private let environment = Environment()
    private let fileManager = FileManager.default

    func stub(for request: URLRequest) -> Data? {
        // return stub
    }

    func stubExists(for request: URLRequest) -> Bool {
        // check if stub exists
    }
}

We can take advantage of the NetworkResponseStubStorage object just introduced to update the resume method of AutomationTestsDataTask:

func resume() {
    if let jsonData = NetworkResponseStubStorage().stub(for: request),
        let url = request.url {
        let response = HTTPURLResponse(url: url, statusCode: 200, httpVersion: nil, headerFields: nil)
        completion(jsonData, response, nil)
    }
}

You might notice that, for the sake of brevity, we are hardcoding the statusCode parameter of the response object. If your app explicitly handles these status codes, you may need to extend the solution to also record/replay the status code for a given request.

The only remaining thing to do is actually return the stubbed response during test runs. In order to do this, we switch the callback in the setUp method of SignInFeature from app.startRecordingStubs(for: self) to app.replayStubs(for: self). The implementation is a simple one-liner added to the extension on XCUIApplication where we implemented startRecordingStubs(for:):

func replayStubs(for testCase: XCTestCase, file: StaticString = #file, line: UInt = #line) {
    setupFileStubs(for: testCase, recordingMode: .replaying, file: file, line: line)
}

So there we have it. With a relatively small amount of code, we’ve been able to design a powerful feature that should go some way to alleviating flakiness in automated tests in a way that is easy to use and saves development time.

Conclusion

In this post, we first demonstrated how it’s possible to stub the responses to network requests made during an automation test in order to improve the stability of our tests with just a small amount of code. We then showed how the recording and replaying of these stubs can be automated using a simple API in order to remove the burden from the developer and enable this stubbing to be done quickly.

References

How to Switch Your App Delegate for Improved Testing

PageRank in Spark

Wed, 24 Jan 2018 00:00:00 GMT

SoundCloud consists of hundreds of millions of tracks, people, albums, and playlists, and navigating this vast collection of music and personalities poses a large challenge, particularly with so many covers, remixes, and original works all in one place.

For search on SoundCloud, one of the ways we approach this problem is by using our own version of the PageRank algorithm, which we affectionately refer to as DiscoRank (Get it? Disco as in discovery and Saturday Night Fever?!).

The job of PageRank is to help rank search results from a query like finding all Go+ tracks called “royals.” At first glance, this task might seem trivial. The first result is, and should indeed be, Lorde’s original song, “Royals.” However, there are plenty of covers and remixes of this track, which leaves us with questions like: Which ones should we show at the top and in which order? What about other tracks in our catalog that have the word “royals” in them? Where should they be in our search results list?

Our use of PageRank started in the summer of 2012 when we introduced SoundCloud Next (now our current website). In the years since, we have grown exponentially, and our implementation of PageRank has had to scale with that enormous growth in several ways — from an in-memory Java implementation, to being distributed in Hadoop using the vanilla MapReduce API, and more recently, by using Scalding. In the summer of 2017, we made the switch to our latest version of PageRank, which runs in Spark, and now we’re open sourcing our implementation as a reusable library!

But before implementing yet another version ourselves, we first tried out the built-in PageRank from Spark. Unfortunately, we quickly realized it didn’t support the graph we were using, as we have nodes with no out-edges, some nodes with no in-edges, and edge weights that are not uniform.

After this first try, we decided to implement PageRank ourselves and tried a variety of the Spark APIs, including GraphX. Although our GraphX implementation was working in small-scale test cases, when we ran at full-scale, we encountered several bugs that are currently unresolved. Based on this experience, we decided that a simple approach using the RDD API would suit us best. We spent about an equal amount of time on implementation and on performance evaluation and tuning, but in the end, Spark was a perfect fit for this job, and we were able to utilize many of the features Spark is known for, such as in-memory data storage, easy parallel computation, and the implementation of iterative learning algorithms.

As a result of this work, we have made improvements to the performance, scalability, and general flexibility of our PageRank implementation. And because it’s able to perform one iteration of PageRank in approximately three to five minutes on a graph of more than 700 million nodes and 15 billion edges, this new implementation has enabled us to update ranks in search more frequently throughout the day, in addition to preparing us for continued growth over the coming years.

With open sourcing our implementation of PageRank, we hope you can benefit from the work we’ve put into this for your own use cases, from search to recommendations. Go check out the project, try it out, open issues, and get involved!

Insights from SoundCloud’s DeveloperBridge Trainee Program

Thu, 18 Jan 2018 00:00:00 GMT

Back in 2016, the SoundCloud People Team collaborated with engineering management in an effort to bring more junior engineers into the company. The result was DeveloperBridge, a paid junior engineering program that ran for 12 months and was based out of our Berlin headquarters.

Trainees gained hands-on experience under the guidance of our engineers, and by working on a variety of assignments in selected teams, they became familiar with different types of engineering tasks and technologies. This, in turn, helped them develop a feel for the areas they enjoyed working in the most. The overall aim of the program was to help the participants develop and then hire them afterward into full-time roles at SoundCloud, while improving diversity in Engineering at the same time.

Now with their traineeship year coming to a close, we’ve asked the participants to reflect on the experience, what they learned, how they feel about their accomplishments, and where they see themselves headed next in their careers as software engineers.

Suzanne Wood

The DeveloperBridge program exposed me to different teams so I could both improve my programming skills and learn more about how SoundCloud enables users to share and discover music. I learned new programming languages and developed confidence in my ability to pick up more.

Over the course of the year, I worked on web development and data processing projects. I created features for and improved the UX of an internal web app using React, Typescript, and JavaScript, and I made a web service using Ruby, JavaScript, Mustache, and regular expressions. I also created ETL (Extract, Transform, Load) processes using different methods to determine which was the most efficient, and I used Spark and SQL together with the Tableau analytics platform to enable the flexible display of data.

During the program, I also learned new approaches to tackling problems. For example, it is good to make small iterative changes to code, because if you make too many changes at once, errors can become blockers. Through collaboration, we shared ideas and approaches, leading to a more maintainable and consistent codebase. I also really like the hardworking yet relaxed office atmosphere — especially the office dogs!

I now have a feel for what programming is really about: It is abstract and creative, and it requires a lot of thought to design a system well. This appeals to me, so I will continue to pursue it as a career. In particular, I’d like to continue with frontend or full-stack web development roles — I like delivering something tangible that people can use and appreciate, with clear and consistent designs.

Suzie has accepted a full-time position on the SoundCloud Payments Team.

Find Suzie on the web here.

Mike Smith

Some things about joining SoundCloud for the DeveloperBridge program met my expectations (ping pong tables! bean bags!), but many didn’t. I didn’t anticipate how much work goes into doing seemingly simple things. I had expected something more structured, but the refreshing reality was that I had the opportunity to pursue what I was interested in and motivated by, and I was allowed to develop at my own pace, which suits my working style completely.

At first, Scala was a challenge. But the support I got and the freedom I was afforded led me to be surprised on a daily basis about how much I can understand.

The people I worked with had years of experience, and they were well-versed in being productive in technology. Observing their behaviors and attitudes gave me insight into what it takes to work at a high level — something that is difficult to teach. I also found I learned most from those with more diverse backgrounds — who could sympathize with being the “odd one out” on the team — as opposed to the people who were the best technical coders. If the company is to continue developing junior talent, fostering diversity must be a key aspect.

Extensive experience with the React/Redux/Typescript/Webpack stack has put me in the position where I’m able to advise other people on these technologies, allowing me to, in some way, pay forward all the help I’ve had from other people along the way. I now consider myself a professional software engineer, which is something I would not have believed a year ago. I’d love to continue developing my skills and eventually use my experience to teach others who want to take the same path I did.

Mike has accepted a full-time position on the SoundCloud Core Clients Team.

Find Mike on the web here.

Lucy Clarke

A few people I knew who worked at SoundCloud told me I would be working with some of the kindest and smartest people they knew — and they were right. I was super excited about the position, having come from a music industry background and having used SoundCloud as a music discovery tool for years.

For most of the year, I have been working on an RSS service for podcasters. This service is built in Scala and replaces an older service that was written in Clojure. I had no previous experience with Scala and had zero exposure to lisp languages. But along the way, I became competent in Scala and was able to work my way through Clojure code. Podcasting is a segment I care a lot about, and I am proud of the work I did on a useful service for podcasters that serves a lot of traffic.

As the RSS service for podcasters was essentially a new project, I was able to be involved with all aspects of building the service — from coding, to integration with CI, Docker, and Kubernetes, to alerting monitoring and scaling traffic between the old and the new service for fast feedback. As the previous service had not been touched in some time, we were also able to reconsider previous decisions around logic. I know more about podcasting, XML, aggregators, and iTunes than I thought possible.

Surprisingly, my pun game also really improved.

I want to continue to develop my skills as a backend engineer. I would love to keep working on products that enable audio creators to do their best work.

Lucy has accepted a full-time position on the SoundCloud Royalties and Reporting Team.

Find Lucy on the web here.

Thea Kupler

My “Rails Girls Summer of Code” placement was hosted by SoundCloud, and this was followed by an internship with SoundCloud, so I was already familiar with the company infrastructure and culture, and I was excited to have found an environment in which I felt comfortable and supported.

I worked on several teams and on several projects during the program. I worked on building a web app to view and compare event data stored in Hadoop, rewrote Java components in Scala, added functionality to a React frontend, and contributed to a Ruby web app. For my final project, I am contributing to a web app written in Clojure. My work includes generating folders and files, zipping everything up, and pushing it to GitHub automatically. This also reminds me of another a thing I have learned and will always still be learning: git, git, and git.

I am proud of everything I accomplished. I took the opportunity to give demos in front of the company. I got the support to share and celebrate my pride and work.

I gained confidence in what I am capable of, which helps me focus on learning, improving, and tackling new challenges, rather than wasting my energy on feeling like an imposter. I still remember the moment when I suddenly started explaining and defending my code choices rather than just silently implementing what I was told to do differently.

I’d love to shift my focus more toward backend web development — either building up the knowledge I already have, or learning something completely new, especially working for a non-profit organization or one in the field of sustainability. Long term, I want to foster my passions in a more balanced way: less coding and hacking, more coaching and healing.

After 18 months of self-led training through internships and this program, Thea has chosen to leave SoundCloud to take some time to pursue other personal and professional options.

Find Thea on the web here and here.

Marina Mircheska

I saw the program as a perfect start to my career in this industry. My interests as a Computer Science major were scattered across a few areas, and I was delighted to have the opportunity to find out what fit best. Moreover, this was about to happen in a very cool Berlin startup with an international team of specialists.

For the better part of this past year, I worked on deep learning for content-based music recommendations. Deep learning is a hot topic, and I got an exciting chance to work with real data and have all the computational power I need.

The DeveloperBridge program was my first opportunity to apply an abstract concept from textbooks and research papers to real data, and the challenges I faced — surprisingly — were not only on the theoretical side. It took some long thinking about design and optimization, experimenting and testing every step and decision while laying the infrastructure of the project. It took a while before training the neural network and getting initial results, and then a different kind of challenge arose: how to debug a machine learning algorithm.

I also worked on a real-time recommendation service, thereby improving my Go skills; built Tableau dashboards for analytics purposes; wrote ETLs using Scala and Spark; and did some exploratory data analysis using Python Notebooks.

Furthermore, I learned how to be part of team meetings and participate in discussions efficiently, while at the same time keeping communication healthy, fair, and respectful. My supervisors and my mentors set an example for patience, generous teaching, and efficient collaboration, and because of that, I will definitely be a better team player in the future.

During my master’s degree, I focused on Data Science and Algorithmics, and my time at SoundCloud helped me gain the experience of a junior data scientist. Considering my interests, I will try to build myself as a Machine Learning Engineer in the coming years.

Marina has accepted a full-time position on the SoundCloud Discovery Team as a data scientist.

Find Marina on the web here.

How SoundCloud Uses HAProxy with Kubernetes for User-Facing Traffic

Thu, 07 Dec 2017 00:00:00 GMT

A little less than two years ago, SoundCloud began the journey of replacing our homegrown deployment platform, Bazooka, with Kubernetes. Kubernetes automates deployment, scaling, and management of containerized applications.

The Problem

An ongoing challenge with dynamic platforms like Kubernetes is routing user traffic — more specifically, routing API requests and website visits from our users to the individual pods running in Kubernetes.

Most of SoundCloud runs in a physical environment, so we can’t leverage the built-in support for cloud load balancers in Kubernetes. At the edge of our infrastructure, a fleet of HAProxy servers terminates SSL connections and, based on simple rules, forwards traffic to various internal services. The configuration for these servers is generated and tested separately before it is deployed to these terminators. But because there are a lot of safeguards built in, this process takes a long time to complete, and it cannot keep up with the rate at which pods in Kubernetes are moving around. The fundamental challenge of getting user traffic to the pods that keep moving around is a mismatch between our static termination layer and the highly dynamic nature of Kubernetes.

The Process

To address this challenge, we initially configured the terminator layer to forward HTTP requests to a separate HAProxy-based Ingress controller, but this solution did not work well for us, as the Ingress controller was intended for low-volume internal traffic and is not very reliable.

Our users generate a lot of traffic, and every problem in the traffic stack means that SoundCloud is not working for someone. Between the Kubernetes Ingress and the terminator configuration, we had two layers of Layer 7 routing that needed to match up and often didn’t. This was frustrating for us as developers and resulted in additional work for us. We also knew that the Ingress controller would not be able to handle the long-lived connections used by some of our clients.

When SoundCloud engineers build applications, we use a custom command-line interface that generates the Namespace, Service, and Deployment. Optionally, it generates the Ingress Kubernetes objects from command-line flags. We added a flag to the tool that generates these objects, which changes the Service to the NodePort type.

Kubernetes allocates a port number that is not yet used in the cluster to this service, and it opens this port on every node in the cluster. Connections to this port on any of the nodes are forwarded to one of the instances for this service. (When we generate the Kubernetes objects, there is a one-to-one correspondence between Service and Deployment objects. For the sake of brevity, we’ll gloss over the details of ReplicaSet, Pod, and Endpoints objects in Kubernetes here.)

Note that this is irreversible for a given Service — Kubernetes does not allow removing the node port from a service. We are still looking for a solution for this — thus far, this has only happened early enough in the service lifecycle that we could delete and recreate the service, but this could lead to service interruptions later on when the service is used in production.

To configure their applications to receive traffic from the internet, application developers declare the public hostname and path, as well as the cluster, namespace, service, and port name for the applications. The systems routing public traffic to applications, such as SSL terminators, CDN distributions, and DNS entries, are configured based on this declaration.

The Implementation

When the terminator configuration is generated, the script queries the Kubernetes cluster to find the assigned node port for each service, along with the list of Kubernetes nodes. Originally, we added all nodes into the terminator configuration, but this turned out to be a problem.

Each terminator independently checks the health of each port on each node. With several dozen terminators and hundreds of Kubernetes nodes, this amounted to tens of thousands of health checks being generated every second. Because of the indirection through the nodes, this was not affected by the size of the service itself, so even services that would normally get very little traffic needed a lot of resources just to be able to satisfy these health checks.

We needed to reduce the number of nodes configured for each service, but we also did not want to send all the traffic through only a limited set of nodes to avoid them becoming a scaling bottleneck. One possible solution would be to simply pick a number of nodes out of the list at random, but this would mean that the configuration completely changes every time it is generated, obscuring the real differences.

Instead, we decided to use rendezvous hashing between the service name and the node address to pick a fixed number of servers per backend. With this method, a different set of nodes are selected for each service, but the selection is always the same as long as the nodes do not change.

We chose a high enough number of nodes so that we didn’t need to worry about one or two nodes going away or the nodes overlapping between different high-traffic services.

To replace a node, we simply restart the pipeline that generates and deploys the terminator configuration. This takes several hours, but it is fully automated. Because each service is routed through a limited number of nodes, we have to take care not to take too many nodes out of service at once. This means we can only replace a limited number of Kubernetes nodes per day, but as of yet, it hasn’t been a problem.

The Node Agent

During short-term maintenance, such as when rebooting nodes for kernel upgrades, we needed the ability to gracefully drain a node. We wrote an agent for the HAProxy agent-check protocol that listens on a fixed port on every node. For the sake of simplicity, we decided to always remove traffic and pods at the same time. Now, when the node is cordoned in Kubernetes, preventing new pods from being scheduled onto it, the agent updates the HAProxy server state to shift traffic away from this node.

From time to time, we also need to shift traffic between different deployments of a service on the same cluster or between clusters. To support this, we extended the agent. By adding some annotations on the Kubernetes Service object, we can instruct it to open another per-service agent-check port that includes not only the node maintenance status, but also the desired weight of this service.

The terminator configuration generation picks this up and configures the agent check accordingly. Application developers can now add multiple Kubernetes backends to any public host and path. For each backend, we select a number of nodes using rendezvous hashing as before and combine the results. By changing the Service annotations, the weight for each backend can then be changed within seconds.

The Future

We are quite happy with the NodePort-based routing, but there are some caveats.

For one, the load distribution between the instances of an application is not very even, but we compensate by slightly over-provisioning the corresponding apps. At this point, the cost of the increased resource usage is minor compared to the cost of engineering a better load balancing mechanism.

Additionally, operating Kubernetes in physical data centers is a challenge. There are standardized solutions for most use cases on cloud providers, but each data center is slightly different — especially if you are not starting from scratch, but rather integrating Kubernetes into an existing infrastructure.

In this blog post, we described how we solved one particular problem for our particular situation. What’s next? The questions we are thinking about currently revolve around having multiple Kubernetes clusters in different locations, and determining how to route traffic to them in a way that results in each user having the best experience possible.

How have you solved this problem? We would love to hear how you run Kubernetes. Leave a comment below!

Leveraging frameworks to speed up our development on iOS - Part 1

Mon, 16 Oct 2017 00:00:00 GMT

Growth in code bases come with exciting scalability challenges. As the size of our iOS codebase and team at SoundCloud grew, we faced challenges: long compile times and conflicts. Our productivity started to suffer as a result. We took inspiration from the work done in the backend (Building Products at SoundCloud) and applied it to mobile development. The main goal was to get back to a state where development is fun, fast, and would scale as the number of contributors grew. We modularized our iOS project by splitting it up into modules with well-defined responsibilities and public interfaces that interconnect them.

It’s been a little over a year since we started our journey towards a modularized iOS app and we’re very happy with the results that we’ve achieved so far. In this series of posts we’ll talk about the modularization journey, the motivation behind deciding for modularizing the app, a technical overview of the approach, and the impact it has on developers when they work on new features.

Context

Originally, the iOS app was organized in two targets: one target that compiled the main app, and another target that compiled and ran the unit tests. The source code was mainly Objective-C, although some developers had already started adding Swift to the codebase. Moreover, the project used CocoaPods to fetch and statically link around 20 dependencies. The product was growing at a good pace and so did the size of the two main targets. The build time became very high, and the lack of some control over how things were getting imported triggered clean builds when a single file was modified. Waiting for the build gave us the perfect amount of time to grab a coffee, but we’d have a serious problem with our caffeine intake if we didn’t do anything about it. In a team of 10 people, doing 20 clean builds per day, each taking 7 minutes, the cost was 1400 minutes, 23.3 hours, or 3 days of work. These build were a waste of time and money. Also, delivering features took too long compared with our other platforms.

On top of all that, CocoaPods and our git branch-based workflow made the problem worse as we had to run pod install on most branch switches (It’s worth noting that CocoaPods has improved a lot since. The latest version prevents Xcode from invalidating its build cache after running pod install.).

Why modularization

Most of our development time went into the compilation of the huge app and tests target. We undertook research by looking at peer companies and conducting interviews. After careful research we put all the options on the table and decided towards splitting the iOS app into modules. With smaller compilation targets, modules would take much less time to compute and run, and developers could quickly iterate over better solutions. Moreover, smaller targets would encourage good API design, and allow us to leverage tools like Playgrounds and example apps to iterate on features without compiling the main app.

Our research showed that most companies going through this transition had bigger teams than us, including teams dedicated to building and maintaining tools to facilitate their transition. We had to come up with a plan to gradually transition to building modular features, iterating over the new architecture while still allowing the team to build features.

Some companies are very successful using React Native, including SoundCloud with Pulse. We ruled out that option because the framework comes with integration costs that we could not assume in the listener application: tooling maintenance and training for instance. However, React Native makes a lot of sense for some features and we may reconsider it in the future.

Advantages

Clean interfaces (APIs)

When all the code is in the same target, having access to any other class is a matter of importing a header (Objective-C), or having an internal access level (the default one in Swift). It’s flexible, but at the same time dangerous since it allows everyone to create dependencies with almost no control. Having no control over how dependencies get imported leads to a very messy dependency graph, coupled code, and eventually to unpredictable states propagating through that graph. In a small app, with a small team, you can avoid these issues with code review. However, with our application and our team this was too hard to do.

Modules help you prevent these issues by exposing an API that you as a developer need to define. As part of your workflow, you need to define how the API of your module will look like and how consumers of your API will interact with it. In Objective-C that means specifying which headers will be public, and in Swift, which components will have the public access level. An API defines a contract in the same way REST APIs do, and developers who consume your API will have to comply with it.

Example apps

Since modules include standalone features they can include example app for testing the features without the main iOS application. We recently built the new home screen in a separate app using test data, which allowed us to test different scenarios:

Having tracks with long titles that exceed the limit of the label.
Having tracks with no artwork.
Having no data because of a connection error.

With example apps, you can iterate on your feature with fast build and test times, designing the public API that the main application will eventually use. Following with the example above, the home feature would expose a view controller that the main application would push on the view hierarchy.

The image below shows the structure of one of our modules that include an example app that you can experiment with.

Playgrounds

Similarly to an example app, Playgrounds allow you to import your module and use its APIs. Changes run in real time and you can even see UI components there. We found Playgrounds very useful for documenting and onboarding.

You can check out this episode that shows how Kickstarter leverages Playgrounds to develop their features and try them with different scenarios.

Fewer merge conflicts

Having a single project that indexes all the files and configurations in your projects leads to conflicts quite often when more than one branch is modifying the project file. By having different projects that contain the modules we still encounter conflicts but they are less probable. As a result, we spend less time solving conflicts on git when someone merges first and also modifies the project file.

Exploring new patterns

Working on modules allows developers to decide about the code architecture and the patterns that they want to use internally. They can even choose the language, for instance, using Objective-C instead of Swift, or maybe even React Native in the future. However, the public APIs of these modules need to comply with a contract to ensure there’s consistency across all our public APIs. We’ll talk in more detail in the upcoming articles about what the contract looks like.

Moreover, we’ve tried to commit to Cocoa patterns for the interfaces of core components that modules have access to. Since developers are familiar with those patterns (delegate, completion block…) they don’t need to spend time figuring out how to hook those components into their features.

Pitfalls

Maintenance

Maintaining a modular setup becomes a cumbersome process when you have a lot of modules. .xcconfig files are a good tool to ensure that all targets share the same configuration, yet it’s not enough. When it comes to specifying the targets that you want for each project, the schemes that are available, and the configuration of those schemes, there’s no tool or API that you can leverage to automate the process of creation and maintenance. We could use a tool like CocoaPods but they would decide the structure of our modules, which is too restrictive for our requirements. Buck is another option that we considered, but it didn’t fully support Swift back then and would have required a lot of investment that we couldn’t afford.

We are internally exploring some tools to automate the process of creating and maintaining new modules.

External dependencies

In our experience, having a large amount of external dependencies brought unnecessary complexity and work, so we tried to reduce them. However, there are some dependencies that are necessary, which we need to fetch and link from the app. Some examples of these dependencies are Crashlytics, Firebase or Facebook. Each dependency is distributed in a different way. Some provide a .podspec that you can use from CocoaPods. Some others publish the compiled framework/library and headers that we can fetch and link from the app. If they support Carthage, they include a project that gets compiled by Carthage and whose output .frameworks gets copied to the Carthage build folder. There’s no standardized way to distribute dependencies and that makes things really hard, especially with no dependency management tool.

There’s some manual work that we had to do in this area to generate a compiled version of the dependency that we can use, and that includes information about the version. Unfortunately, there’s no automation that we could do since as I mentioned, there’s no homogeneity in the setup of the dependencies.

In the next post of this series, we’ll dive into technical details. We’ll present the current project architecture, guiding you through the principles of it, as well as some iterations the architecture has gone through based on the feedback from the team. Stay tuned!

SoundCloud's Data Science Process

Wed, 04 Oct 2017 00:00:00 GMT

Here at SoundCloud, we’ve been working on helping our Data Scientists be more effective, happy, and productive. We revamped our organizational structure, clearly defined the role of a Data Scientist and a Data Engineer, introduced working groups to solve common problems (like this), and positioned ourselves to do incredible work! Most recently, we started thinking about the work that a Data Scientist does, and how best to describe and share the process that we use to work on a business problem. Based on the experiences of our Data Scientists, we distilled a set of steps, tips and general guidance representing the best practices that we collectively know of and agree to as a community of practitioners.

The process represents an informal agreement amongst the team on how we accomplish data science objectives, and it serves as a basis with which to reflect on how to improve ourselves and our work. When we work on tasks, we use the process to bring higher levels of quality to our work and to improve our ability to communicate with our peers on progress, successes, and failures. The process itself is iterated on and adapted as we see systematic failures appearing in our work, or ways in which we can improve on our work.

One thing worth highlighting is the iterative nature of our work here at SoundCloud. We work daily with Product Managers, Designers, and Engineers and we do so in a highly collaborative and iterative way. As such, we have designed our process to not only allow for iterative work, but to embrace it as a fundamental principle. We aim to deliver high-value work on a regular basis to our stakeholders and to adapt to a rapidly changing environment and available information. Think of it as a Bayesian process — we’re constantly updating our prior!

When you see the steps below, you should keep in mind that these are logical steps of the work process and that the actual workflow contains loops and iterations — an outcome of one step might require revisiting previous steps. In most cases, we expect first iterations to be as small in scope as possible and that completing steps is more important than adding complexity to steps so that we are able to deliver incremental value earlier. As such, following the process in practice should end up looking like the following diagram, as opposed to a single pass through each step once and only once.

Speaking of iterative work, this is our first iteration on capturing our data science process. We’re happy with and proud of what we have collected here, and we’re looking forward to figuring out the next steps. We want to see where the process does not fit and what changes we should make to it. There might be practices that we should add or some that we should remove. We’d also like to introduce a set of values and principles that we can use to guide our decision-making as Data Scientists.

We look forward to learning, adapting and iterating as we gain more experience as a group. Below you will find a condensed version of our Data Science process and we’d love to know what you think about it. How do you work on a Data Science task or problem? Have we forgotten any stages or important attributes of each stage? Is this something you’d like to adopt for your work? Let us know what you think, and join us in the conversation!

Data Science Process

Authors: Özgür Demir, Josh Devins, Max Jakob, Janette Lehmann, Christoph Sawade, Warren Winter

Problem Definition

At the outset, we collaboratively make a definition of the objective, ensuring that all stakeholders are on the same page about what problem we’re trying to solve. To accompany this we define the metrics to measure success, to make sure that we agree on when we are successful. This is similar to the definition of done in agile engineering practices.

Understand the business need (in collaboration with stakeholders)
Challenge and validate assumptions about the business needs
Identify and prioritize subproblems, including metrics
Narrow the scope to minimal size, in order to achieve the business goal

Preparation

Before diving into a potential solution, we take some time to first prepare ourselves and the data. We build some intuition and knowledge of the problem domain, with the support of domain experts and research. We spend some time exploring our data and finding the relevant variables or attributes of the datasets available to us. We finally define our solution space by narrowing down from all possible solutions into the general area. An example would be understanding if a supervised, binary classification is a suitable approach, or if we need a multi-label or multi-class classifier.

Collect all available data, understand if it is sufficient and representative
Exploratory data analysis, including any manual labeling
Get peer feedback on the solution space and any conclusions drawn

Solution Development

Solution development is where we really dig into solving the problem at hand. We create a ranked list of candidate approaches for solving the problem, and after obtaining a simple baseline, we implement one or more prototype(s). As we go, we make sure that we have documentation of the decision-making process (e.g. in a journal of decisions), which serves as a kind of story-line supporting our chosen approaches and solutions, problems or edge-cases that we encountered, quirks with the data or a library, and so forth.

Review prior art, literature, existing implementations, etc.
Establish a baseline, through simple means
Consider trade-offs such as between the goodness of fit, ease of use, complexity, maintenance, community, etc.
Prototype solution(s), evaluating the models (e.g. accuracy, precision/recall) and operational performance (e.g. scalability, runtime performance)
Challenge and validate assumptions that the solution solves the business problem
Get peer feedback on solutions, implementation ideas, and evaluation procedures

Production-Ready Deployment

Once a solution has been validated through offline evaluation (where possible), we prepare and deploy a production-grade implementation. Where possible, this will be deployed as an A/B test first, or launched in “dark mode” on a subset of traffic in order to observe the effects and provide further validation of the solution.

Understand how the solution fits into existing infrastructure
Ensure quality standards are met for runtime performance (e.g. with SLAs/SLOs), automation, testing, documentation, code reviews, etc.
Document how the approach has been validated in production
Monitor quality (semantic and operational)
Communicate results to stakeholders and broader audience (e.g. with a demo)

Validation and Maintenance

Once a solution is in production, we need to keep an eye on things and evaluate the metrics that we started with. We analyze A/B test results and regularly monitor the solution to ensure that over time, it keeps on meeting our expectations and solving our stated business problem.

Monitor performance
Correlate A/B test results with offline evaluation
Regular re-training of models (understand freshness requirements)
Cycle back to the problem statement (e.g. the problem is solved or partially solved, the problem statement needs reformulation)

Deliver software faster by managing work in progress, not by adding overtime

Wed, 27 Sep 2017 00:00:00 GMT

Product development flow (flow) is the rate at which our products are developed, from idea to deployment. Good flow means that products should pass through the development cycle quickly and continuously.

Problem Statement

SoundCloud faces challenges with its product development flow. SoundCloud’s CTO, Artem Fishman, summed up these challenges succinctly during the May 2017 Engineering Town Hall.

“We have a ton of [good] product pressure… We work really hard, but are fundamentally slow to deliver… This resonated everywhere. It takes a very long time to ship something here.”

His conclusion was reached by interviewing over a hundred employees from the Technology Organization. It is clear that employees understand these challenges on a qualitative and intuitive level.

These are classic symptoms of an overloaded product development queue.

The problem was then for project managers to develop quantitative metrics for understanding our product development flow and guidelines for improving the flow issues that the company already knew on qualitative and intuitive levels.

Post Structure

This post describes recent efforts to measure and improve software product development flow at SoundCloud. It demonstrates that we can deliver software faster by managing Work in Progress (WIP) and product quality rather than pushing employees to work overtime.

The second section describes the discovery phase in which we define and measure some variables for improving flow. Project managers used tools from the Project Development KPIs Project (PDKP) to understand the problems in our workflow. Understanding that data, they derived the strategy below for improving SoundCloud’s product development flow.

The company should commit to fewer products at a time
Teams should work on fewer product features at a time
Engineers should work on fewer tasks at a time
Engineering should reduce its code inventory
Engineering should monitor and reduce the number of bugs over time

The third section describes our continuous learning process and our first set of guidelines for improving flow.

The fourth and last section describes preliminary results of our efforts.

Defining success and variables

This section identifies lead and cycle time as metrics of success in product development and motivates their use. It identifies variables that have an impact on lead and cycle time.

Lead and cycle time

To improve business outcomes, businesses define the metrics for the desired outcome then iterate towards achieving that outcome by observing if actions taken improve or worsen the metrics. Members of the Process Optimization Group at SoundCloud use lead and cycle time as two important measures of our success in product development.

Lead time is the time between the initiation and completion of a product’s development. At SoundCloud, this is the time between when a product owner receives a work request to the time when the work is completed.

Cycle time is the time between when the production of a product is started and when it is completed. At SoundCloud this is the time between when an engineering team receives a feature request to the time that the software is deployed to users.

Keeping lead and cycle times low has many benefits, four of which are noted here. The first benefit is that we ship products to users faster, before our competitors. Being first increases the chance that we capture market share.

The second benefit is the reduction of backlog size. Product managers are incentivized not to commit to projects too far in advance. Since requirements for new products change rapidly, a minimal backlog means more responsive plans and product ideas are used instead of outdated ones.

The third benefit concerns the size of products shipped. Focusing on minimizing lead time incentivizes teams to deliver new products as MVPs and smaller changes to existing products. This more granular approach decreases the probability of large failures and over-investment in projects that will not succeed.

The fourth and most important benefit is that SoundCloud can gain information on its product development faster. By shipping faster, we get market information before our competitors. We get feedback on a product earlier in its development cycle. We can make small adjustments in strategies towards our desired outcome rather than large corrections after over-investing in a product that had limited success.

How can SoundCloud improve lead and cycle time? There are several important variables we can leverage.

Variables affecting lead and cycle time

This section presents the definitions of variables affecting lead and cycle time as well as an explanation for the effect of each.

The ratio of WIP to contributors

The ratio of Work in Progress (WIP) to contributors is a measure of how much contributors in a team are multitasking on average. PDKP estimates this by dividing the number of WIP issues in a Jira project by the number of engineers active on that project.

Keeping the ratio of WIP to contributors slightly below 1.0 can decrease cycle time in two ways. First, if engineers develop one product at a time, products will be ready sooner than if they multitask. Say an engineer has two tasks, each of which will take two days. They can either work on both in parallel or one after the other. The schedules of these two options are shown below.

Scheduling two tasks for one engineer

In Schedule 1, the engineer works on both in parallel, task 1 is complete at the end of day 3 and task 4 is complete at the end of day 4. The more the engineer interleaves the work, the longer it takes to complete both tasks. In the extreme case—multitasking within each day—both projects might not be completed until the end of day 4. In Schedule 2, the engineer works on the first and then the second, the first is completed at the end of day two and the second at the end of day four. By working on one thing at a time, the engineers have completed a task earlier without adding work!

The second reason to keep the ratio of WIP to contributors ratio slightly below 1.0 concerns the ability of individual members to change development roles quickly. To understand why, see the idealized model of a team’s workflow below.

Software engineering is an invisible “U-shaped flow cell.”

Engineers need to quickly transition between planning, developing, reviewing, and fixing bugs. They fulfill multiple roles in the software development process. This collaboration pattern is similar to U-shaped flow cells in Lean manufacturing.

Processing tasks in U-shaped cells can be very efficient, but not if the queue of tasks is overloaded. If more tasks are in progress than there are engineers to complete it, items will form queues waiting for attention. For example, say an engineer developing code is overloaded with new feature requests. A queue of feature requests will form. Further, it is likely that they will not be able to review code completed by their teammates, contributing to the formation of a second queue in the review stage. Either cycle time will rise or quality will suffer. Engineers need to switch roles to where queues are forming to push tasks through the queue.

Analogously, overloaded teams with a high WIP to contributors ratio will be less capable of dealing with incoming bug fixes and ad-hoc requests.

The number of products in WIP

“Products in WIP” are products in progress. PDKP estimates the number of products in WIP by counting the number of issues in a Jira project that represent a product. For most teams, these are ‘Epic’ or ‘Story’ issues. Products can be either for internal or external users.

Decreasing the number of products in WIP can decrease lead and cycle time. To illustrate, imagine we have two tasks each of which takes four days and can be shared between two engineers with no overhead. There are two schedules for this work shown below: either each engineer takes one task or they pair on the first, then the second.

Scheduling two tasks for two engineers

In Schedule 1, each engineer takes a single task, both products ship in four days. In Schedule 2, the first task is ready in two days and the second at four days. We’ve shipped the first product two days earlier without adding any work. Pairing adds some overhead, but getting information from shipping a product sooner is often worth it. These same principles apply to the planning phases of product development.

Inventory

In business, inventory is the components of a product that have not been assembled or manifested into a product ready for sale. In engineering, our inventory is anything we work on that results in a feature that is yet undelivered including undeployed code, RFCs, and unfinished Jira issues. PDKP estimates inventory with the number of commits or number of additions and deletions in pull requests in repositories that have been updated in the last three months.

Increasing amounts of code inventory is a symptom of bad flow and has a negative effect on cycle time. The more inventory there is, the more development time has accumulated in work not being seen by our users. Generally, inventory depreciates in value as it ages.

Ideally, we want to achieve what is called a “one piece flow,” a process where all code inventory is actively being worked on. This means that there should be at most one product in progress per team and one task in progress per engineer.

The challenge in accomplishing one piece flow in software development is that code inventory is invisible. In a factory that produces physical goods, inventory is obvious, it stacks up on the factory floor. In a software company, it’s not so obvious.

Software inventory isn’t visible in the same way manufacturing inventory is.

For this reason, we need to take measures to monitor code inventory and make sure it gets shipped in a timely manner.

The ratio of features to engineers

The ratio of features to engineers measures how many features each team member must maintain on average. We estimate it by dividing the number of configuration files indicating an individual feature in a team’s active repositories by the number of engineers active on a team’s Jira board.

Higher ratios correlate with higher rates of bug reports and maintenance overhead. If a team has an abnormally high ratio, the company should consider providing them with more headcount or redistributing their responsibilities to get a more even distribution.

Bugs

Our exposure to bugs is estimated by the number of issues of issue type ‘Bug’ in a Jira project.

Persistent bugs have a negative impact on lead and cycle time. Users and Community Support re-report the same bugs which then have to be reinvestigated before confirming they are duplicates. If engineers build on top of buggy code, the resultant implementation can be flawed, at worst resulting in wasted code that cannot stay in production. Most importantly, bugs make it harder for engineers to learn what is going wrong and to prevent future problems.

Lower bound on time to backlog completion

Multiplying a team’s cycle time by the number of products remaining in their backlog estimates a lower bound on how long it would take a team to complete their backlog.

Having a large backlog has three negative effects. First, it increases our lead time. Planning too far in advance creates a queue of work that teams cannot work through quickly and takes development time away from line managers and engineers. Second, it decreases our flexibility in changing plans in the future. Once something is on the backlog, teams feel committed to deliver them. Third, it decreases motivation.

What did we do about it?

This section describes SoundCloud’s continuous learning and improvement cycle where its project managers collect flow metrics and apply analysis thereof in guidelines to improve product development flow.

For the first set of guidelines, project managers used principles from manufacturing and queueing theory to encourage teams to work on fewer products at a time and engineers work on fewer tasks at a time.

Importantly, the guidelines don’t suggest that teams work harder. They are intended to increase focus, not the total number of hours worked.

Metrics collection

Starting in March 2017, PDKP has collected lead time, cycle time, WIP, bugs, contributor, and code inventory metrics from SoundCloud’s Jira and GitHub organizations.

To date, 18 engineering teams have configured the system to collect lead and cycle time metrics. Inventory metrics are collected for all development in SoundCloud’s GitHub organization.

All metrics are aggregated and presented in the PDKP dashboard and summary dashboard.

Guidelines and motivating analysis

1. The company should commit to fewer products at a time

SoundCloud can reduce cycle time and increase its flexibility in planning new products by reducing the number of products it commits to. Teams had high lower bounds on time to backlog completion, suggesting that overcommitment was a problem.

Of the 18 teams in the PDKP metrics collection, the median lower bound on time to backlog completion is 766 days, or about 2.1 years. While this number could represent misuse of Jira, it suggested an opportunity to shift more time from planning future products to focusing on more immediate products.

2. Teams should work on fewer product features at a time

Provided the work can be parallelized, teams should work on one product at a time in order to maximize flow. The number of products teams had in progress was too high.

Across the 18 teams sampled, PDKP reported the median number of products in progress at once is four. An in-person survey of teams in the Creators Organization reported a similar number.

It is unlikely that there is enough downtime in the development of products that teams needed to have four in progress at any given time to be productive.

3. Engineers should work on fewer tasks at a time

In theory, engineers should work on one task at a time until it is completed to maximize flow. The ratio of tasks in WIP to engineers was too high.

Across the 18 teams sampled, PDKP reported the average ratio of WIP to engineers was consistently over 2.0.

Of course, it is not always practical to work on one thing at a time while developing software. For example, teams with long running batch jobs will have enough downtime while waiting for the job to finish to do meaningful work on other tasks. However, like teams with too many products, it is unlikely that the majority of teams with high ratios in the sample have this specific problem.

Importantly, high engineer utilization is not a goal. One important result of queuing theory is that not all nodes in the queue have to be fully utilized to maximize the throughput of the queue overall. In fact, it’s good if our engineers have a little idle time.

4. Engineering should reduce its code inventory

SoundCloud has a high amount of code inventory. Reducing it is a big opportunity for us to improve these metrics.

Over SoundCloud’s GitHub repositories updated in the last three months at the time of the first metrics collection, there have been between 1,200 to 1,500 commits in open pull requests representing 200,000 to 300,000 lines of code. The average age of this inventory was 65 days old. On average, teams have 22 commits in open pull requests. That’s a lot.

Teams at SoundCloud have had success in increasing flow by decreasing inventory. The Content ID Team reduced its inventory over a period of two months, bringing the daily average number of commits in open PRs from 46 to less than 5 and the average age of those commits from 14 to less than 1 day. During this time, the team’s cycle time dropped from 12 days to 3 days.

Given this longitudinal experimental design, it’s only possible to show correlation—not causation—between reducing inventory and cycle time. However, the correlation is positive and suggests further effort on this front could help.

Importantly, code review should not be rushed. Rather, it is better to carefully and thoroughly review code as it reaches the review stage promptly or use a continuous integration review process where merging code into the master branch is not blocked by code review.

5. Engineering should monitor and reduce the number of bugs over time

Monitoring and reducing the number of bugs in a team’s system has a positive impact on lead and cycle time. SoundCloud is currently implementing a standardized bug reporting process. This is another opportunity to speed up both development and the learning feedback cycle.

Guidelines for management

The number of products product management commits to at once caused teams to overload their product development queues. Engineering managers then overloaded engineers by accepting multiple products into WIP at once.

Overloaded product queues propagate from product managers to project managers to engineers.

Note that the propagation of queue overload can cause both the single engineer and multiple engineers scheduling problems.

Product and engineering management must lead and support changes to the way we work in order to improve flow, especially since some of these practices have become habitual in the company. Accordingly, the following guidelines were given to product managers.

1. The company should commit to fewer products at a time - Ideally, teams and product leadership should agree on a limit for the number of products that can be in progress for each team.

2. Teams should work on fewer product features at a time - Product managers should prioritize one product at time, one feature at a time and encourage the teams to deliver likewise if possible.

The following guidelines were given to engineering managers.

2. Teams should work on fewer product features at a time - Engineering managers should be conscious of pulling disparate product features onto the board at the same time. They should encourage the team to focus on a limited number of features at once, ideally one. They should encourage dividing work for a product feature into subtasks that can be shared in the team to enable that focus.

3. Engineers should work on fewer tasks at a time - Engineering managers should educate their teams about flow management and encourage them to focus on a task until it’s done, if possible.

4. Engineering should reduce its code inventory - Engineering managers should monitor their team’s code inventory and encourage them to keep it low.

5. Engineering should monitor and reduce the number of bugs over time - Engineering managers should monitor their team’s backlog of bugs and schedule them to be fixed.

Guidelines for teams

Teams were given the following guidelines.

3. Engineers should work on fewer tasks at a time - Engineers should learn the basics of flow management and try to focus on one feature at a time if possible.

4. Engineering should reduce its code inventory - Engineers should review code promptly once it hits the review stage and alert others when work is blocked.

5. Engineering should monitor and reduce the number of bugs over time - Engineers should fix bugs!

A tool for improving flow

Simple, visual controls are provided as part of PDKP to help project managers improve their product development flow. These are presented on the PDKP Summary Dashboard, shown below.

The PDKP Summary Dashboard.

The metrics below are displayed on the board.

Lead time
Cycle time
Ratio of features to engineers

A second set of metrics, shown below, is paired with colors to suggest specific actions. Green means the team’s performance against the metric is good, yellow means performance against the metric is likely OK, red means the metric should be brought down to promote flow.

The estimated lower bound of days until backlog completion
The number of unresolved bugs in the team’s backlog
The number of products in WIP
The ratio of WIP to engineers
The number of commits in open PRs
The average age of code in open PRs

The motivations for optimizing these metrics was discussed in a previous section.

The actions suggested by the action colors on the Summary Dashboard will not always be appropriate in your day-to-day work. Sometimes they will be wrong. However, simply asking questions about your workflow when a metric lights up red will likely improve your product development flow.

Importantly, actions never suggest that a team isn’t working hard enough. They are intended to increase focus, not the total number of hours worked.

Preliminary results

Since the addition of the last teams to the PDKP project on June 23rd, 2017, SoundCloud has seen a reported 39% decrease in the average lead time from a mean of 106.4 days to 64.7 days and a 52% decrease in average cycle time from a mean of 77.8 days to 37.2 days.

Aggregate lead time metrics since June 23rd, 2017.

Aggregate cycle time metrics since June 23rd, 2017.

This decrease is correlated with product and engineering leadership successfully decreasing the average number of products SoundCloud’s teams have in WIP from an average of 7.4 products per team to 3.1 products per team.

Aggregate metrics of the number of products in WIP decreasing since June 23rd, 2017.

More recently, engineering managers and engineering teams successfully reduced our code inventory from a rolling average of about 1,200 commits to about 500 commits. It is likely that we have not yet seen the full benefits of this reduction reflected in our lead and cycle times.

A recent decrease in code inventory.

It is possible that much of the improvement in our measured lead and cycle times is due to cleanup and better use of our Jira. Tickets in Jira now better represent what is actually happening in reality. If this is true, at least we are getting a better understanding of what the true numbers are.

Future improvements

SoundCloud still has a great deal of room left to improve its product development flow. Specifically, the number of products teams work on at a time, the number of issues engineers work on at a time, and its amount of code inventory needs to be further reduced.

The number of products teams are working on at a time has successfully been reduced, but should be reduced further. Teams still work on an average of 4.5 products at a time. While it does not always make sense for a whole team to work on one product simultaneously, having the average be just slightly below the average number of engineers per team (about five), suggests there is room for improvement.

The number of tasks that engineers have in WIP has not improved and should be reduced. Since June 23rd, 2017, it has remained at an average of about 2.0 per engineer. Bringing this number closer to 1.0 would likely decrease our lead and cycle times.

SoundCloud’s amount of code inventory also remains a problem. Further reducing this number represents another big opportunity to reduce our lead and cycle times.

This said, we’ve improved our product development flow a great deal. We’re excited to learn more about our development cycle and to make future improvements.

Inside a SoundCloud Microservice

Fri, 28 Jul 2017 00:00:00 GMT

If you’re a regular visitor to this blog, you might be aware that we have been transitioning to a microservices based architecture over the past four to five years, as we have shared insights into the process and the related challenges on multiple occasions. To recap, adopting a microservices architecture has allowed us to regain team autonomy by breaking up our monolithic backend into dozens of decoupled services, each encapsulating a well defined portion of our product domain. Every service is built and deployed individually, communicating with other services over the network via light-weight data interchange formats such as JSON or Thrift. What we haven’t touched on so far is how a microservice at SoundCloud looks backstage.

The answer is: it depends! A necessary property of a microservices driven architecture is that a microservice represents a unit of encapsulation. As long as there is some well-understood contract in the form of an API that other services can consume, anything else becomes an implementation detail. Consequently, moving between code bases may reveal different languages, stacks, and patterns, and whether this is a blessing or curse is a hotly debated topic for someone else to explore.

What’s covered in this article is written from the perspective of a single team—the Creators Team—and as such highlights some of the programming patterns that we unilaterally settled on.

Your Server as a Function

With that out of the way, it might be interesting to know what kinds of services our team owns. Since we’re responsible for anything powering the user-facing products that serve our creators (i.e. musicians and podcasters rather than listeners) the services we build and operate are quite varied: track management, messaging, RSS feed syndication, statistics and APIs powering our mobile app (SoundCloud Pulse) are just some examples. All our services are written in Scala and built on top of Finagle with the help of a few light-weight, shared internal libraries adding support for cross-cutting concerns such as configuration injection, telemetry, and session termination.

Microservices are often intermediate nodes in a graph of services, acting as façades where an incoming request translates to N outgoing requests upstream, the responses to which are then combined into a single response back downstream to the client. In the fictional scenario illustrated below, ServiceA needs to reach out to and collect responses from ServiceB and ServiceC, which in turn might have more dependencies such as other services, databases, and caches.

One solution to this “scatter-gather” problem based on functional programming was pioneered by Twitter and presented in the very approachable paper Your Server as a Function, which outlines the ideas that eventually materialized in Finagle. Finagle models each node as a pure function from some request to some eventual response:

class Service[-Req, +Rep] extends (Req => Future[Rep])

While Finagle does an excellent job at abstracting away the tedious details behind communication protocols, connection management, and concurrency, it does leave us with certain complexities when working with this abstraction. Specifically, we will explore solutions to the following problems:

How to collect results from multiple service calls into a single data structure
How to explicitly model failures in a service’s logic layer using Scala’s Either type
How to bring the above together in a convenient way using monad transformers

Parallel Futures

The service signature above tells us that we’re dealing with effectful functions: they return Futures, not the things we’re actually after. This means we will have to lean on different combinators to extract the desired result. Imagine we want to fetch a track and all user comments it received into some hypothetical TrackWithComments object:

for {
  track <- fetchTrack(42)
  comments <- fetchTrackComments(42)
} yield TrackWithComments(track, comments)

Scala will desugar for-comprehensions into a combination of flatMap and map calls, which means that comments will be fetched only after we have retrieved the track. However, these two calls do not depend on each other, so we could run them in parallel instead. There are documented solutions to the problem of executing Scala futures in parallel in for-comprehensions, but they are non-intuitive and do not clearly express our intent.

Twitter’s Future implementation gives us a powerful combinator, join, to solve this problem both more elegantly and explicitly:

val res: Future[TrackWithComments] = for {
  (track, comments) <- Future.join(
    fetchTrack(42),
    fetchTrackComments(42)
   )
} yield TrackWithComments(track, comments)

These instructions explicitly encode our intent to obtain a pair of independent values: given (Future[A], Future[B]) we want Future[(A, B)]. Moreover, if either of the original futures fail, the result is a failed Future, which allows us to propagate errors as return values instead of throwing exceptions. Looking at this, you might be reminded of database joins: given two tables, the SQL join operator will take the cartesian product of their rows and return a new table where rows are tuples containing data from both sources.

In distributed systems where data crosses network boundaries frequently, failure is expected and should be embraced. We will now explore how we represent failure as a first class concept in our systems, without error handling ever getting in the way of productivity. We accomplish this by using Scala’s Either class, a return type that allows us to focus on the “happy path”.

Down The Future Stack

Futures are internally represented by the Try structure. Try[A] is a union type, which can be either successful (carrying a value of A) or failed (carrying an exception.) Exceptions are well suited to dealing with operational issues such as I/O failures, runtime platform errors, and unmet system expectations, but when it comes to representing failures in our problem domain, we avoid using exceptions. This is because logical failures in an application are best represented by proper values, and their presence should be made explicit in the type system instead of disappearing from plain view in unchecked exceptions. The upshot is that any service call in the logic layer of our systems returns a type that can either be a success or a failure:

def fetchTrack(id: Long): Future[Either[ApplicationError, Track]] = ???

where ApplicationError is a union type of possible things that can go wrong. For instance, we can define errors like the following:

sealed trait ApplicationError
case object NotFound extends ApplicationError
case object AccessDenied extends ApplicationError
...

Using Either as the return type instead of a Track directly states that any call to fetchTrack will return either a valid track, or a “not found” error, or an “access denied” error, but not all, meaning all possible outcomes are explicit in the function type, and must be dealt with accordingly. Future is already a functor so it comes with the map combinator defined, which allows us to focus on the happy path and “dive into” a successful Future:

// only executes the lambda if the Future was successful
fetchTrack(42) map { trackOrError => ??? }

However, fetchTrack now returns an Either value, not a track. We would ideally like to map trackOrError again to continue following the happy path, but indeed this only works as of Scala 2.12. In Scala 2.11 or older (which is what our systems are deployed with), Either is not “biased” toward either of the outcomes it captures (in our case an ApplicationError or Track), so no functor instance is defined that allows us to focus on the happy path. Fortunately, we found a solution in Typelevel’s Cats library, which compresses an enormous amount of utility into a small library. Let’s check it out!

Catnip’ed

Cats is not a reference to the furry animal but is short for “categories”, a nod to the branch of mathematics that studies structure in mathematics itself. It turns out that category theory is concerned with abstractions that are extremely useful when working with strongly typed languages. Specifically, category theory provides us with “blueprints” or “recipes” for the traversal and transformation of types in ways that are provably correct. One such recipe is encoded in the Functor type class, which allows us to turn any type constructor into a functor, given some rules are obeyed. Cats already provides a functor instance for Scala’s Either type, so we can now map over Eithers:

def fetchTrack(id: Long): Future[Either[ApplicationError, Track]] = ???

fetchTrack(42) map { trackOrError => trackOrError map { track => ??? } }

The drawback of nesting effects is that we now have two levels of machinery: Future and Either, which have to be dealt with separately. This makes it cumbersome to get our hands on a track. Ideally we want a functor instance which folds both into one: if the Future succeeds, and the Either contains a “happy outcome”, then map the result. It turns out that Cats provides precisely such an abstraction: it’s called a monad transformer, and for Either it’s called EitherT. In our case specifically, since we’re always dealing with Futures and ApplicationErrors, we decided to glue these together into a compact internal library written on top of EitherT to model results of any computation in our systems. We call it Outcome.

The Final Outcome

Outcome is a set of type aliases and type converters that bind together Future and a well thought out ApplicationError hierarchy (with broad applicability to most or all our services) using Cats’ EitherT type:

type OutcomeF[A] = EitherT[Future, ApplicationError, A]

This definition allows us to specify return types in a compact fashion while maintaining our focus on the happy path:

def fetchTrack(trackId: Long): OutcomeF[Track] = ???

fetchTrack(42) map { track => track.title } // works!

We can also recover from potential errors:

fetchTrack(42) map { track => track.title } recover { 
  case NotFound => "(unknown)"
}

In fact, we have recover clauses like these in all our request handlers, so we can easily map a NotFound error to an HTTP 404 response code. But that’s not all. You may recall from above how Future.join can flip futures inside out. Using the EitherT transformer, we can do the same for outcomes:

val trackWithComments: OutcomeF[TrackWithComments] = 
        (fetchTrack(42) |@| fetchTrackComments(42)).map(TrackWithComments)

Here, |@| is the product combinator of Cats’ Cartesian type which joins outcomes into a tuple of values without stepping out of the “outcome context”. We can then conveniently map this tuple onto a properly named data structure.

Summary

We started out by exploring what makes writing microservices challenging. The frequent crossing of network boundaries requires us to deal with asynchronous, error prone operations. We furthermore pointed out that we draw a sharp distinction between operational errors and those arising in our product domain. To deal with the latter, we used Scala’s Either type, leaving us with more explicit but verbose service function calls. Finally we revealed how Cats can restore compactness of service function calls via EitherT and Cartesian, and how we capture their functionality in a convenient type alias: OutcomeF. We barely scratched the surface of Cats in this article, it provides a wealth of utility when opting for a more functional style of programming. However, we hope we could give you a glimpse at the patterns you would find in a typical Scala code base in SoundCloud’s Creators Team and hope they may turn out as useful for you as they did for us.

Until next time!

Matthias and the Creators Team

Remote device sign-in

Wed, 28 Jun 2017 00:00:00 GMT

A method for signing in to a device that doesn’t have a keyboard

When we were developing our SoundCloud app for Xbox One, something became very obvious during usability testing: signing in with a game controller really sucks. Entering text requires navigating a virtual keyboard to individual letters, numbers, and characters one at a time – such a nightmare! Plus, letters, numbers, and special characters are spread across three screens. The more secure your password is, the worse the experience is.

We needed a way for people to sign in to the app some other way. Something secure and simple and fast.

The solution, in brief

The person wants to sign in to SoundCloud on Xbox One. They visit soundcloud.com/activate on another device, such as their phone or laptop, and enter a short code. Boom, they are now signed in to SoundCloud on Xbox One. Magic!

We needed to consider that the person might initiate this flow on a laptop, tablet or a mobile phone. We hoped to take advantage of the person already being signed in on their second device.

How it works

Consider a person who has two devices:

Device A: a game console or a TV or really any device with no keyboard or otherwise awful usability for entering text.
Device B: a device where it’s easy to enter text, like a phone or a laptop.

The person wants to be signed in to your service on Device A. The person has Device B nearby.

The app on Device A requests a short easy-to-read code from your service and displays it along with an easy-to-read URL.

The app on Device A polls your service every few seconds to see if the code is associated with a user.

The person pulls Device B from between the couch cushions, opens a browser, and goes to that easy-to-read URL. On Device B, the person might already be signed in to your service. If they aren’t, they either sign in or create an account.

Once authenticated, the person is prompted to enter the short easy-to-read code.

Device B asks your service if the code that was entered is valid. For security reasons (outlined in a later section), the person is asked to confirm that they want to allow Device A to be signed in as them.

Upon confirmation by the person, Device B updates the code using the authenticated user’s access token, to associate the code with the target user.

At the moment of confirmation, the code is associated with the user. The next time Device A polls, it sees that it can use the code to request an access token for that user. Device A then makes a request for an access token using the code.

Et voilà! Now Device A has an access token for the target user. In other words, the person is signed in to Device A as the desired user.

This pattern is not something we invented. Our flow is influenced by a similar flow used by YouTube on TVs and Google Sign-In for TVs and Devices.

Using an authenticated session on Device B

This method of signing in means the person does not have to enter their password on Device A. If they are signed in to your service on Device B, it is possible for them to avoid needing to sign in again at all.

At SoundCloud, we have a web app at soundcloud.com, and we have native apps available for Android and iOS.

If the person visits soundcloud.com/activate on a laptop, we use our standard means of requiring authentication before showing the prompt for the person to enter the activation code.

On mobile, we chose to reuse the same “activate” web app opened in a web view in the native app. If the person visits soundcloud.com/activate on a mobile device, we invite them to open our native app. Inside the app, we use our standard means of requiring authentication, and then we open a web view with the user’s access token appended in a URL fragment to an oauth2-style callback web address:

https://example.com/activate_oauth2_callback#access_token=ACCESS_TOKEN

This allows the web app to get the access token from the URL fragment. We put secrets in a fragment instead of a query parameter so that secrets are not sent to the web server and therefore not stored in the web server logs. This is the recommended way of passing access tokens and other secrets in a web address.

We also instruct the web app how to display the page using a query parameter:

https://example.com/activate_oauth2_callback?display=android#access_token=ACCESS_TOKEN

You could of course build this form into your native app directly. We like that our implementation allows us to iterate quickly on this brand new feature. We can easily make security or usability improvements to the interface without shipping a new version of our native apps.

Choosing codes that are easy to read

You want the code to be as short as possible while still having enough possible codes available so that everyone who wants to sign in at the same time can.

You also need to have a sparse enough usage of the possible codes such that it is unlikely that a typo will result in the person entering a valid code that has been issued at the same time to someone else.

Imagine if your codes were one number long, ranging from 0 through 9. The risk of an unnoticed typo is very low, but you could still only issue 10 codes at one time.

If your codes were two letters long, then you could issue 26 x 26 = 676 codes at the same time. There is now a risk of an unnoticed typo, which means you can’t safely issue all 676 codes at the same time.

If you use just numbers, and your codes are 6 digits long, then there are one million codes available, assuming you allow leading zeros. Pretty good! That might be enough for your needs.

If you use just letters, and your codes are 4 letters long, then there just under half a millions codes available. Still pretty good, and the codes are shorter. Not bad!

If you chose to use a mix of letters and numbers, then do not use characters that are hard to tell apart. Specifically, the letter O and the number 0 are hard to tell apart, as are the letter I and the number 1. Even if you use fonts that make these characters distinct, you should avoid these altogether.

Definitely don’t use special characters. That’s just asking for usability problems. If you use letters, display the codes in uppercase for readability, but verify the codes with case insensitivity. Our form for entering the code uses CSS to transform the characters into uppercase for ease of entry.

We chose to use a mix of letters and numbers, and we chose to use 6 characters, which gives us over a billion codes that we can issue at one time. If a billion seems like way more than one needs, read on for further security considerations.

Security considerations

In its most basic form, this flow has a number of security problems ranging from mild and unlikely to wildly irresponsible. Mitigating these security considerations requires adding details to the basic implementation.

Accidentally granting Device A access to the wrong user

When the person visits the your service on Device B, a user may already be signed in there. People who share a device, or those with more than one account on your service, may want to sign in with a different user than the one that is currently authenticated. If they are not informed which user is authenticated on Device B, they might accidentally grant Device A access to the wrong user.

To mitigate this risk

Show the person which user is authenticated on Device B, or display several authenticated users, and allow them to choose which user will be granted access.

Accidentally granting access to someone else’s device

Imagine that two similar codes, such as XXN and XXM, were issued at the same time. XXN was issued to Device AN and shown to Person N. XXM was issued to Device AM and shown to Person M. Person N is authenticated as User N. If they mistype and enter code XXM before Person M does, then Device AM will become authenticated as User N. Oops!

To mitigate this risk

Have a big enough range of possible codes that typo collisions are unlikely. If you have very a large range of possible codes, the chances that a typo is also a valid code is very small.

If possible, collect information from Device A when you create the code that the person could recognize when they are granting access. For example, if the device has a name like “Deejay’s TV”, then collect that from Device A, and show this information to the person when they are confirming that they want to grant the device access. We found that we did not have access to such data on Xbox One, however, so the best we could do was capture the type of device.

Using up the possible codes so no one can sign in

An attacker could destroy the ability for people to sign in by exhausting all the available codes. If your codes are generated on request, then as your possible codes are being used up, your service may have a near impossible time finding one it can issue.

To mitigate this risk

Implement a rate limit for creating codes. Limit based on all the facets of the request that you can, such as the IP and any unique device identifier. You will need to investigate to determine what the peak rate would be for legitimate traffic.

Your codes should expire so you can reuse codes after a time. If you reuse codes, then you need a large enough pool of codes so that you don’t re-issue the same code too soon between uses. Your pool of codes should be able to carry you through at least several days before you need to start re-using them.

Guessing codes and using them to get access tokens

An attacker could try to poll for random codes, hoping to find one that is activated so they can exchange the code for a user access token.

To mitigate this risk

You could have such a very large number of possible codes that guessing a valid one is nearly impossible. However, a code that works for a person needs to be short and use a limited character set, and a bot can burn through guessing many codes if they are easy for people.

You could add rate limiting for polling, and then cross your fingers and hope that you have enough possible codes, and your rate limits are good enough that the attacker won’t be able to guess a valid code while staying within your rate limits. You should add rate limiting for polling, but there is a much more secure way to mitigate this risk.

When you issue a code, you should also issue a very long, hard to guess polling token. You should require this polling token for unauthenticated requests for the code. You can see this secure feature described in the Google Sign-In for TVs and Devices documentation. They call the easy-to-read code “user_code”, and they call the polling token “device_code”.

Tricking people into giving away access to their account

There is a category of attacks where an attacker attempts to influence their victim to take an action that is not in their best interest. These are called “social engineering attacks”.

For example, the attacker could take advantage of your UI design choices and their ability to supply some aspect of the display text, and use it to trick people into thinking the UI does something different than it does.

Take this example, where “Deejay’s TV” is an open text field called device_name that is supplied when the code is created:

From a design perspective, this UI seems succinct and clear.

Now imagine an attacker sends a victim an email saying “Visit this URL and enter this code to get a FREE subscription!” When the victim gets to the page and enters the code it looks like this:

The attacker has created a code with the device_name “SoundCIoud Premium Add-on”. The design makes it possible to fool people into giving away access to their account.

To mitigate this risk

Use wording and design elements that make it clear what is happening. Some labels and text might seem redundant in the standard scenario, but are necessary to prevent this type of attack.

When the code is created, you could store the requesting IP or resolved geography, and then provide a warning if the code is entered in suspiciously different geography.

Have tight enough expirations on your codes such that this kind of attack would need to occur in a tight time frame. If your codes are only valid for 5 minutes, then the attacker has to find a way to provoke an action from their victim within 5 minutes.

Designing and implementing a new authentication flow

From the very beginning, we included colleagues from our Security Team to perform a Threat Model analysis. We built out our API design using this analysis as a basis, instead of waiting until later to consider the security implications. During development, we had an eye on how to secure our new feature at all times. We adjusted the graphical interface based on a security review as well.

Our team really enjoyed thinking through this problem. We were able to make the painful experience of entering text with a game controller seamless for our users. We had a lot of fun delivering a shiny new authentication experience that’s secure and feels like magic.

A Better Model of Data Ownership

Tue, 20 Jun 2017 00:00:00 GMT

Once upon a time, we had a single monolith of software, one mothership running everything. At SoundCloud, the proliferation of microservices came from moving functionality out of the mothership. There are plenty of benefits to splitting up features in this way. We want the same benefits for our data as well, by defining ownership of datasets and ensuring that the right teams own the right datasets.

In this post we’ll briefly recap the motivations for splitting up the monolith, describe what we mean by ownership of (micro)services, and see that the responsibilities of ownership apply to datasets too. With the help of some examples, we’ll see how to decide who should be the owner of a dataset, where the costs arise when ownership is in the wrong place, and the visibility concerns that are addressed with this stronger definition of ownership. Finally, a brief word on the task of getting from where we are to a closer adherence to this model.

Recap: why split up a monolith?

It’s much easier to keep cohesion high and coupling low when there’s a boundary between components. We know this from writing code, where we split code into smaller pieces when it gets too large.

At the level of building services, that split might be across modules or code repositories, and access might be over HTTP. Defining these interfaces between components allows you to design a clearer architecture, and often leads to simpler maintenance, though at the cost of complexity that comes from communicating over the network.

Clear interfaces also reduce the coordination required when multiple teams work together. The teams agree on the interface, what the data and schema looks like, and can then work independently to build and maintain components. This also allows the system to scale as teams grow independently.

We saw the lack of clear interfaces within our monolith create problems. It cost more time than it should have to make progress on features, and cost more to maintain them. Importantly, these costs of working with a feature were often not visible because they were distributed around everyone who touched it – which, in the monolith, meant everyone.

All of this has been covered before. The important point is that clear interfaces allow us to define ownership of the features exposed through these interfaces.

What is ownership?

Ownership of a service brings a number of responsibilities to a team. These include

availability and timeliness – clients should be able to access data and operations provided by the service, and expect the data to be up to date (whatever that means for this service).
quality and correctness – the service should do what was agreed with its users, and when it doesn’t the owner should fix the bugs. This puts a burden on the service provider to manage, for example, the lifecycle of response schemas in a way that its clients can handle.
identification of consumers – the service owner should know who their users are, so that when changes are made they can be negotiated with the consumers before breaking things elsewhere.

The key insight is that none of this is specific to services. All of these responsibilities apply equally well to an owner of a dataset, a piece of shared infrastructure, or really anything that is an integration point between two teams! This is because a service is just one way to provide an interface to some data, giving a way to read and modify the backing dataset.

What does ownership mean for datasets?

Similarly to owning a service, owning a dataset means that you have the responsibility to produce data according to some contract with your consumers. Data should be available within some understood time window and should be correct according to that contract. As the producer, you have the responsibility to fix problems with the data.

You should also be able to know who your consumers are. This visibility makes it possible to make changes to what you publish over time. There are technological fixes for this. With services this can be done by authentication/authorisation on the service, something simple like requiring a request header stating who the client is, or more sophisticated such as distributed tracing. With datasets you can monitor access to wherever the dataset is stored (e.g. reads from HDFS or an RDBMS), or adopt more asynchronous ideas like scraping for dataset locations and the jobs that read/write them, and building a dependency graph between them.

How do you decide who’s the owner?

To begin with, the owner of a dataset must also be the producer – the team that runs the process that creates the data. This is because the owner can only fulfil their responsibilities if they have control over writing the data.

There are two main factors that come into deciding which team should be the owner. We’ll go through them both here with an example. These two ideas are enough to reason about the ownership of a lot of datasets. (There are also cases where it may be clear that ownership is currently in the wrong place, but where there isn’t a clear team that should own it. This includes datasets that are populated from multiple sources, where ownership of an entity is shared between teams.)

Semantics

The team producing a dataset should be the team responsible for the semantics of that dataset. Equivalently, a team not responsible for the semantics should not be producing the dataset.

A dataset is only meaningful with some definition of the information in the dataset. Many teams might have some idea of what a dataset means, but only one team has the responsibility of designing and evolving the semantics. That team should be the owner of the dataset.

We see teams trying to rebuild views of datasets from elsewhere in order to fit their precise needs. In this toy example, suppose that this Tracks team creates an event (in the event sourcing sense) each time a new track becomes available on the platform, and uses that to build a dataset summarising all of the tracks. Parts of that process might be simple (e.g. showing counts of tracks, totals of media length), but some may be much more complicated (e.g. tracks have geographical permissions modelled in a different system, and we want to summarise the number of tracks per country).

Now consider a Reports team using this information to build reports. If they need more than a summary, the published dataset may not be enough. A direct way to get the information they need would be to read all of the events that went into that summary, and build the new dataset themselves. But that requires knowledge of all of the details of tracks and the permissions model and for all of that to be exposed in the events. The logic will be duplicated in the two teams, separated from where it should be. This leads to a high risk of inconsistency in different views of data.

At the start of a project using this data, it would be difficult to predict the cost of the work, because the expertise is in the other team. Over time the cost of maintenance of the dataset is also hidden (and larger) because the work is distributed around the consumers. When the semantics of the data change later, it will be easy to miss making the required change everywhere. Handling this all in events can be particularly difficult, because changes to the schema require manipulating the history of events.

A better solution is for the work to be done in the team that has full knowledge of the semantics and the responsibility of maintaining the dataset. In this case, the Tracks team should build that dataset with enough information for the consumer, and the Reports team should use the published dataset. It’s clearer what the costs are and work is not duplicated. When changes happen to the semantics, the owning team simply publishes a new dataset.

Encapsulation

Don’t consume data from other systems’ internal data stores

The internal data structures of a component (of any size: system, service or single class) are non-public so that invariants can be maintained. Other parties (calling code, other teams) get access through an interface. This level of indirection through a published interface allows the owner of a system to make changes to the data store without other users having to coordinate.

In this example, consider how data gets from various systems in the company to a data warehouse used by data analysts. Such a tool can be great for analysis (find all of the data in one place, easy to query) but can lead to problems of maintenance later.

Consider a tool to take a full snapshot from a database and push it to the data warehouse, where it’s available with the same schema and data as at the source. This provides a really easy way to get your data into one place. But it also couples the data warehouse to the internal database of your service – specifically it couples together the schema of your database, the data warehouse, and any query that reads that data and expects a certain schema. When you want to change your supposedly-internal database schema, you suddenly have to coordinate with a lot of downstream users. Worse, given that the data warehouse is a centrally accessible resource, you probably can’t know who those users are.

This setup has made the internal database an interface to the data, breaking the system’s encapsulation.

Indirection to the rescue. Rather than using the snapshot tool to copy from the database to the data warehouse, you as the team producing the data should copy the data somewhere else, say on a shared filesystem such as HDFS. You’ll need to define a schema for the snapshot, even if this is (initially) only a representation of the source schema. You’ll want to consider the design of this interface, too. For example, include only the fields you need first, adding others later as required. As the schema evolves you’ll need to think about backwards compatibility, to ensure that consumers will continue to be able to read the data.

This approach gives a new integration point, and a way to decouple the lifecycle of your database schema from that of your consumers. This doesn’t solve all of your problems (what if my schema has to change in a way that isn’t backwards-compatible?) but it does give a place to start and makes it easier to manage those more difficult changes.

What does this mean for my team?

Overall, the point of arranging ownership better is that as a company we spend less on maintenance. This will mean work moving between teams and consolidating in a single place per dataset. In some cases this will mean a team needs to take on more work; in some cases less.

Let’s look at the viewpoints for a few different roles.

I’m a producer

Where you produce data there is an expectation that it should be exposed appropriately to consumers who need it. This may mean some new work to understand their requirements and build the integration. You should understand the severity of failure or data loss on your consumers.

Where there are existing dependencies on data you own, but where you don’t own the process that produces that dataset, there is an opportunity to remediate technical debt. This can happen by introducing a publishing workflow that you own and that makes sense in your domain, and negotiating usage with consumers. Your consumers will then have much easier access to this data, and you will be in control of all of the moving parts when you need to make changes.

I’m a consumer

Where you consume data, there is an expectation that you do less rebuilding of datasets, and instead, negotiate with the producer of your data to have them produce it in an appropriate form.

Where you are already building datasets whose semantics you don’t control, work with the team that should own the dataset so that they produce and publish the data you need in a way that’s appropriate for both. This may be as simple as the producing team running an existing job, or may involve asking them to build something new to expose the data.

I’m a product owner

Where you own a feature, consumers of the data generated by your feature are your users too. When considering priorities for your team, you’ll need to balance the needs of your internal data consumers with those of your external users.

Show me the data!

How can this work in practice? A common fear arises when you need some data, when it isn’t available in the right form and when your project is short on time. It may appear quicker to build the data yourself than to request that the work be prioritised in another team, where you’re competing for attention among their other priorities.

There are essentially three options that distribute the initial cost (writing the job to produce the data) and the ongoing maintenance cost. Either way, the code to produce the dataset has to be built somewhere.

consumer builds and owns – the consumer takes on all of the cost and the wider organisation takes no shared benefit
consumer builds, producer owns – the consumer, in cooperation with the producer, builds the dataset and then hands off ownership to the producer. The consumer takes the up-front cost, the producer takes the longer term cost, and the total future cost is reduced because no-one needs to duplicate the implementation.
producer builds and owns – the producer does the initial work and takes all of the cost, and again the benefit is shared. This is the no-compromise approach, and may not be possible in short-term due to balancing other priorities.

In any case, the outcome of this project should be that a curated dataset exists, the dataset is owned by the producer of the data, and consumers integrate with this and not the source data.

Conclusion

When accessing and using data that is owned by other teams, certain patterns lead to high costs of maintenance that are not apparent at the start. It can be difficult to track these costs later when they become distributed around multiple teams.

As a consumer of data, avoid building datasets whose semantics you don’t own, because it’s duplicate effort and you leave yourself at the mercy of future changes you don’t control. Use published interfaces to access data; asking the producer of the data to create one is preferable to doing it yourself or accessing their internal data stores directly. As a producer, it’s important to think about how the data your feature generates will be used, and how to expose to other teams in a way that doesn’t spread maintenance responsibilities.

Following these two principles should make it easier to keep your data integrations well understood and maintainable, by giving better visibility over data that is shared between teams.

Lessons in resilience at SoundCloud

Wed, 07 Dec 2016 00:00:00 GMT

Building and operating services distributed across a network is hard. Failures are inevitable. The way forward is having resiliency as a key part of design decisions.

This post talks about two key aspects of resiliency when doing RPC at scale - the circuit breaker pattern, and its power combined with client-side load balancing.

Circuit breakers v1.0

A basic definition of the Circuit Breaker pattern and its applicability is explained in Martin Fowler’s post on the topic.

SoundCloud’s original implementation meant a client’s view of a remote service transitioned between three pre-defined states:

Closed -> Everything seems fine, let all requests go through. After n consecutive failures, transition into Open.
Open -> The request, if let through, has a high probability of failing, given recent data. Fail the request pre-emptively and allow the remote service to recover. The circuit remains open for a pre-defined duration, and all requests are failed during that time. Thereafter an attempt to reset is made by transitioning into Half-Open mode.
Half-Open -> Something bad happened a while back, let’s be cautious and let a single request through. If it passes, transition to Closed, else to Open.

With this implementation, a single failure becomes the deciding factor for opening a circuit, blocking all requests to the remote service for a period. This meant degraded end user experience and having to manually restart client systems to reset the state.

The missing parts

As you start operating systems under high load, the following questions arise:

Can we make more accurate predictions about the health of a remote service by using a moving window and tracking success as a percentage of total requests, as opposed to opening on n consecutive failures with unbounded duration?
What happens when one or a few instances (limited to a specific network zone in a data-center, maybe) of a service are unhealthy, but a majority of other instances are healthy and available? Is it fair to fail all requests to a service in such a situation? This is especially relevant in a microservices architecture, where scaling horizontally is common.
How can remote services convey more information about their health? A HTTP-based service may be up and running but repeatedly responding with 5xx. Can we make clients smarter by using application specific information to decide when it’s a good idea to avoid overloading services?

A better alternative

In a series of previous posts, we’ve described how Finagle plays a key role in building our services.

Given JSON over HTTP is used for inter-service communication at SoundCloud by most services, resiliency at this layer is critical to our uptime and dealing with all possible classes of failures.

Earlier this year, we started exploring solutions aimed at improving resilience for our HTTP clients.

We found that most of the gaps detailed above could be addressed through a combination of modules included in the Finagle stack for building clients.

This was good news, but we still needed custom tweaks. The most important of these was to identify and compose multiple available modules, to help us define policies for how clients behave in failure scenarios and meet our expectations for high availability and a great user experience.

Specifically, we could map each question to a module in Finagle that could potentially address the gaps:

Better than n consecutive failures

The problem : An unbounded duration, instead of a moving window, means clients infer inaccurate information about the health of remote services. A failure accrual module in Finagle allows clients to define a policy for success in terms of:

a moving window specified as a duration or number of requests
a success rate, expressed as a percentage of requests that should return an acceptable response
a duration for which to mark a specific instance of a service as unavailable

This captures all dimensions by which we want to define success thresholds.

Client’s view of individual service instances

The problem : A client’s aggregated view of a remote service means a single rogue instance can trick you into believing that the whole service is unhealthy.

Finagle monitors health of services, per instance, and applies the success policy defined by clients to decide if it can allow requests through to a given instance.

A few unhealthy nodes can no longer render your service completely unavailable. Big win!

In conjunction with client side load balancing, this creates a dynamic view of a service for clients.

Clients can now continue fetching data, with requests smartly routed only to service instances marked healthy, while unhealthy instances are temporarily ignored and allowed to recover.

Using the Application layer for decision making

The problem : network layer information alone is often not good enough to make informed decisions about health of remote services.

The ability for clients to define what classifies as failures, using information available in Application layer protocols, is powerful. This gives fine-grained control, and is an improvement over the standard network layer signals to classify remote invocations as success or failure.

Clients should be able to classify failures from services, based on protocol specific data (e.g.: a HTTP 5xx response code).

This provides much needed application specific context to circuit breakers and results in accurate decisions regarding the health of individual instances of a service.

Support for response classification was introduced in Finagle earlier this year, just in time for us to put it to good use.

Wrapping up

It is important to acknowledge that even the task of composing appropriate modules and configuring thresholds takes a fair bit of learning until you get it right.

We switched to using a moving window of duration instead of number of requests, as we realised it was hard to define thresholds that worked equally well for high and low traffic services.

Similar implementations exist for services that talk to database systems, cache nodes etc.

Given that we were already on the Finagle stack, the capabilities listed were compelling. A combination of Ribbon + Hystrix (Java-based libraries, aimed at fault tolerance, from Netflix) make for a decent alternative.

The above points aim to provide a high level summary of capabilities to look for when choosing to implement or use a framework aimed at improving resiliency.

Related concerns

What should be the policy around clients retrying requests?
When dealing with sudden surge in incoming traffic, how can we cordon off services at the edge layer and keep critical core systems available?

We hope to share our experiences on these aspects, as we move forward, and scale out further.

API Sign-up Changes

Fri, 23 Sep 2016 00:00:00 GMT

Today we are launching a new API application process.

Starting today, developers will need to fill out an application form to request access to our API.

Here is why.

As SoundCloud has grown, we’ve seen our API used in countless innovative ways that support and enhance the SoundCloud experience for our creators and listeners around the globe. At the same time, we’ve also seen a number of apps or services that act on behalf of our users without their explicit permission or attempt to use creator content in ways that are not permitted by our API Terms of Use. Apps or services that operate in this way do not support our creators, listeners or the SoundCloud community.

Here is what it means.

Developers who currently have API credentials and comply with our API Terms of Use may continue to use the SoundCloud API. All new requests for an API key must go through the review process, where applications will be reviewed on a case by case basis, in line with our API Terms of Use. If your application is successful, you will receive an API key.

To review our API Terms of Use, click here.

If you wish to submit an API Application Form, click here.

Microservices and the monolith

Fri, 26 Aug 2016 00:00:00 GMT

In a previous series of blog posts, we covered our decision to move away from a monolithic architecture, replacing it with microservices, interacting synchronously with each other over HTTP, and asynchronously using events. In this post, we review our progress toward this goal, and talk about the conditions and strategy required to decommission our monolith.

Let’s review briefly. We are presented with a problem; namely, we are unable to confidently make changes to the monolithic system powering SoundCloud, and that the growth of its database schema exceeds our ability to support it. So we need to decommission the monolith, and decided to migrate to a microservices architecture. Our plan for migrating away from a monolithic architecture to one based on microservices has been:

Invest in internal tools and libraries to make the creation of microservices easier.
Expect that new features can be built outside of the monolith, using only the internal API it provides to access its data, or the events it emits on state changes.
When making significant changes to a feature or entity which exists inside the monolith, consider implementing these in a new service which is outside the monolith. We call these extraction projects.

This strategy has given us some great results. We have drastically reduced the time and number of decisions required to create a new service. These services benefit from world-class telemetry capabilities provided by Prometheus; confidence in deployments thanks to Docker, Kubernetes and the work we do to improve testing of service interactions; and an efficient and powerful network stack in Finagle. Work done in any of these areas has a powerful effect on the productivity of all engineers using these tools and libraries. If an engineer working in the core group makes a change which increases the productivity of many other engineers, the net effect can be equal to hiring an additional engineer. As we grow, these small effects result in dramatic improvements visible across the entire engineering organization.

As a strategy for decommissioning our monolith, though, this strategy has not served us well. We have noticed that:

An extraction project, if its goal is to remove all code related to an entity from the monolithic codebase, is a multiple-month undertaking with dependencies on all teams who use that entity.
Deploying a new service which serves all needs around an entity requires understanding — and, in cases where database joins are performed, changing — the code related to that entity in the monolithic codebase.
Investing heavily in the skills and tools needed for our new microservices architecture makes it harder to find the engineers who are willing and able to work on the monolithic codebase. Many engineers feel intimidated by it and do not feel a sense of ownership for it.

That being said, we have been able to extract some services. Most often, this has happened in teams where engineers have experience with the monolith. In these cases, the engineers were able to make the case for an extraction project: the feature they were beginning to work on required integration with the monolith, and could only be expected to work for as long as the monolith continued to work. With database growth rates as they are, and diminishing knowledge of the codebase, this was a short enough period that the decision to do an extraction project was clear.

In many other cases, though, developers were not able to make this case, or were not even aware that they should, because they felt so removed from the monolith. In these cases, we have even seen services bypass the monolith and access its database directly, because the developers working on a new feature lacked the knowledge of how to modify its code, and were far quicker on our newer, better-supported, Scala stack. When we come to work on an extraction later, services accessing the database directly present a problem for the project. They require a special negotiation about how to integrate, rather than the standard “from this date on, please use the Foo service for this endpoint, rather than the monolith.” Our investment in making services easier to build has created the conditions for behaviors which make it harder to decommission our monolith.

How, then, should we revise our strategy, and decommission the monolith faster? We are exploring a new approach:

Recognize that, for an extraction project whose definition of done is “remove all code related to an entity from the monolith,” the part that will take the longest and be the most complicated is to change all call sites related to this entity. This is especially difficult in cases where the calling code relies on an endpoint which joins two entities, and the future architecture will see these entities belonging to different services. To accelerate this effort, we must distribute the work.
Change the definition of done for an extraction project to “there is a service available which supports all access patterns for this entity which we can support in future, but there is still use of the monolith by some downstreams integrating with this entity.” Since this means an extraction will not be complete until downstream services make changes, the below points ensure progress toward the eventual goal of complete extraction from the monolith.
Use monitoring to measure progress in migrating calling code.
Produce guidelines which explain how and why to compose responses from multiple backing services. This guidance should include what to do about the resulting consistency issues (“the tracks service has returned me a track belonging to user 123, but the users service tells me that user doesn’t exist”).
Produce clear guidance around the ways in which services of different kinds can integrate with an entity. For example, a service should never access another service’s database directly for online use cases. Hold engineers to account for ensuring their systems follow this guidance.
Prioritize extraction projects based on their size and rate of growth in the monolith’s database, since this is the most pressing risk to the health of that codebase.
Make use of the monolith experts we have to guide and accelerate the creation of services which will replace parts of it.

We hope that this approach will get us to the goal of decommissioning our monolith faster, and continue to make use of the investment we have made in our microservices ecosystem, while dealing with the forces which led it to work against this goal.

Are you working to decommission a monolith and move to microservices? We’d love to hear from you about ideas or approaches which have worked for you.

React Native at SoundCloud

Wed, 03 Aug 2016 00:00:00 GMT

About a year ago we faced an interesting question at SoundCloud: can we build SoundCloud Pulse — our app for creators — with React Native? Is a five-month-old technology mature enough to become part of SoundCloud’s tech stack?

A couple of months prior to this question, we started to design our second set of native apps for Android and iOS. We focused on iOS first because that is where the majority of the creators were. Finding iOS engineers proved harder than we thought, which is a situation likely familiar to you. For Android, we partnered with Novoda. We did not want to have many months between the iOS and Android releases. Meanwhile and independently, the design team was running user-testing sessions with React Native-based prototypes and we also had capacity from very eager web engineers.

Warming up

We had two main goals at the start. Including all teams whose input was valuable for this project and convincing ourselves of React Native. Input from our iOS team with many years of mobile app development experience was very valuable and reassuring. At the same time we were able to give them context as to why we were investigating React Native. Openly talking about the project in the company and including people in the decision making process helped us a lot in learning what React Native could and couldn’t do:

Which elements can and should be written natively?
Which elements can and should be written in JavaScript?
Can it compete with native apps in a way that users won’t notice it is “different”?
Does React Native support the native share dialogue?
Does it support deep linking?
Does it support securely storing access tokens?
Does it support Facebook & Google sign-in SDKs?

We also had questions about how well it would integrate with our existing native libraries:

Can we run acceptance tests with our existing iOS tooling?
How can we implement internationalization?
Can we integrate it into our continuous-integration pipeline?
What does the performance look like for long lists? In general?
Does it have a negative effect on battery life?

During a two-week spike, we built a prototype of the SoundCloud Pulse app to have a better understanding of React Native in practice. After the first week, we had already fleshed out most screens of the app and were surprised how quickly we were progressing. During the second week, we wanted to focus on the unanswered questions and write some native code. It was easy to bridge our existing iOS libraries with our React Native prototype. We completed the prototype with support from our iOS engineers and concluded that we had enough confidence in React Native to go ahead with the project full time.

The journey

We started working on the app with two JavaScript engineers and one iOS engineer. There were no designs for the iOS app yet but we knew that we needed to reach feature parity with the Android app and so we used the Android app as a blueprint for the iOS version.

It turned out to be a great idea to have an iOS developer working side by side with the JavaScript developers. There was a lot of knowledge sharing in the initial couple of weeks and we were able to keep up the fast pace from the spike sprint. We finished entire screens in single days and even built more features into the app than we had on Android in the beginning (e.g. 1Password integration).

In contrast to the spike sprint we focused more on achieving high test coverage. Coming from a mostly web background we were used to be able to deploy hot fixes within a couple of minutes in case of a runtime error. On mobile this is not possible and so we had to test more thoroughly before shipping a new version of our app. We thought about using services like CodePush that allow you to push an updated version of your app without needing to wait for the app store review process. However, all those services were not stable enough in the beginning and we decided not to use them to reduce the amount of sources of errors.

Development of the app started mid-October 2015 and we shipped the first version mid-February. It took us four months to write the app with three developers. Four months from a spike code base to a unit-tested, component-tested, and integration-tested application. We estimated that it would take us six months to finish the app but we underestimated how fast React Native allowed us to be.

Things we’ve learned along the way

Native libraries

For many popular native libraries / APIs / SDKs someone probably has written a bridge to React Native already. Of course the Facebook SDK is one of them, so is Google Signin but also things like Appboy, access to the iOS Keychain and many more.

If your biggest fear of jumping on the React Native bandwagon is that you have already amassed many great Objective-C libraries that you don’t want to lose, don’t worry, it is very easy to use them. React Native’s native modules are well equipped to bridge anything you have over to be accessed by JavaScript, and they are really quick to implement. It helps having someone with Objective-C knowledge in your team or someone who can help out, but the API is very understandable even for non iOS developers.

A perfect example of this was our event-logging library. It was battle tested in our listener app, it had support for batch sending events, so it would be lighter on the network traffic, and it could handle offline / bad connectivity. We were able to extract it from the listener application without too much effort. The rest was as simple as bringing a few methods like initialize, pageView and click to JavaScript. \o/

Native infrastructure

We didn’t need to reinvent the wheel when it came to building and testing our application either. Since React Native applications are built pretty much like any other iOS app (not counting the extra step of compiling JavaScript of course ;)) our iOS and testing teams were able to set up a CI environment very similar to the current structure within a couple of days. The acceptance tests were written in the same style as our iOS developers benefiting a lot from their experience and it making it easier to set up.

Another great help was knowledge around how to distribute an application to the app store and the process that revolves around that. Since we were mostly web developers and did not have any experience with that process, setting it up on the CI saved us a considerable amount of time running builds manually and uploading them to the app store by hand.

Web libraries

We could reuse a lot of our internal web modules with little to no modification. One good example is the way we handle internationalization. We are using the same library that we are using on all of our web-based applications, so we only needed to adapt our string parsing slightly.

Another example is a library for defining API endpoints in a centralized way. We were able to use it straight up, no changes necessary, by passing React Native’s fetch function as the transport layer. These were just two examples but there were many more small utilities / helpers that we were able to just drop in.

React Native Gotchas

Of course no technology is perfect and so React Native comes with its own set of small gotchas. None of them are deal breakers for us, but you should still know about them before starting a React Native project.

The biggest one by far is the fact that React Native is evolving very quickly, which of course is great, but it does incur a cost. There is a new version every two weeks (along with a release candidate of the next version) and the “Breaking Changes” section is often extensive. If you update on a regular basis you’ll be mostly fine; the bigger potential issues come from the third party libraries you use. As an example, now you import React from react instead of from react-native which is a problem if the third party libraries have not been updated to do the same. On top of that, Facebook definitely values pushing forward more than safely updating which was most apparent in an update to Babel 6 that broke many libraries and took weeks for everyone to catch up. All of this basically means you need to put a little more time aside for keeping up-to-date than with more mature frameworks.

The other little pain point is documentation. It is not as extensive as we wish and there are still some completely undocumented features / components, but this is getting better and you can see a big investment happening in this area.

Why React Native worked well for us

The main driver for us at SoundCloud to use React Native was the lack of enough mobile engineers to start development on a new mobile app. This is probably one of the main arguments for React Native if you are in a similar position of having more web engineers than mobile engineers. While this situation still holds true for us to some extent we have come to appreciate React Native for many more reasons than its JavaScript roots.

It is quite obvious that a better developer experience was a top priority for Facebook when creating React Native and it really shows. Automatically live reloading your app when your code changes massively shortens the feedback loop and using Chrome to debug your application feels very much at home to any web developer. If you also happen to use Redux and a logger middleware, knowing what is going on in your app at any point is very straightforward. It is a very comforting feeling to know the tools are there for you when you need them and its effect on developer happiness, development speed and confidence can not be understated. If you need one reason to give it a try, this is it. Happy developers and a faster turnaround are hard to beat. Product managers, decision makers & designers listen up, I’m talking to you here :)

But there’s more: the open source community provides many ready-to-use libraries and components and is very responsive to React Native’s changes, questions and improvement suggestions. It feels good to be part of this positive community.

Another huge benefit that we only recently started to explore is the cross platform nature of React Native. Facebook mentions about 85 percent code reuse between their iOS and Android implementation of Ads Manager, and with the right app structure that number is very realistic to expect. This means that once you have the structure there, building a new feature for your app on both platforms will only have an additional cost of 10 - 20 percent instead of 100 percent. That’s huge.

The ability to update your application on the fly, as mentioned earlier, also allows immediate bug fixes, a/b tests and the warm, cozy feeling that you can push an update whenever you want: a dream come true in the mobile development space (as long as Google and Apple allow it, which is always a little hard to predict).

What’s next

Based on the great experience we had with React Native, we plan on using it in more places at SoundCloud. Right now there is a team of designers working on a replacement for our prototyping / user-testing application written in React Native. Our designers found it much simpler to work on a React-based application than on a native application and are capable of building the application by themselves without frequent developer input. They are also working on bringing this application to Android to increase the pool of users that we can test with and to test Android-specific features.

Speaking of Android, we have spent some time to prepare the app for running on Android and got a stripped-down version of the app working in just a couple of hacker time sessions. We are currently investigating further steps in that direction, such as replacing the current Android Pulse app with the React Native version. Another area we are looking into is integrating React Native views inside our native applications. We’re looking forward to these experiments and hope they go as well as this one.

Get in touch with us

Jan Monschke
Peter Minarik @pietropizzi

Removal of Groups endpoints

Mon, 01 Aug 2016 00:00:00 GMT

As well as adding new features and updates, we review existing features to see if they’re still beneficial to the community.

As we dug into the best ways for curators to connect with artists and fans, we found that Groups aren’t working as well as reposts and curated playlists.

With that in mind, we’ve decided to phase out Groups on Monday, August 22nd to make room for future updates.

From this date, all Groups-related GET requests will no longer return results, and PUT requests will not persist any changes.

If your application relies on Groups, please update it by August 22nd, 2016.

Synchronous communication for microservices: current status and learnings

Wed, 27 Jul 2016 00:00:00 GMT

Since we started breaking our monolith and introduced a microservices architecture we rely a lot on synchronous request-response style communication. In this blog post we’ll go over our current status and some of the lessons we learned.

Finagle and jvmkit

Most of our microservices are built on top of Finagle and are written in Scala. Finagle is a highly extensible and protocol agnostic RPC framework developed by Twitter.

Finagle clients and servers use non blocking I/O on top of Netty. Non blocking I/O improves our scalability since we have a lot of scatter-gather style microservices or microservices that are I/O bound on external resources like databases.

Finagle is also highly configurable and extensible which means we can tailor it to work well with our infrastructure.

To make it easy for developers to build microservices that play nicely with the SoundCloud infrastructure and conventions we have an internal library called jvmkit which is maintained by our Core Services team. Some of the functionality jvmkit provides:

Wrappers around Finagle HTTP, memcached and MySQL with reasonable default configurations and integration with Prometheus
DNS based service discovery implementation
HTTP request routing on top of Finagle
JSON parsing support for request / response entities
Logging support according to our standards
API support for rollout our feature toggle system

Teams are responsible from development till running their microservices in production so they can decide not to use jvmkit as long as they stick to our standard protocol. In practice however most of our microservices do use jvmkit. By using jvmkit teams have more time to focus on delivering business features.

HTTP/1.1 + JSON

Almost all our services use HTTP/1.1 + JSON for communication.

The advantage of HTTP + JSON is that it is supported everywhere so we don’t lock ourselves into a specific technology or programming language.

There is very good tooling support for HTTP + JSON which makes it easy to debug and interact with our services.

The HTTP method indicates useful properties of requests. It indicates if a request is idempotent and safe (read-only). These properties tell us if requests can be safely retried or cached. We found this useful when replaying requests to test new revisions of a service. Indication of these properties are missing from other protocols we looked into.

Of course HTTP + JSON results in more overhead compared to other more optimized protocols. We have some services using Finagle ThriftMux and have thought about making ThriftMux the recommended protocol but we stepped away from that idea. ThriftMux is not ubiquitous and would lock us into Finagle if we want to get all the benefits.

Client side service discovery and load balancing

When we started using microservices we used separate load balancer processes as entry points for our services. With the increase in traffic we ran into failing load balancer processes which caused several outages. Our load balancers were single points of failure.

These outages made us move to client side service discovery and load balancing. We wrote a Java service discovery implementation that queries DNS SRV records to get the instances for a given service. Writing it in Java means it can be integrated in any JVM based language and framework that supports Service Discovery. We obviously integrated it in Finagle by using the Finagle Resolver api. Having support in Finagle means we have client side service discovery and load balancing support for HTTP, ThriftMux, MySQL and memcached.

Resilience against failures

When investigating the root cause of some of our outages we realized that we had to invest more in improving our resilience against failures. It quickly showed how important it is to use a mature library or system like Finagle which is already equipped with important features like:

Timeouts: every request across the network needs realistic timeouts. This protects you from issues with the systems you depend on.
Transport and application level Circuit Breakers: supports ignoring hosts that become unavailable or unhealthy with automatic recovery when they become healthy again. It protects you from issues with systems you depend on and allows them to recover.
Conditional retries: retrying can be effective but you shouldn’t retry unconditionally as this can result in retry storms that can overload the systems you depend on. Retry budgets allow you to configure a percentage of requests that can be retried on top of a minimum number within a time frame.
Limiting concurrent requests: If you know the capacity your service can handle you can protect it from overload by limiting the maximum number of concurrent requests and waiters (queue size).

Using and configuring these features have allowed us to better deal with failures which are inevitable in a distributed system like a microservices architecture.

We also introduced automated integration tests for jvmkit which show and document how our systems will react to failures. These tests also notify us about potential behavioural changes when upgrading Finagle.

No more client libraries

When we started implementing microservices teams who implemented a service typically also provided a client library that could be used by other teams to access their service. Those client libraries were implemented in the same programming language as the service and were hiding network communication and serializing / deserializing of requests and responses behind an easy to use api. The idea was that the code to access a service only needed to be written once and could be re-used by many.

After a while we noticed the client libraries approach also brings a lot of disadvantages:

A client library brings in dependencies. It might bring in a HTTP library, JSON parsing library etc. This might result in conflicts with the libraries already used by your service. If you depend on more than 1 client library this problem only increases resulting in unnecessary dependencies and conflicts between teams.
A client library might become bloated with custom logic. Using the client library might be the only way to successfully use the service and prevents it from being used by applications using other technologies. This imposes the use of a programming language and technology on other teams.
A client library can have default error handling logic which is probably not a good fit for all use cases. Users of a service should be aware of the error scenario’s that can happen and act upon them in the best way for their use case.
A client library will likely parse all returned fields of a response even if the client is only interested in a subset. This not only might have performance implications but also violates the tolerant reader principle.

We think these disadvantages far outweigh the benefits. The use of client libraries results in increased coupling and dependencies. Instead of publishing clients we encourage teams to publish their API’s. Consumers write their own clients in a way they find most appropriate using their technology of choice.

What’s next?

We hope to introduce Consumer Driven Contract tests soon. These tests will catch breaking changes between services and consumers during build time. Some teams are currently working on proof of concepts and are building out and validating the process.

A logical step for us would be to move from HTTP/1.1 to HTTP/2 as our default protocol. This would improve latency while sticking with a widely supported protocol without technology or programming language lock-in.

We might also replace JSON with a more efficient serialization protocol like protocol buffers. This would further improve performance and should result in less handwritten serialization and deserialization code but this still needs more experimentation and investigation.

Prometheus has come of age – a reflection on the development of an open-source project

Tue, 19 Jul 2016 00:00:00 GMT

On Monday this week, the Prometheus authors have released version 1.0.0 of the central component of the Prometheus monitoring and alerting system, the Prometheus server. (Other components will follow suit over the next months.) This is a major milestone for the project. Read more about it on the Prometheus blog, and check out the announcement of the CNCF, which has recently accepted Prometheus as a hosted project.

In just a few years, Prometheus has gone a long way from its beginnings as a pet project of Matt T. Proud and Julius Volz to a widely adopted monitoring system with an amazingly active community of more than 300 contributors and a user-base spanning many different organizations, including companies like CoreOS, DigitalOcean, Docker, Ericsson, Percona, and Weaveworks. A pivotal midpoint in the history of the project was the public announcement in January 2015 in an earlier backstage blog post.

In this article, I would like to reflect a bit on the childhood of Prometheus at SoundCloud, and on the benefits a company can gain from developing projects in the open under a free software license.

SoundCloud has a very liberal open-source policy. We have published a large number of projects in our GitHub organization. Matt and Julius, who had joined SoundCloud as engineers back in 2012, knew from the beginning that Prometheus would have a much larger scope than any of those projects. When they suggested that SoundCloud should invest in the development of a completely new monitoring system, the resistance they had to overcome was not so much related to the open-source character of the project but to its scope. After all, why should a music-streaming company with a relatively small engineering force dare to fundamentally change the way open-source monitoring is done? It was a bold move, indeed. But it was also the time of #monitoringsucks, and no open-source effort in sight to make a real dent. So we took the torch and went on our Promethean mission, cautiously avoiding the eagles.

In hindsight, we know that it has paid off. For one, SoundCloud has just finished the quarter with the highest availability in the history of the company — which was also the quarter with the most features and the highest complexity the site ever had. This tremendous accomplishment would have been impossible without a novel way of monitoring our growing microservice architecture — the way enabled by Prometheus.

As the main source of early contributions to the project, SoundCloud was also the earliest adopter. A nice strategic technical advantage, which – naively thought – could have been so much larger if Prometheus had been developed in secret. However, that would ignore the overwhelming advantages of an open-source development. First and foremost, without the countless contributions from the community, Prometheus would not be what it is today, and the technical advantage would be much smaller. Kickstarting a successful open-source project that is then joined by many other parties is a good way for a small company to get something going it could never have accomplished on its own. The SoundCloud open-source policy states “free bug fixes from the community” as one of the key benefits of open-sourcing projects. In the case of Prometheus, that’s quite an understatement, to say the least.

Tech credibility and the resulting impact on recruiting is another important benefit. Excellent engineers like to work for companies with excellent projects, and they like to work on those projects in the open, where they benefit a whole community and provide visibility for the engineers, too. And then there is a perhaps surprising side-effect we can observe by now with Prometheus: If an open-source project is popular, some of your new-hires will already be familiar with it, resulting in a smoother and shorter onboarding.

Prometheus has grown up to a real titan. We are eagerly looking forward to the further evolution of the project and to what humankind will do with the fire that was given to them.

Björn is a production engineer at SoundCloud. He joined the company and the Prometheus project in 2013 and has been a Prometheus core developer ever since.

Building radio stations at SoundCloud

Tue, 05 Jul 2016 00:00:00 GMT

Over the last 100 years we have dialed into radio stations at home, on the road, or in the office to access a curated mix of top hits delivered to us by our favorite DJ. With more and more of our daily activities taking place online, we find our source of music now comes from a mix of our mobile phones, our desktop computers and the radio - and is available to us whenever we need it. The amount of music available today is endless. The music scene has never been so vibrant. A proliferation of production tools at low cost and the ease at which an artist can share their sounds, gives everyone the opportunity to share their work at any time, with only a few clicks.

Keeping this in mind, we set out to determine: How can we bring a radio like experience to SoundCloud that would enable you to tune-in and lean back?

The lean back experience

We consider the lean back experience those moments when you need music but don’t have the time to curate or choose something specific - at the gym, during your commute or those moments in the office when you need something to help your through the day.

The secret sauce for a successful radio station

Imagine a radio station that jumps from a jazz instrumental to the new EDM ‘floorkiller’. That just wouldn’t work.

To create an enjoyable radio experience, it’s important the experience maintains consistency and context for the user. To create this experience on SoundCloud, we considered the following key ingredients that have lasted the test of time for radio as we built our latest discovery feature, Artist Stations.

Novelty. You want to hear something new. That’s why the mainstream radio stations revamp their playlists periodically, usually every month, so listeners return to hear the newest hits.

Familiarity. At the same time, you also want to hear something that’s familiar. You know the lyrics, or it’s an artist you like, or simply you heard that track before. Music often it closely tied with life events and experiences and hearing a familiar song can trigger memories that make us smile.

Context. Since we are in a lean back experience mode, our brain works better when changes in external factors are minimal. Imagine a radio station that jumps from a country song to the new EDM ‘floorkiller’. That would not work, that’s why providing context and maintaining that context consistently is crucial for every radio experience.

The future of radio stations.

Two weeks ago we launched Suggested Tracks on Mobile and Discover tab on Web, the first of a series of features built on top of our machine learning infrastructure.

We strongly believe the content available on our platform is unique, and so are the millions of listeners that tune-in on SoundCloud every day. Each of them has their own specific taste and sense of what is novel and familiar.

By unlocking our catalog to algorithms trained to behave as radio curators, we can get the best of both worlds — an exceptional traditional radio experience coupled with an endless catalog of content available to you with just a few taps on your screen.

This is the ultimate democratization of the radio experience. Our machines are composing a radio experience by detecting what is new, what is familiar to the user, and what is trending or popular among listeners in the SoundCloud community. All while keeping the context consistent with the Station you’ve selected, and your personal taste profile.

Continuous improvement

We are extremely excited to share this feature with you, and I hope you’ll enjoy it as much as we do. Track and Artist Stations are currently available on mobile, and will be coming to web in late summer.

As our algorithms are constantly in evolution and our data models continue to refine themselves, the more you listen, the better suggestions you will receive and the more input we will receive, the better stations perform for everyone on SoundCloud. Stay tuned for more iterations of stations and more ways to discover the more than 125 million tracks available on SoundCloud.

Can a machine surprise you? We believe so.

Tue, 21 Jun 2016 00:00:00 GMT

With more than 125 million tracks from over 12 million creators heard each month on our platform, SoundCloud is uniquely positioned to offer listeners a full spectrum of music discovery.

Classic hits, the latest releases, gems from underground talent and the best of what’s up-and-coming – all in one place.

How can you make great content discoverable and available at ease? How can you create a unique experience for every single user?

Beginning today, we’re introducing Suggested Tracks, on web and on mobile, to bring listeners even more ways to discover music on SoundCloud.

Digital Serendipity

Months before Suggested Tracks developed to what it is today, SoundCloud’s recommendation team set out to answer one simple:

**_Can a machine surprise us?_**

For quite some time, the SoundCloud Core Data Science team has been working on state of the art machine learning projects to help its users discover new content. The idea of surprise in relation to discovery was a new concept. In order to further explore and develop this idea, it required a very different approach and perspective.

We concluded that music generates emotion, and therefore we would need to view the element of surprise from the perspective of emotion.

We knew if we really wanted to infuse the element of surprise into music discovery, we had to push our systems to understand the emotional element of our users.

It was about taking traditional ways of discovering music to a new level. To a place where feelings were taken into consideration, and the “how” along with the “what,” was addressed in the computational effort.

Overall, we wanted to create a serendipity effect. The moment where you find something new, something engaging that makes you move naturally from a passive listening experience to an active state of mind. The moment where….

*The bass kicks in, you’re 30 seconds into the song, it’s a catchy groove. It sounds familiar, but you’ve never heard the track before. At 45 seconds, you have already liked the track by pushing our iconic heart button, and by the time the lead singer gets to the bridge, you have already shared the track on your favorite social network.*

It’s a eureka moment. You just found a new, undiscovered track none of your friends know. And you love it. You feel it is the right track at the right moment. And it feels good.

What if that track was recommended by a machine? An extremely sophisticated algorithm that just picked the track for you with the simple goal of surprising you?

This is what we call Digital Serendipity.

It’s all about you

So can it be true? Can a machine know you that well? Not just recommending good tracks but amazing tracks?

We believe the answer is yes. The very same machine has been learning from you all this time. It’s as if you had a skilled musicologist right next to you, carefully listening to what you’ve been playing and interacting with, all this time.

With Suggested Tracks, this musicologist is not only a passive observer but a proactive entity. The more you listen, like, repost and engage with a track, the better our artificial agent gets to know you and your musical taste. The more you interact, the more it learns and perfects itself day after day.

Micro Moments

Suggested Tracks is not a recurrent playlist or a weekly chart. Our new feature updates more frequently than any other service out there so you always have fresh new content for your listening pleasure. Now all of life’s micro moments will have the perfect soundtrack.

Enough about Suggested Tracks, go and try it out!

Data pipelines with Apache Crunch and Java 8

Wed, 01 Jun 2016 00:00:00 GMT

With Java 8 now in the mainstream, Scala and Clojure are no longer the only choices to develop readable, functional code for big data technology on the JVM. In this post we see how SoundCloud is leveraging Apache Crunch and the new Crunch Lambda module to do the high-volume data processing tasks which are essential at early stages in our batch data pipeline efficiently, robustly and simply in Java 8.

Hadoop MapReduce is not dead

It may not be cool or trendy any more, but Hadoop MapReduce (based on the Google MapReduce paper from 2004) still has an important role to play in today’s world of big data technology. Whilst Spark captures people’s imagination as a replacement for it, with its interactive response times and suitability for machine learning, it’s not a silver bullet to every batch processing requirement. It can be unstable at times, and uses far more cluster resources (especially memory) than its simpler predecessor. Spark is used in the application domain for products like search, recommendations and classification tasks, but for preparing our datasets for downstream usage, MapReduce still has a huge part to play. It still provides a robust and relatively reliable platform which is ideally suited to transforming, cleaning and partitioning our huge data volumes ready for use by the insights and product engineering teams.

Type safety is crucial

Working with data at this end of the pipeline has different developer requirements too. Whilst at the application end of the pipeline the workflows are often based on picking out individual fields from the data and aggregating them in various ways (such functionality is well provided on MapReduce by Hive and Scalding), at early stages of our pipelines we are working with complete objects with strict schemas to adhere to, so trying to work with a fields-based API would be very dangerous.

Event data at SoundCloud is stored in the structured binary Protobuf format, so being able to work directly with Java Protobuf objects rather than an API-specific view saves us lots of mapping code, and prevents errors if we were to get it wrong.

Crunch provides a useful abstraction for this kind of workload, because it works with type safety and structured records at its heart. It’s based on Google’s FlumeJava paper, and provides a general-purpose boilerplate-free developer-friendly typesafe Java API for developing data jobs to run on Hadoop MapReduce.

Crunch code is relatively simple to understand, as this classic “word count” example demonstrates:

Pipeline crunch = new MRPipeline(WordCountJob.class);
crunch.read(From.textFile("/path/on/hdfs"))
      .parallelDo(new DoFn<String, String>() {
          public void process(String s, Emitter<String> emitter) {
              for (String word: s.split(" ")) {
                  emitter.emit(word);
              }
          }
      }, strings())
      .count()
      .parallelDo(new MapFn<Pair<String, Long>, String>() {
          public String map(Pair<String, Long> wordCount) {
              return wordCount.first() + ":" + wordCount.second();
          }
      }, strings())
      .write(To.textFile("/path/to/output"));
crunch.done();
}

Reading from top to bottom, we see that we’re reading a collection of Strings from a text file, then applying a DoFn which tokenizes each line into words and outputs each word. We then use the built-in count() method to count occurrences, then we apply a MapFn operation which formats those pairs back into String lines and writes them to a text file. The execution is lazy, so crunch.done() tells Crunch to run everything.

Unfortunately this code snippet suffers in the same way that a lot of functional-style Java code has suffered with over the years. Those anonymous inner classes are ugly and add a huge amount of redundant information, making it difficult to read and irritating to write.

Java 8 to the rescue

Luckily, Java 8 gave us the gift of Lambda expressions, method references and Streams; all of which are intended specifically to make functional programming in Java more user friendly. With the rest of our code starting to look more functional and concise, the Crunch parts of the code started to look even more archaic then they did before, so I set about building a thin Java-8-friendly wrapper around the Crunch API to make the situation better. That wrapper became Crunch Lambda, which as of version 0.14.0 is now part of the main Crunch project (in the crunch-lambda submodule, artifact org.apache.crunch:crunch-lambda:0.14.0)

Mini side-note about Crunch versions: Some people are worried about using Crunch because it’s not version “1” yet, but Crunch version numbers are incredibly conservative. We and many other large organisations have been using Crunch in production for some time and we consider it very stable.

Crunch Lambda

Let’s revisit the above example with Crunch Lambda instead:

Pipeline crunch = new MRPipeline(WordCountJobLambda.class);
Lambda.wrap(
  crunch.read   (From.textFile("/path/on/hdfs")))
        .flatMap(line -> Arrays.stream(line.split(" ")), strings())
        .count  ()
        .map    (wc -> wc.first() + ":" + wc.second(), strings())
        .write  (To.textFile("/path/to/output"));
crunch.done();
}

See how the type information is only required once now for each transformation and there’s no distracting superfluous information.

What about the real world?

Let’s take a look at how we’re using Crunch at SoundCloud in one of our data jobs.

The EventStatsJob counts the number of events that have successfully made it through to the end of the core event pipeline and splits it out by application and event type. We use this data to compare against data collected in real-time as events are first collected to check for any anomalies in our computation.

Once it is collected the data, it emits it as data points to a Prometheus PushGateway so they can be graphed over time.

Pipeline crunch = new MRPipeline(EventStatsJob.class);

// Prometheus setup
PushGateway pushGateway = new PushGateway(PUSH_GATEWAY_ADDRESS);
Gauge gauge = Gauge.build()
    .name("event_count")
    .labelNames("event_type", "application")
    .create();

// Read events from Protobuf SequenceFile
LTable<Long, Event> events = 
  Lambda.wrap(
    crunch.read(
      From.sequenceFile(
        dataPath,
        Writables.longs(),
        PTypes.protos(Event.class, WritableTypeFamily.getInstance()))));

// Count events by their stats key and set Gauge values
events.values()
      .map(StatsKey::fromEvent, StatsKey.pType())
      .count()
      .materialize()
      .forEach(statsRecord ->
          gauge.labels(
                  statsRecord.first().eventType,
                  statsRecord.first().application)
               .set(statsRecord.second()));

// Send the gauge contents to the PushGateway
pushGateway.push(gauge, "event_stats_job");

Here we see how simply Crunch code can interoperate with normal Java code. It calls out to a nearby normal method to perform a distributed map operation (via the StatsKey::fromEvent method reference) and then brings the results back locally via the materialize() method to operate on as a normal Java Stream. Note that we didn’t even specify an output file, because they only output from this job is to feed metrics to Prometheus; Crunch handles all the temporary files and IO for you.

The StatsKey custom type is a simple immutable wrapper for the eventType and application (it has a few more fields in real life, but these are omitted here for brevity). We can easily work with this in Crunch by giving it a conversion mechanism in and out of a built-in Crunch type, as seen in the pType() method here:

public static class StatsKey {
  public final String eventType;
  public final String application;

  public StatsKey(String eventType, String application) {
      this.eventType = eventType;
      this.application = application;
  }

  public static PType<StatsKey> pType() {
      return derivedImmutable(
              StatsKey.class,
              mapFn(t -> new StatsKey(t.first(), t.second())),
              mapFn(k -> Pair.of(k.eventType, k.application)),
              pairs(strings(), strings()));
  }

  public static StatsKey fromEvent(Event event) {
      /* ... */
  }
}

This PType serialization/deserialization behaviour descriptor is provided built-in to Crunch for many serialization types, including Protobuf (as you can see with the PTypes.protos call above) and Avro.

A more complex example - Creating toplists

As part of the Soundcloud Stats product, we present toplists to sound creators in different dimensions - for example “my most-favourited tracks” or “top countries where my sound is played”. This is the main part of the code to compute these:

LTable<Bucket, Long> summedBuckets =
  events
    .flatMap(this::createFactsFromEvent,
             tableOf(proto(Bucket.class), longs()))
    .groupByKey()
    .reduceValues((a, b) -> a + b)
    .filterByValue(count -> count > 0L); // Remove negative counts

return summedBuckets
    .map(this::remapDimensionKeys,
         tableOf(proto(FactKey.class), proto(TopListPair.class)))
    .groupByKey()
    .mapValues(rowSet -> findTopK(k, rowSet),       
               collections(proto(TopListPair.class)));

Here you can see that we are doing some much more complex operations than in our previous example.

Firstly we transform each events into a set of “facts”. A “sound played” event, for example, might generate several facts. One based on which country it occured in, one based on which creator published the sound, and so on. The createFactsFromEvent method takes an Event as its parameter, and returns a Java Stream of facts corresponding to each of these so-called “dimensions”. We group by the key (eg. ‘myawesomesound’), dimension (eg. “country”) and its value (eg. “Sweden”) - together called a ‘bucket’ - and sum the Long values as a reduce operation. As some events (such as “sound un-favourited”) can cause a negative impact on the counts, we filter out any totals which are non-positive, as these tend to look strange to users.

Following this, the remapDimensionKeys operation effectively moves the dimension value (eg. “Sweden”) from the key part of the record to the value part, so we’re left with a FactKey(“myawesomesound”, “country”) and a TopListPair(“Sweden”, 581). We then groupByKey() with this new key to collect all the TopListPairs for each FactKey, then run them through the findTopK to obtain the K TopListPairs with the greatest count.

public Collection<TopListPair> findTopK(int k, Stream<TopListPair> input) {
   SortedSet<TopListPair> set = new TreeSet<>(
           Comparator.comparingLong(TopListPair::getCount)
                   .reversed()
                   .thenComparing(TopListPair::getDimensionValue));
   input.forEach(pair -> {
       set.add(pair);
       if (set.size() > k) {
           set.remove(set.last());
       }
   });
   return set;
}

As you can see, there’s absolutely nothing unusual about this part of the computation. We use a standard streaming top-k algorithm to create the result set for the key.

Testing

Finally, one of the greatest advantages of using Crunch is how it facilitates easy local testing of your pipeline logic. A combination of an in-memory implementation of the Pipeline object MemPipeline and the fact that the API encourages using normal Java methods to compose your pipes (for simple low-level unit testing) means that it’s really easy to have confidence in your pipeline before it even makes it on to the cluster.

Future

We’ve got our eyes firmly on the Apache Beam project as it progresses. Inspired by work from FlumeJava, Crunch, Google DataFlow, Spark and Flink (and also running on at least the latter 3) it aims to generalise the expression of data pipelines like this away from Hadoop altogether and break the cycle of having to rewrite everything to keep up with the latest technology. This sounds like a big step in the right direction, but we’re confident writing new Crunch code now isn’t a waste of time either, as it looks very likely so far that translating from Crunch to Beam would be a relatively simple operation when the time comes.

David is a Data Engineer at SoundCloud and a committer to the Apache Crunch project. You can learn more by reading the Crunch User Guide and the Javadocs for Crunch Lambda. If you’re interested in working on problems like these, check our open positions on the SoundCloud Jobs page

Playlist search returns only compact representations as of July 15, 2016

Fri, 13 May 2016 00:00:00 GMT

We previously introduced new playlist representations. The compact representation returns only the playlist itself without any of the tracks. For most users, this is preferable – the compact representation significantly reduces the size of the JSON payload for large playlists and provides faster API response times.

Beginning July 15, 2016, all queries using the q parameter will only return this compact representation for playlists. Other playlist representations will no longer be available when using the q parameter.

To fetch the track listings for one of the returned playlists, you can make an additional request to the /playlists/{id} resource by using the corresponding playlist ID.

If your application expects track listings from playlist search results, update your application by July 15, 2016.

Open-sourcing LightCycle for Android

Mon, 21 Mar 2016 00:00:00 GMT

Last week, we open-sourced LightCycle, an Android library that helps break logic out of Activity and Fragment classes into small, self-contained components called LightCycles.

Components that typically need to be aware of Activity and Fragment lifecycle events include presenters, UI tracking code, input processors and more. We’ve been using LightCycle extensively in the SoundCloud Music & Audio and SoundCloud Pulse apps over the last year.

Motivation

LightCycle lets self-contained classes respond to Android’s lifecycle events. This encourages composition over inheritance and promotes components that follow the single responsibility principle. We believe it helps us write more readable, maintainable and testable code.

If this sounds familiar, RoboGuice has a similar concept called Events. However, LightCycle is a small, standalone solution that works well with all dependency injection frameworks.

Usage

public class MyActivity extends LightCycleAppCompatActivity<MyBaseActivity> {

    @LightCycle MyController controller = new MyController();

    @Override
    protected void setActivityContentView() {
        setContentView(R.layout.main);
    }
}

public class MyController extends DefaultActivityLightCycle<MyActivity> {

    @Override
    public void onPause(MyActivity activity) {
        // MyActivity was paused
    }
}

The Activitiy and Fragment classes are responsible for configuring Android specifics and declaring LightCycles. LightCycles themselves can focus on business or presentation logic. Not being tightly coupled to the framework makes these components simpler to unit test (even without Robolectric).

Get it

The code and documentation is available on GitHub.

Acknowledgements. Huge thanks to all the contributors: Brian Egan, Fernando Cejas, Jamie McDonald, Jon Schmidt, Karolina Kafel, Matthias Käppler, Pedro Lopes.

Congratulations, you have a lot of code!"Congratulations, you have a lot of code!" Remedying Android’s method limit - Part 2

Tue, 06 Oct 2015 00:00:00 GMT

In part one we described how running into Android’s method limit may leave you unable to build, and offered strategies you can employ to make your app fit into a single DEX file. In this part we share an alternative option: using multiple DEX files.

MultiDex

If you have exhausted your options for slimming down your DEX, then there is really only one option left: MultiDex. Enabling MultiDex mode is a simple switch in your build scripts, and it allows you to grow your app beyond the 65k method limit by letting extra code spill over into additional DEX files. This sounds like a great option, but it has a number of repercussions you should be aware of, especially when using it in production and targeting pre-Lollipop versions of Android. The official documentation has more information about these problems.

In a nutshell, multidexing means chopping up your code into multiple DEX files, where normally just one DEX file is created. ART—Android’s new runtime—deals with this effortlessly, since any number of DEX files will be compiled into a single ELF binary before your application loads. However, if you’re aiming at a minSdkLevel below 21 (Lollipop) and need to run on devices that use Dalvik, the tool chain must perform some extra bookkeeping for this to work. This comes with some strings attached:

You must include the MultiDex support library and modify your application to either inherit from MultiDexApplication or invoke the MultiDex loader manually. This is because Dalvik cannot natively deal with multiple DEX files. Luckily, this is safe to do even if you don’t use MultiDex in production builds.
Your build times will increase, since the tool chain will employ ProGuard internally to split out your main application components and their direct dependencies into a primary DEX file, while moving all remaining code to a minimum number of additional DEX files. According to Android SDK lead Xavier Ducrohet, the Android Tools Team is “more than aware than this is a huge problem right now and we’re working on fixing this, but it’s going to be a while.” (quoted from a Slack conversation)
The preDexLibaries flag will become inoperable, because the above step cannot be carried out without taking into account all class files, including those from external dependencies. This means that whenever you change a single line of code, the tools need to completely re-dex your app, including all library dependencies. ART does not have this limitation, as all DEX files (one for each runtime library plus those containing your app code) will be embedded in a single OAT file when being compiled down to machine code. Libraries can still be pre-dexed at build time, because they don’t need to be combined at build time.
To use libraries which need to be aware of the full class graph, like Dagger, make sure that additional DEX files are loaded before you create your object graph. That means you cannot create your object graph in your application constructor anymore; instead, you can override attachBaseContext and create it after calling through to super.

A workaround we adopted to get back some of the lost build performance is to create a product flavor which specifies minSdkLevel 21. The toolchain can assume ART as the sole runtime at this API level, and using this product flavor during development is a fair compromise. Keep in mind though that this will affect things like the lint tool, which checks for misuse of platform APIs taking into account your range of API levels. That said, with the workaround in place your build server should still build product flavors that specify the shipping API levels.

To get an idea of just how much build times would be improved by applying the suggested workaround, we ran a few tests and timed different kinds of builds both on the CLI and in Android Studio using MultiDex. What we measured were 4 things:

time to test (TTT) using gradle-aware make: time it takes for the test results window to appear in the IDE when running a unit test, i.e. actual test run time is not accounted for
TTT using Android Studio’s default IntelliJ style make
time to launcher (TTL): time it takes for the launcher window to appear in the IDE when building/running the app,
building on the CLI using the assemble[Flavor]Debug task.

All of those were run in three variations: after a clean, after no changes, and after a single source file had changed.

Here, time is measured in seconds, so lower is better. From the graphs, looking at the orange bars, you can see that incremental build times have improved drastically when targeting 21 in development, since library pre-dexing is functional again, both in the IDE and on the command line. Another interesting find here is to leverage what in IntelliJ/AndroidStudio is referred to as “Gradle-aware Make”, which consistently outperforms the default setup when running tests. In the configuration of your IntelliJ runners you can specify which jobs will be executed before launching a test or the app. The default is to use “Make”, which apparently will undermine some of the sophistication we get out of a Gradle based build in terms of incremental builds. By changing this step to app:assemble[Flavor]DebugUnitTest you can shave off unnecessary time from your test runs.

Moving to native

Just to look at the problem from multiple angles, another way to sidestep the DEX method problem is to simply not make code end up in the DEX file to begin with. Pushing code down to the native layer has the benefit that it can live in separate binary modules that can be side loaded at runtime, thus not adding to your overall method count during build time. The decision of whether this is a sensible step to make for you depends largely on your product, organization and team expertise and shouldn’t be made lightly, so we’re merely pointing this out for the sake of completeness.

Method count visibility

So far we focused on means to solve the DEX count issue. In order to not have it happen at the most inopportune time, it would be desirable to raise awareness around the current method count and notify developers when approaching the method limit. At SoundCloud we have started to monitor application size by plotting the DEX method count trend to a Jenkins graph using the Plot plugin:

We leverage a Gradle plugin to collect the method counts for different build types and store them for every build. Once we go past a certain threshold, we will mark the build as unstable and inform the developers via email. This will hopefully give us enough leeway to address the issue before maneuvering ourselves into a corner again.

Conclusion

Working in a fast moving environment is challenging, and being able to keep delivering product updates is essential to the business. To prevent worst case scenarios like the inability to build, developing both more sensitivity around the cost of third party libraries and getting deeper insights into application health can help. Make sure you understand the impact of leaning on external libraries: how much hidden complexity are you dragging into your app by using it? In case of hitting the limit, know your options: trimming down dependencies and MultiDex is what we ended up leaning on. Last but not least, visibility is king: raising awareness around application size and the looming method limit will allow you to take action ahead of time.

If you’d like to join the Android team, check out our current openings here

Introducing SoundCloud JavaScript SDK 3.0.0

Thu, 01 Oct 2015 00:00:00 GMT

We are happy to announce version 3.0.0 of our SoundCloud JavaScript SDK.

The new SDK improves stream security and content uploading functionality, and modernizes the technology stack.

Version 3 of the SoundCloud JavaScript SDK is a major update and is not backwards compatible. That said, the changes that you need to make your web app work with the new SDK are easy to implement. Please refer to Migrating to JavaScript SDK 3.0.0 to upgrade.

ECMAScript 2015 and CommonJS

The original version of the SDK was written in CoffeeScript, which is no longer a core technology at SoundCloud. This update provided us the opportunity to migrate our source code from CoffeeScript to ECMAScript 2015.

The new SDK is now using the Babel compiler for ES2015 support, and webpack as our bundler. This has the additional benefit of making the SoundCloud JavaScript SDK 3.0.0 compliant with CommonJS.

Because of this, we can now take advantage of the variety of packages that are available via npm, and users can install the SDK via npm as well. Please refer to the npm page for details.

JavaScript Promises

Promises have become a core part of JavaScript with the new ES2015 specification, and they allow for better composability and a easier control flow. Internally, we work with Promises a lot and wanted to give external developers the ability to benefit from an easier API.

Web Audio API

The previous SDK neither provided a way to record sounds from Web Audio applications nor a way to upload Web Audio recordings. It shipped with a Flash component that handled recording and uploading of recordings. There was no way to specify an external file for uploading.

The new SDK ships with a recorder component that uses Web Audio and getUserMedia instead of Flash, and the component allows you to pass in arbitrary AudioNodes that will be recorded. This makes it much easier to integrate the SoundCloud JavaScript SDK 3.0.0 into creator applications that rely on Web Audio. The SDK provides a dedicated method to publish recordings directly from your web app.

Secure streaming

The new SDK now includes a new player component. This component improves security for creator content and provides the improved playback stability.

Documentation

We also took the time to all of rewrite our documentation and code examples so that you can start with the new SDK immediately.

Congratulations, you have a lot of code!"Congratulations, you have a lot of code!" Remedying Android’s method limit - Part 1

Mon, 21 Sep 2015 00:00:00 GMT

At SoundCloud we have been building for the Android platform since 2010. Much has changed since then: the team has grown, the list of features has grown, and our audience has grown. Today, eight engineers are working full time on the official SoundCloud app, across various areas, with contributions pouring in from other parts of the organization. Due to the growing complexity and number of contributions, the app’s size has grown substantially. Currently the app consists of approximately 1200 Java source files, not counting tests, containing approximately 86000 lines of code. This doesn’t include native code, such as our playback or recording stacks.

We’re not the first to run into Android’s limits in terms of build tools. An internal limitation of Dalvik’s byte code format (DEX), which I will explain in more detail, can leave you unable to build after your codebase reaches a certain size. If you fail to anticipate this, it might happen during the most inconvenient time, such as when you are preparing for a release. Part of our job in Core Engineering at SoundCloud is to make sure our developers are happy and productive; not being able to build our app anymore makes for neither happy nor productive developers.

While there are a number of posts on this topic, I would like to describe in more detail what we have done to combat Android’s method limit, what things worked well and what didn’t work so well, what it actually means to use the dx tool’s --multi-dex switch and what you can do to improve application health with regards to size.

The article is in two parts: part one lays out the basic problem and describes a number of options that you have to stay under the method limit. Part two focuses on Android’s MultiDex mode, which you can use to grow your app beyond the method limit.

The DEX Method limit

Android’s DEX format dates back to a time when applications were a lot smaller and devices were more resource constrained. To accommodate for this, the size of a DEX file’s method index is limited to 16 bits, which means if your application references more than 65,536 (2^16) methods in total, the dx tool will refuse to build your app. That number includes “Android framework methods, library methods, and methods in your own code”. 65k might sound like a lot, but when Android’s own support libraries account for about a third of that, and if on top of that you’re using one or two large libraries like Guava or Jackson, you rapidly approach that limit. To find out where you’re at, there are convenient tools to count the number of methods in your app’s DEX file.

To address a common point of confusion, the method limit described above is unrelated to “jumbo mode”, but it has a similar cause. Jumbo mode pertains to the number of strings that can be referenced in a DEX file, which by default are indexed using 16 bit wide integers. Therefore, if your application encodes more than 2^16 strings, the dx tool will fail as well. For string references however, there is a remedy: DEX supports “jumbo opcodes” which allow for 32 bit wide string references. The jumboMode flag in an Android Gradle build script enables this mode, allowing up to 2^32 strings to be referenced. However no equivalent switch exists for method references, so jumbo mode does nothing for you when your DEX file exceeds the number of methods allowed.

The following sections will outline attempts we made to address the method limit imposed on an application.

Overcoming the method limit

It looks like we’ve hit a wall–not being able to assemble an APK anymore is not where you want to be. There are a number of options to overcome the limit, each with drawbacks and benefits. Let’s touch on the most obvious one first: putting your app on a diet!

Trimming dependencies

When we first hit the DEX method limit, we made the mistake of taking shortcuts to dodge it, such as staying on outdated but smaller versions of libraries like the Play services. Don’t do that!

Sure, old dependencies wil buy you time for the short term if you’re stuck with a broken build. However, do invest effort into making sure you don’t immediately hit the limit again. Since removing dependencies from your app isn’t something you do in an afternoon and since MultiDex sounded scary and was something we initially considered a last resort, we opted for a hack: trimming down libraries using ProGuard and use those in debug versions of the app.

If the release build is healthy in terms of method count, but the debug build isn’t, it means we’re dragging around a lot of unused stuff, so why not get rid of it in the first place? Thus was born: The Repacker!

The idea is simple: have a Gradle task that performs a debug build with a special ProGuard config that only removes, neither optimizing nor obfuscating, finds all surviving classes for a particular library we’re targeting, rolls them into a new JAR and deploys it to our artifact repository under a new version. Then change the app to depend on the trimmed down version of the library which contains only the methods it actually needs. We’ve done this successfully for Guava, Jackson and the Facebook SDK, but it works best for libraries that change infrequently.

This has several drawbacks. First, if you do this with a library like Guava, where you don’t know in advance what parts of the library you want to use, you will have to keep repacking the library as you need different parts. Second, since ProGuard sometimes accidentally removes parts that are actually in use, one might have to repack multiple times with different configurations to achieve the desired result. Lastly, we lost support for source JARs and JavaDocs, although we could have brought them back with further workarounds. While it sounded good on paper, we found this solution difficult to work with in the end, and it will not help at all if you’re exceeding the method count even after ProGuard strips away classes.

Losing dependencies

Cutting down the number of dependencies might sound obvious, but there are a few things worth pointing out.

First, we as developers are not always conscious enough of the possible repercussions of our lasting choices. Here’s an example: choosing your building blocks. Take Guava for instance. If you’re not familiar with the library, it contains a wealth of utilities that aim to close many of the gaps in Java’s language library, especially when you’re stuck with Java 6 (which you are unless you’re targeting KitKat and above). Unfortunately, the library is massive; it’s a single JAR clocking in at roughly 5MB and leaves a substantial footprint on your method allowance. It’s easy to just drop that dependency in your build script and unlock all the niceties it offers, but since it operates at a foundational level, it is bound to appear everywhere. Libraries like these are very difficult to rip out of your application later on, making these choices expensive to revisit. Moreover, you’d be surprised how much stuff a single method you use can drag on board along with it. Guava in particular is highly self-referential, so using one part of it will most likely make you unknowingly use a good chunk of its other—superficially unrelated—parts.

We didn’t have this foresight and used Guava extensively; fortunately, it was one of those 80/20 things where in the majority of our code, we’d use a minority of the library. We took two weeks to remove Guava from our code base and salvage those high-value parts that we were using frequently into a utility library that is now fully owned and maintained by us. The great thing is that in the process we shed a lot of unnecessary code, such as Guava’s workarounds for GWT and older Java versions, which are all meaningless on Android. This made the code easier to read, test, and understand. Jackson is another offender in terms of method count. Because we didn’t encapsulate it sufficiently behind APIs we own, we haven’t yet been able to replace it. In summary: before you bake something you have little or no control over into the foundations of your app, think twice. The safest decisions are those that are reversible. Especially in phases of rapid growth and uncertainty which are inherent to software start-ups.

In the next and final part, we will turn to MultiDex and what you should know about when and how to use it.

Introducing Rate Limits

Tue, 16 Jun 2015 00:00:00 GMT

At SoundCloud, we’re building an ecosystem where creativity thrives. Developers are an important part of that ecosystem. We’re continually inspired by how you use the SoundCloud API to support creators and listeners in innovative ways.

But as the ecosystem has grown, we’re dealing with an increasing number of applications that abuse creator content by violating our developer Terms of Use. To help control this type of behaviour, we’re introducing a daily rate limit on API play requests.

Beginning July 1, client applications will be limited to 15,000 play requests per 24 hour period. These limits are applied to developer applications using the SoundCloud API, and have no impact on the SoundCloud embedded player.

Only a small number of developers will be affected by this change, and we’ve contacted them via email to ensure a smooth transition.

What this means for you:

You should review our developer documentation on rate limiting. Limiting API access is common for API providers. If you’re unfamiliar with rate limits or how they work, we’re here to help.
You should monitor your play requests.
Make sure the email account associated with your app is up-to-date, and keep an eye out for communication from our team.

This change does not affect the SoundCloud embedded player, and if you have not heard from us, your app is unaffected.

In the coming months we’ll be introducing an application process for developers who’d like additional access.

This change is an opportunity for us to refocus our efforts and renew our commitment to developers. You’re an integral part of the SoundCloud community, and we look forward to seeing what you build next.

New playlist representations

Mon, 08 Jun 2015 00:00:00 GMT

Requests for playlists have always included the full track objects contained within. This representation may be convenient for playlists with ten or twenty tracks but can cause problems for playlists that contain hundreds or thousands of tracks. Requesting such large playlists could result in requests that take a long time to respond and that eventually timeout.

Today, we introduce two new representations for the /playlists resource: compact and id.

If you add representation=compact to a playlist request, the request will return only the playlist itself, without any of the tracks it contains. This representation may be useful if you’re purely interested in data about the playlist itself and not the tracks contained within.

Alternatively, if you set the representation=id parameter, it will return the playlist along with IDs of the tracks contained within, without all the associated track meta data (artist, artwork, duration, etc.) If necessary, you can then individually fetch additional information about each track by filling those IDs into the /tracks resource. This allows for greater parallelization and can help make your application more responsive when working with large playlists.

By default, requests for playlists will continue to include all the contained tracks. There is no need to update your application but we encourage you to take advantage of these new, more efficient representations of playlists.

Apple's June 1 64-bit deadline

Mon, 18 May 2015 00:00:00 GMT

In October 2014, Apple announced that all submissions to the App Store must include 64-bit support by June 1, 2015. The SoundCloud API for Cocoa contains 32-bit dependencies and will not be updated, because it has been discontinued. Anyone using the SoundCloud API for Cocoa will need to will need to migrate away from it if they wish to update their app after June 1.

To ease this transition we have built a sample app that demonstrates how to authorize a user via OAuth using only built-in Foundation libraries.

Once an access_token has been obtained via OAuth, you can make GET requests like so:

let urlSession = NSURLSession.sharedSession()
let urlString = "https://api.soundcloud.com/me"
let urlComponents = NSURLComponents(string: urlString)!
urlComponents.queryItems = [ NSURLQueryItem(name: "oauth_token", value: "insert an OAuth token here")]
let url = urlComponents.URL!

let dataTask = urlSession.dataTaskWithRequest(NSURLRequest(URL: url)) { (data, response, error) -> Void in
   if let jsonOutput = NSJSONSerialization.JSONObjectWithData(data, options: nil, error: nil) as? [String:AnyObject] {
       // do stuff with JSON
   }
}

dataTask.resume()

POST requests that contain multipart data (e.g. uploading a track) look like this:

import UIKit

class ViewController: UIViewController {
    func uploadTrack(title: String, trackPath: String) {
        let urlSession = NSURLSession.sharedSession()
        let urlRequest = getURLRequest(title, audioPath: trackPath)
        let task = urlSession.dataTaskWithRequest(urlRequest) { (data, response, error) -> Void in
            if let httpResponse = response as? NSHTTPURLResponse {
                println("returned \(httpResponse.statusCode)")
            }
            if data != nil, let response = NSString(data: data, encoding: NSUTF8StringEncoding) {
                println(response)
            }
            if let err = error {
                println(err)
            }
        }
        task.resume()
    }

    func getURLRequest(title: String, audioPath: String) -> NSURLRequest {
        let boundary = NSUUID().UUIDString
        let request = NSMutableURLRequest(URL: NSURL(string: "https://api.soundcloud.com/tracks")!)
        request.HTTPMethod = "POST"
        request.HTTPBody = getPostData("insert an OAuth token here", boundary: boundary, title: title, audioPath: audioPath)
        let contentType = "multipart/form-data; boundary=" + boundary
        request.setValue(contentType, forHTTPHeaderField: "Content-Type")
        return request
    }

    func getPostData(token: String, boundary: String, title: String, audioPath: String) -> NSData {
        let boundaryStart = "--\(boundary)\r\n"
        let boundaryEnd = "\r\n--\(boundary)--\r\n"
        let bodyData : NSMutableData = NSMutableData()

        // add the token
        var tokenSection = boundaryStart
        tokenSection += "Content-Disposition: form-data; name=\"oauth_token\"\r\n\r\n"
        tokenSection += "\(token)\r\n"
        bodyData.appendData(tokenSection.dataUsingEncoding(NSUTF8StringEncoding)!)

        // add the track title
        var titleSection = boundaryStart
        titleSection += "Content-Disposition: form-data; name=\"track[title]\"\r\n\r\n"
        titleSection += "\(title)\r\n"
        bodyData.appendData(titleSection.dataUsingEncoding(NSUTF8StringEncoding)!)

        // add the audio file
        let trackData = NSData(contentsOfFile: audioPath)!
        var trackSection = boundaryStart
        trackSection += "Content-Disposition: form-data; name=\"track[asset_data]\"; "
        trackSection += "filename=\"\(audioPath.lastPathComponent)\"\r\n"
        trackSection += "Content-Type: application/octet-stream\r\n"
        trackSection += "\r\n"
        bodyData.appendData(trackSection.dataUsingEncoding(NSUTF8StringEncoding)!)
        bodyData.appendData(trackData)
        bodyData.appendData(boundaryEnd.dataUsingEncoding(NSUTF8StringEncoding)!)
        return bodyData
    }
}

Note: This example assumes access tokens will never expire, however, we encourage you not to make this assumption in your production code. Instead, build your app assuming that tokens will periodically expire and can be refreshed using a refresh token. For details on how to use a refresh token, see Section 1.5 of the OAuth 2.0 specification.

Linked partitioning to replace offset-based pagination

Mon, 02 Feb 2015 00:00:00 GMT

The SoundCloud API will be dropping support for offset-based pagination on March 2, 2015, in favor of linked partitioning.

To page through a JSON response, pass the linked_partitioning=1 parameter along with your request and it will return a collection, along with a next_href property if there are additional results. To fetch the next page of results, simply follow that URI. If the response does not contain a next_href property, you have reached the end of the results.

You can read more about linked partitioning in the Pagination section of our HTTP API Guide, including code examples in JavaScript, PHP, Python, and Ruby.

The limit parameter continues to be supported with linked partitioning. The default limit is 50 with a maximum value of 200.

Please update your code to replace the offset parameter with linked_partitioning. If you have any questions about this update, please notify us via email.

Prometheus: Monitoring at SoundCloud

Mon, 26 Jan 2015 00:00:00 GMT

In previous blog posts, we discussed how SoundCloud has been moving towards a microservice architecture. Soon we had hundreds of services, with many thousand instances running and changing at the same time. With our existing monitoring set-up, mostly based on StatsD and Graphite, we ran into a number of serious limitations. What we really needed was a system with the following features:

A multi-dimensional data model, so that data can be sliced and diced at will, along dimensions like instance, service, endpoint, and method.
Operational simplicity, so that you can spin up a monitoring server where and when you want, even on your local workstation, without setting up a distributed storage backend or reconfiguring the world.
Scalable data collection and decentralized architecture, so that you can reliably monitor the many instances of your services, and independent teams can set up independent monitoring servers.
Finally, a powerful query language that leverages the data model for meaningful alerting (including easy silencing) and graphing (for dashboards and for ad-hoc exploration).

All of these features existed in various systems. However, we could not identify a system that combined them all until a colleague started an ambitious pet project in 2012 that aimed to do so. Shortly thereafter, we decided to develop it into SoundCloud’s monitoring system: Prometheus was born.

Prometheus is written in Go and has been open source from the beginning. Despite being intentionally quiet about it, we gained a few committed external users and contributors, most notably from Docker and Boxever. Since then, Prometheus has become the standard monitoring solution at SoundCloud. With the level of maturity reached, we would like to tell you more about it.

Our friends from Boxever and Docker are also publishing blog posts simultaneously about their usage of Prometheus:

Architecture

This diagram illustrates the overall architecture of Prometheus and some of its ecosystem components:

Prometheus servers scrape metrics from instrumented jobs, either directly or via an intermediary push gateway for short-lived jobs. They store all scraped samples locally and run rules over this data to either record new timeseries from existing data or generate alerts. PromDash (a web-based dashboard builder) or other API consumers can be used to visualize the collected data.

Prometheus’s main features are:

A multi-dimensional data model.
A flexible query language to leverage this dimensionality.
No reliance on distributed storage; single server nodes are autonomous.
Time series collection happens via a pull model over HTTP.
Pushing time series is supported via an intermediary gateway.
Targets are discovered via service discovery or static configuration.
Multiple modes of graphing and dashboarding support.

See also the Overview section of the documentation.

Data model

Prometheus fundamentally stores all data as time series: streams of timestamped values belonging to the same metric and the same set of labeled dimensions. Timestamps have a millisecond resolution, while values are always 64-bit floats.

The metric name specifies the general feature of a system that is measured. For example, a metric to count the total number of HTTP requests received by an API server might be called api_http_requests_total. Adding labels (key/value pairs) to this metric enables Prometheus’s dimensional data model: any given combination of labels for the same metric name results in a separate time series. One example time series for this metric might represent all the received HTTP requests that used the method GET for querying the endpoint /api/tracks and resulted in a 200 HTTP status code. For naming a time series like that, we use this notation:

api_http_requests_total{method="GET", endpoint="/api/tracks", status="200"}

The query language then allows filtering and aggregation based on these dimensions. Prometheus’s label-based data model is fundamentally the same as the one of OpenTSDB and even has a similar syntax.

Metrics in Prometheus have a type to indicate their meaning and usage. Prometheus currently supports three metric types:

A counter is a cumulative metric that represents a value that only ever goes up, like a request count.
A gauge is a metric that represents a value that can arbitrarily go up and down, like a temperature or a queue length.
A summary is a client-side calculated histogram of observations, usually used to measure request durations or sizes.

For more information on Prometheus’s data model, see the Data model and Metric types sections in the documentation.

Query language

The Prometheus query language allows you to slice and dice the dimensional data for ad-hoc exploration, graphing, and alerting. We’ll talk about the use of the query language for dashboards and alerting in the next two sections. Your first steps with the query language are best done using the expression browser offered by each Prometheus server.

As an example, we have instrumented Bazooka, our internal Heroku-style deployment system, with Prometheus. This provides us with per-instance metrics about memory usage, memory limits, CPU usage, out-of-memory failures, instance restarts, and so on. Each of these metrics is labeled by application, process type, Git revision, environment, and instance ID. This has frequently been useful for debugging both single-instance problems, as well as for doing cluster-wide resource usage analysis. For instance, badly-configured Bazooka applications might set large container memory limits, yet not use a lot of the reserved memory. This screenshot shows a query for the worst RAM wasters across a Bazooka cluster (click the image to view a full-sized version):

Similar queries allow us to identify potentially misconfigured applications and reach out to their developers.

As another related example, we might want to graph the top three CPU users on a Bazooka cluster as measured over the last five minutes, grouped by application and process type:

For intense queries that take a long time to evaluate, you can configure Prometheus to precompute them in the background. Precomputed expressions are called rules. They have a name, which is used like the name of a metric in other expressions.

For more details about the query language, see the documentation.

Dashboards

The one-stop solution for the fanciest dashboards is PromDash, a GUI-based dashboard builder with a SQL backend. It talks to any number of Prometheus servers via an HTTP API and graphs their data in highly configurable dashboards. Even Graphite graphs can be included.

If you want to serve simple dashboards from the Prometheus server directly (without the need of a separate application with SQL backend), have a look at console templates. SoundCloud uses PromDash as its main way for visualizing Prometheus data.

Alerting

Alerts are defined using the same powerful query language described above. A separate binary, the Alertmanager, handles alert notifications and aggregations and enables silencing by any label set. If an alert fires, Alertmanager can send an email or page you through an external alerting service like PagerDuty.

We also offer a Nagios plugin as a bridge from the parts that are already monitored by Prometheus to the existing Nagios alerting set-up that many companies have in place. In fact, SoundCloud uses exactly that configuration for production alerting, which is why the Alertmanager is still less mature than the rest of the Prometheus ecosystem. We plan to move to fully native Prometheus alerting soon.

For more information, see the documentation about configuring alerting rules and alerting best practices.

Storage

One of our goals was to run a Prometheus server with millions of time series and tens of thousands of samples ingested per second on a single server using its local disk. You might think everything should be distributed-something nowadays, but distributed systems demand payment in complexity and operational burdens. How many of you would love to run OpenTSDB but shy away from maintaining a whole Hadoop cluster just for that? With Prometheus, you shard servers by function and run multiple replicas of the same shard for redundancy. Cross-server queries are not possible, but we are working on a hierarchical federation scheme where a higher-level Prometheus server collects selected metrics from subordinated Prometheus servers.

Designing the local time series database for Prometheus was an adventure of its own. We explored and compared many existing local storage implementations. Based on our research, we decided to use LevelDB for indices and our own design for storing the actual sample values and timestamps.

The disk storage of a server is naturally limited (depending on the rate of incoming samples, we usually keep several weeks or months around, without any downsampling), and the disk can fail permanently (resulting in data loss, at least on that one replica). Whoever requires true durability of samples will need some kind of distributed storage. Prometheus already has experimental support for streaming collected samples to OpenTSDB as a permanent storage, turning the Prometheus ecosystem into a collection machinery for OpenTSDB. One day, we would like to support the complete round-trip, i.e. Prometheus queries can read data back from OpenTSDB. Other time series databases might get support too, but the differences in the data model are a challenge.

Getting your service ready for Prometheus

Prometheus-style monitoring relies on instrumenting your own code directly. This is very different from traditional strategies to monitor applications, where the monitored application is often completely ignorant of the monitoring system, which uses plugins or agents to talk to the application in its own language. The high-quality service-based monitoring you get from your handwritten instrumentation is usually worth the price.

To instrument code, we offer client libraries for Go, Java, and Ruby. As the Java client library is usable from any JVM language, this completely covers the needs of SoundCloud. A Python client is also under development. These client libraries allow you to track and expose metrics over HTTP. The default exposition format is protocol buffers, but we also support a very simple text-based format, which can be used for easy “library-less” exposition of metrics even from shell scripts.

To mimic the ways of traditional monitoring approaches, there are a number of exporters for certain types of servers, like the HAProxy Exporter to export metrics of an HAProxy or the Node Exporter to export varius node attributes.

Finally, if you have short-lived jobs like cronjobs that cannot wait for Prometheus to scrape them, there is the Pushgateway, to which you can push metrics. The Pushgateway will then present these metrics permanently to Prometheus for scraping.

Conclusion

Prometheus has proven to be very useful at SoundCloud. It drives our internal monitoring, facilitates firefighting during outages and provides further insight into them for postmortems. We run Prometheus as an open-source project because we believe it can benefit the infrastructure community as a whole. If you are interested in getting involved, check out the Prometheus community.

Special thanks to Matt T. Proud, who initially started the Prometheus project.

SoundCloud in Scalding case study by Concurrent Inc.

Tue, 02 Dec 2014 00:00:00 GMT

Recently we teamed up with Concurrent Inc., the backers of the data-processing framework Cascading, to do a case study of how we use Scalding for some of our data-driven products such as Search. Scalding enables us to iterate quickly, test easily, and it allows for loose coupling of some of our data-processing pipelines.

Check back for future posts about our use of other data-processing tools, and frameworks such as Spark.

XML responses deprecated

Mon, 17 Nov 2014 00:00:00 GMT

The SoundCloud API will be dropping support for Extensible Markup Language (XML) responses. XML will be phased out on the following schedule:

XML is currently the default response format for requests without an explicit format specified in the path (e.g. /tracks) or Accept header. This default will be changed to JSON on December 1, 2014.
Explicit requests for XML — specified either in the path (e.g. /tracks.xml) or an Accept: application/xml header — will continue to be supported until December 15, 2014. After that point, only JSON responses will be supported.

SoundCloud has been using JSON exclusively for internal APIs for several years. Dropping support for XML in our public API will allow us to focus on providing consistent and reliable service.

If your app still uses XML responses, please start working to upgrade it to JSON immediately. If your app does not currently use XML responses, it should be unaffected by this change.

If you are unable to migrate your app from XML to JSON for some reason, we recommend accessing the SoundCloud API through a proxy server that converts JSON to XML.

Please let us know if you have any questions about this update via email.

Building the new SoundCloud iOS application — Part II: Waveform rendering

Mon, 15 Sep 2014 00:00:00 GMT

When we rebuilt our iOS app, the player was the core focus. The interactive waveform was at the center of the design. It was important both that the player be fast and look good.

Initial implementation

We iterated on the waveform view until it was as responsive as possible. The initial implementation focused on replicating the design, which heavily used CoreGraphics. A single custom view calculates the current bar offset based on its time property. It then draws each of the waveform samples that are in the current visible section of the waveform. Each sample is either rendered as a filled rectangle for unplayed samples, or by adding clip paths to the context then drawing a linear CGGradient for the played samples.

To handle progress updates, we used the callback block on AVPlayer:

[player addPeriodicTimeObserverForInterval:CMTimeMake(1, 60)
                                     queue:dispatch_get_main_queue()
                                usingBlock:^(CMTime time) {
    [waveformView setPlaybackTime:CMTimeGetSeconds(time)];
}];

This generates 60 callbacks per second to try and achieve the highest possible frame rate. This solution worked perfectly during testing on the simulator. Unfortunately, testing on an iPhone 4 did not produce similar results:

The preceding trace displays an initial frame rate of 17 FPS that drops to 10 FPS over time. As time progresses, there are more played waveform samples, which are expensive gradient fills and take more CPU time than unplayed samples. The CPU profiler displays that 90% of CPU time is spent in the drawRect: method of our waveform view.

Reducing waveform renders

We needed a new approach that did not require redrawing the entire waveform 60 times a second. The waveform samples never need to change in size, but only in tint color. This makes it possible to render the waveform once, and apply it as an alpha mask to a layer filled with a gradient on the left half, and white on the right half.

This increases the FPS to the maximum of 60 with approximately 50% GPU usage. The CPU usage dropped to approximately 10%.

Using system animations

We can now stop using the AVPlayer timer callback for waveform progress and use a standard UIView animation instead:

[UIView animateWithDuration:CMTimeGetSeconds(player.currentItem.duration)
                      delay:0
                    options:UIViewAnimationOptionCurveLinear
                 animations:^{
                    self.waveformView.progress = 1;
                }
                completion:nil];

This drops CPU usage down to approximately 0%, without noticeably increasing the GPU usage. This would seem to be an obvious win, unfortunately it introduces some visual glitches. Watching the animation on longer tracks shows an amplitude phasing effect on the waveform samples.

Pixel-aligned waveform rendering	Interpolated waveform rendering

While the mask is animated across the waveform view, the animation will not only be drawn at integer pixel positions. When the mask is at a fractional pixel position, it causes the edges of the mask to be interpolated across each of the neighboring pixels. This reduces the alpha of the mask at the edge of each sample. This lower alpha reduces the amplitude of the visible waveform samples. The overall effect is that as the mask is translated across the view, the waveform samples animate between a sharp dark state and a lighter blurry state.

We need a way to tell CoreAnimation to only move the mask to pixel-aligned positions. This is possible by using a keyframe animation that is set to discrete calculation mode, which does not interpolate between each keyframe.

A high-level API was added for keyframe animations in iOS7, but the CoreAnimation interface in this case is a bit clearer:

- (CAAnimation *)keyframeAnimationFrom:(CGFloat)start to:(CGFloat)end
{
    CAKeyframeAnimation *animation =
    	[CAKeyframeAnimation animationWithKeyPath:@"position.x"];

    CGFloat scale = [[UIScreen mainScreen] scale];
    CGFloat increment = copysign(1, end - start) / scale;
    NSUInteger numberOfSteps = ABS((end - start) / increment);

    NSMutableArray *positions =
    	[NSMutableArray arrayWithCapacity:numberOfSteps];
    for (NSUInteger i = 0; i < numberOfSteps; i++) {
        [positions addObject:@(start + i * increment)];
    }

    animation.values = positions;
    animation.calculationMode = kCAAnimationDiscrete;
    animation.removedOnCompletion = YES;
    return animation;
}

Adding this animation to the mask layer produces the same animation, but only at pixel-aligned positions. This has a performance benefit; animating longer tracks produces animations with the FPS reduced to the maximum possible FPS for the number of animation keyframes, which reduces the device utilization accordingly. In this case, the distance of waveform animation is 680 pixels. If the played track is 10 minutes long, the FPS is 680 animation pixels over 600 seconds ≈ 1 pixel position per second. CoreAnimation schedules the animation frames intelligently, so that we only get a 1 FPS animation with correspondingly reduced 1% device utilization.

Increasing up-front cost for cheaper animations

The CPU performance is at an acceptable level, but 50% GPU usage is still high. Most of this is due to the high cost of masking, which requires a texture blend on the GPU. As we fetch a waveforms samples asynchronously for each track, initial render time for the view is less important than performance during playback. Instead of masking the waveform samples, they can be drawn twice: once in the unplayed state, once in the played state with gradient applied. That way, each waveform is contained in separate views that clip their subviews. To adjust the waveform position within these windows, the bounds origin can be adjusted for each subview to slide over its content, similar to how a UIScrollView works.

You can offset the bounds origin of the left clip view by its width. The result is a seamless view that looks exactly the same as the masking effect. Because we are only moving views around in the hierarchy, redrawing does not need to occur and the same keyframe animations can be applied to each of the clip views. After applying these changes, the GPU utilization drops to less than 20% for 60 FPS animation. This was considered fast enough, and is our final iteration of the waveform renderer.

Conclusion

Understanding the performance cost of drawing techniques on iOS can be tricky. The best way to achieve acceptable performance is to start simple, profile often, and iterate until you hit your target. We significantly improved performance by profiling and identifying the bottleneck at each step, and most of the drawing code from our initial naïve implementation survived to the final performant iteration.

We obtained the biggest win by shifting work from the CPU to the GPU. Any CoreGraphics drawing done using drawRect: uses the CPU to fill the layers contents. This is often unavoidable, but if the content seldom changes, the layer contents can be cached and manipulated by CoreAnimation on the GPU. After the drawing reduces to manipulating UIView properties, nearly all of the work can be performed using animations, thereby reducing the amount of CPU view state updates.

It is important to consider GPU usage, but this is harder to understand intuitively. Profiling with Instruments provides helpful insight, especially the OpenGL ES Driver template, which shows animation FPS and the percentage utilization of the GPU. The main tricks here are to reduce blending and masking, ideally using opaque layers wherever possible. The simulator option “Color Blended Layers” can be useful to identify where you have unnecessary overdraw. For more details see the Apple documentation and WWDC videos.

Building the new SoundCloud iOS application — Part I: The reactive paradigm

Mon, 07 Jul 2014 00:00:00 GMT

Recently, SoundCloud launched the new iOS application which was a complete rewrite of the existing iOS application. The Mobile engineering team saw this as an opportunity to build a solid foundation for the future of SoundCloud on iOS and to experiment with new technologies and processes at the same time.

In the world of mobile, you deal with data, errors, threads and concurrency a lot. The common scenario starts with a user tapping on the screen. The application jumps off of the main UI thread, does some I/O related work, some database-related operations along with some transformations to the data. The application then jumps back to the UI thread to render some new information onto the screen.

Both the Android and iOS platforms provide some tools to deal with the different aspects of this scenario yet they are far from ideal. Some of them do not provide much in terms of error handling, which forces you to write boiler-plate code while some of them force you to deal with low-level concurrency primitives. You might have to add some additional libraries to your project so that you do not have to write filtering and sorting predicate code.

We knew early on we wanted to avoid these issues and thus came into the picture the functional reactive paradigm and Reactive Cocoa.

In short, Reactive Cocoa allows you to create composable, event-driven (finite or infinite) streams of data while making use of functional composition to perform transformations on those streams. Erik Meijer is best known in this space with his Reactive Extensions on the .NET platform which also spawned the JVM based implementation RxJava. By adopting this paradigm, we now have a uniform way of dealing with data models and operators that can apply transformations on those data models while taking care of low level concurrency primitives so that one does not have to be concerned about threads or the difficult task of concurrent programming.

Let’s take an example

Like most mobile applications, the SoundCloud iOS application is a typical case of an API client with local storage. It fetches JSON data from the API via HTTP and parses it into API model objects. The persistent store technology we use is Core Data. We decided early on that we wanted to isolate the API from our storage representation so there is a final mapping step involved where we convert API models to Core Data models.

We break this down into smaller units of work: we have

Execute a network request. Parse the JSON response.
Transform JSON objects into API model objects.
Transform API model objects into Core Data models.

1. Executing the network request

For simplicity, assume that the network-access layer implements the following method:

- (RACSignal *)executeRequest:(NSURL *)url;

We do not pass in any delegates or callback blocks, we just give it an NSURL and get back a RACSignal representing a possible asynchronous operation, or future. To obtain the data from that operation, we can subscribe to the signal using subscribeNext:error:completed:

- (RACDisposable *)subscribeNext:(void (^) (id result))nextBlock
                           error:(void (^) (NSError *error))errorBlock
                       completed:(void (^) (void))completedBlock

You might recognize the familiar-looking error and success-callback blocks from other asynchronous APIs. This is where some of Reactive Cocoa’s and FRP’s strengths lie as we shall see later.

2. Parsing the JSON response

After the network request has been made, the JSON response needs to be parsed into an API model representation. For this we use a thin layer around Github’s Mantle library, which wraps parsing in a RACSignal and pushes the result (or error) to the subscriber:

- (RACSignal *)parseResponse:(id)data
{
  return [RACSignal createSignal:^(id<RACSubscriber> subscriber) {
    NSError *error = nil;
    id apiModel = [MTLJSONAdapter modelOfClass:ApiTrack.class
                            fromJSONDictionary:data
                                         error:&error];
    if (error) {
      [subscriber sendError:error];
    } else {
      [subscriber sendNext:apiModel];
      [subscriber sendCompleted];
    }
  }
}

To achieve the composition of operations that we have mentioned earlier, we wrapped the functionality of existing libraries with signals, where appropriate.

3. Persisting the API model with Core Data

In our architecture, the database represents the single source of truth. Therefore to show tracks to the user we first need to store them as Core Data objects. We have a collection of adapter classes that are responsible for mapping API model objects to Core Data model objects. An ApiTrackAdapter might look as follows:

- (RACSignal *)adaptObject:(ApiTrack *)apiTrack
{
  return [[self findOrBuildTrackWithUrn:apiTrack.urn]
                           map:^(CoreDataTrack *coreDataTrack) {
      coreDataTrack.title = apiTrack.title;
      // set other properties
      return coreDataTrack;
  }];
}

Putting it all together

We now have the building blocks to issue a network request, parse the JSON, and store it as a Core Data object. RAC makes it very easy to compose the individual methods functionally by feeding the output of each operation as an input to the next one. The following example uses flattenMap:

-(RACSignal *)loadAndStoreTrack:(NSURL *)url
{
  return [[requestHandler executeRequest:url] flattenMap:^(id json) {
    return [[parser parseResponse:json] flattenMap:^(ApiTrack *track) {
      return [adapter adaptObject:track];
    }];
  }];
}

The flattenMap: method maps or transforms values emitted by a signal and produces a new signal as a result. In this example, the newly created signal returned by loadAndStoreTrack: would either return the adapted Core Data track object or error if any of the operations failed. In addition to flattenMap:, there is a whole range of predefined functional operators like filter:or reduce: that can be applied to signals.

RAC Schedulers

We left out one powerful feature of RAC which is the ability to parametrize concurrency. To ensure that the application stays responsive, we want to perform the network I/O and model parsing in a background queue.

Core Data operations are different, we do not have a choice there. They have to be executed on a predefined private queue, otherwise we risk creating deadlocks in our application.

With the help of RACScheduler we can easily control where the side-effects of a signal are performed by simply calling subscribeOn: on it with a custom scheduler implementation:

-(RACSignal *)loadAndStoreTrack:(NSURL *)url
{
  return [[requestHandler executeRequest:url] flattenMap:^(id json) {
    return [[parser parseResponse:json] flattenMap:^(ApiTrack *track) {
      return [[adapter adaptObject:track]
                       subscribeOn:CoreDataScheduler.instance];
    }];
  }];
}

Here, we use a scheduler that is aware of the current Core Data context to ensure that adaptObject: is executed on the right queue by wrapping everything internally with performBlock:.

If we want to update our UI with the title of the track we just fetched, we could do something like the following:

[[trackService loadAndStoreTrack:trackUrl] subscribeNext:^(Track *track) {
   self.trackView.text = track.title;
} error:^(NSError *error) {
  // handle errors
}];

To ensure that this final update happens on the UI thread we can tell RAC to deliver us the information back on the main thread by using the deliverOn: method:

[[[trackService loadAndStoreTrack:trackUrl]
      deliverOn:RACScheduler.mainThreadScheduler]
  subscribeNext:^(Track *track) {
   self.trackView.text = track.title;
} error:^(NSError *error) {
  // handle errors
}];

By breaking down each operation within this common scenario into isolated units of work, it becomes easier to perform the operations we need on the desired threads by taking advantage of Reactive Cocoa’s scheduling abilities. The functional reactive paradigm has also helped us to compose these independent operations one after another by using operators such as flattenMap. Although adopting FRP and ReactiveCocoa has had its difficulties, we have learned many lessons along the way.

Steep learning curve

Adopting FRP requires a change of perspective, especially for developers who are not used to a functional programming style. Methods do not return values directly, they return intermediate objects (signals) that take callbacks. This can lead to more verbose code, especially when the code is heavily nested which is common when doing more complex things with RAC.

Therefore, it is important to have short and well-named methods, for example a method signature like -(RACSignal *)signal does not communicate anything about the type of values the caller is going to receive.

Another problem is the sheer number of methods or operators defined on a base classes like RACStream / RACSignal. In practice only a few (like flattenMap: or filter:) are used on a regular basis, but the remaining 80% tend to confuse developers who are new to the framework.

Memory management

Memory management can be problematic because of RAC’s heavy use of blocks which can easily lead to retain cycles. They can be avoided by breaking the cycle with weak references to self (@weakify / @strongify macros).

One of RAC’s promises is to reduce the amount of state you need to keep around in your code. This is true but you still need to manage the state introduced by the framework itself, which comes in the form of RACDisposable, an object returned as a result of signal subscription. A common pattern we introduced is to bind the lifetime of the subscription to the lifetime of the object with asScopedDisposable:

self.disposable = [[signal subscribeNext:^{ /* stuff */ }] asScopedDisposable];

Overdoing it

It is easy to fall into the trap of trying to apply FRP to every single problem one encounters (also known as the golden hammer syndrome), thereby unnecessarily complicating the code. Defining clear boundaries and rules between the reactive and non-reactive parts of the code base is important to minimize verbosity and to use the power of FRP And Reactive Cocoa where appropriate.

Performance

There are inherent performance problems within RAC. For example, a simple imperative for loop is guaranteed to execute much faster than flattenMap: which introduces a lot of internal method dispatching, object allocation, and state handling.

In most cases this overhead is not noticeable, especially when I/O latency is involved, as in the preceding examples.

However in situations where performance really matters, such as fast UI rendering, it makes sense to avoid RAC completely.

Debugging

We found this to be a non-issue if your application components are well designed and individually tested. Backtraces tend to get longer but this can be alleviated with some extra tooling like custom LLDB filters. A healthy amount of debug logging across critical components also does not hurt.

Testing

Testing a method that returns a RACSignal is more complicated than testing code that returns plain value objects, but it can be made less painful with a testing library that supports custom matchers. We have created a collection of matchers for expecta that lets us write concise tests. For example:

RACSignal *signal = [subject executeRequest:url];
expect(signal).to.sendSingle(@{ @"track": @{ @"title": @"foo" } });

We found that adopting FRP tends to produce easily testable components because they are generally designed to perform one single task, which is to produce an output given a specific input.

It took a while for the team to get up to speed with FRP and Reactive Cocoa and to learn for which parts of the application it can be used most effectively. Right now it has become an indispensable part of our mobile development efforts, both on Android and iOS. The functional reactive approach has made it easier to build complex functionality out of smaller pieces whilst simplifying concurrency and error handling.

Real-Time Counts with Stitch

Thu, 03 Jul 2014 00:00:00 GMT

Here at SoundCloud, in order to provide counts and a time series of counts in real time, we created something called Stitch.

Stitch was initially developed to provide timelines and counts for our stats pages, which are where users can see which of their tracks are played and when.

Stitch is a wrapper around a Cassandra database. It has a web application that provides read access to the counts through an HTTP API. The counts are written to Cassandra in two distinct ways, and it’s possible to use either one or both of them:

Real Time: For real-time updates, Stitch has a processor application that handles a stream of events coming from a broker and increments the appropriate counts in Cassandra.
Batch: The batch part is a MapReduce job running on [Hadoop] that reads event logs and calculates the overall totals, and then bulk loads this into Cassandra.

Building Products at SoundCloud—Part III: Microservices in Scala and Finagle

Fri, 13 Jun 2014 00:00:00 GMT

In the first two parts of this series, we talked about how SoundCloud started breaking away from a monolithic Ruby on Rails application into a microservices architecture. In this part we will talk a bit more about the platforms and languages in which we tend to write these microservices.

At the same time that we started the process of building systems outside the Mothership (our Rails monolith) we started breaking our large team of engineers into smaller teams that focused on one specific area of our platform.

It was a phase of high experimentation, and instead of defining which languages or runtimes these teams should use, we had the rule of thumb write it in whatever you feel confident enough putting in production and being on-call for.

This led to a Cambrian Explosion of languages, runtimes and skills. We had systems being developed in everything from Perl to Julia, including Haskell, Erlang, and node.js.

While this process proved quite productive in creating new systems, we started having problems when maintaining them. The bus factor for several of our systems was very low, and we eventually decided to consolidate our tools.

Based on the expertise and preferences across teams, and an assessment of the industry and our peers, we decided to stick to the JVM and select JRuby, Clojure, and Scala as our company-wide supported languages for product development. For infrastructure and tooling, we also support Go and Ruby.

Turns out that selecting the runtime and language is just one step in building products in a microservices architecture. Another important aspect an organization has to think about is what stack to use for things like RPC, resilience, and concurrency.

After some research and prototyping, we ended up with three alternatives: a pure Netty implementation, the Netflix stack, and the Finagle stack.

Using pure Netty was tempting at first. The framework is well documented and maintained, and the support for HTTP, our main protocol for RPC, is good. After a while, though, we found ourselves implementing abstractions on top of it to do basic things for the concurrency and resilience requirements of our systems. If such abstractions were to be required, we would rather use something that exists than re-invent the wheel.

We tried the Netflix stack, and a while back Joseph Wilk wrote about our experience with Hystrix and Clojure. Hystrix does very well in the resilience and concurrency requirements, but its API based on the Command pattern was a turnoff. In our experience, Hystrix commands do not compose very well unless you also use RxJava, and although we use this library for several back-end systems and our Android application, we decided that the reactive approach was not the best for all of our use cases.

We then started trying out Finagle, a protocol-agnostic RPC system developed by Twitter and used by many companies our size. Finagle does very well in our three requirements, and its design is based on a familiar and extensible Pipes-and-Filters meets Futures model.

The first issue we found with Finagle is that, as opposed to the other alternatives, it is written in Scala, therefore the language runtime jar file is required even for a Clojure or JRuby application. We decided that this wasn’t too important, though it adds about 5MB to the transitive dependencies’ footprint, the language runtime is very stable and does not change often.

The other big issue was to adapt the framework to our conventions. Twitter uses mostly Thrift for RPC; we use HTTP. They use ZooKeeper for Service Discovery; we use DNS. They use a Java properties-based configuration system; we use environment variables. They have their own telemetry system; we have our own telemetry system (we’re not ready to show it just yet, but stay tuned for some exciting news there). Fortunately, Finagle has some very nice abstractions for these areas, and most of the issues were solved with very minimal changes and there was no need to patch the framework itself.

We then had to deal with the very messy state of Futures in Scala. Heather Miller, from the Scala core team, explained the history and changes introduced by newer versions of the language in a great presentation. But in summary, what we have across the Scala ecosystem are several different implementations of Futures and Promises, with Finagle coupled to Twitter’s Futures. Although Scala allows for compatibility between these implementations, we decided to use Twitter’s everywhere, and invest time in helping the Finagle community move closer to the most recent versions of Scala rather than debug weird issues that this interoperability might spawn.

With these issues addressed, we focused on how best to develop applications using Finagle. Luckly, Finagle’s design philosophy is nicely described by Marius Eriksen, one of its core contributors, in his paper Your Server as a Function. You don’t need to follow these principles in your userland code, but in our experience everything integrates much better if you do. Using a Functional programming language like Scala makes following these principles quite easy, as they map very well to pure functions and combinators.

We have used Finagle for HTTP, Thrift, memcached, Redis, and MySQL. Every request to the SoundCloud platform is very likely hitting at least one of our Finagle-powered microservices, and the performance we have from these is quite amazing.

In the last part of this series of blog posts, we will be talking about how Finagle and Scala are being used to move away from a one-size-fits-all RESTful API to optmized back-ends for our applications.

Building Products at SoundCloud—Part II: Breaking the Monolith

Thu, 12 Jun 2014 00:00:00 GMT

In the previous post, we talked about how we enabled our teams to build microservices in Scala, Clojure, and JRuby without coupling them with our legacy monolithic Rails system. After the architecture changes were made, our teams were free to build their new features and enhancements in a much more flexible environment. An important question remained, though: how do we extract the features from the monolithic Rails application called Mothership?

Splitting a legacy application is never easy, but luckily there are plenty of industry and academic publications to help you out.

The first step in any activity like this is to identify and apply the criteria used to define the units to be extracted. At SoundCloud, we have decided to use the work of Eric Evans and Martin Fowler in what is called a Bounded Context. An obvious example of Bounded Context in our domain was user-to-user messages. This was a well-contained feature set, highly cohesive, and not too coupled with the rest of the domain, as it just needs to hold references to users.

After we identified the Bounded Context, the next task was to find a way to extract it. Unfortunately, Rails’ ActiveRecord framework often leads to a very coupled design. The code dealing with such messages was as follows:

 def index
  if (InboxItem === item)
    respond mailbox_items_in_collection.index.paginate(:page => params[:page])
  else
    respond mailbox_items_in_collection.paginate(
      :joins => "INNER JOIN messages ON #{safe_collection}_items.message_id = messages.id",
      :page  => params[:page],
      :order => 'messages.created_at DESC')
  end
end

Because we wanted to extract the messages’ Bounded Context into a separate microservice, we needed the code above to be more flexible. The first step we took was to refactor this code into what Michael Feathers describes as a seam:

A seam is a place where you can alter behavior in your program without editing in that place.

So we changed our code a little bit:

def index
  conversations = cursor_for do |cursor|
    conversations_service.conversations_for(
    current_user,
    cursor[:offset],
    cursor[:limit])
  end

  respond collection_for(conversations, :conversations)
end

The first version of the conversations_service#conversations_for method was not that different from the previous code; it performed the exact same ActiveRecord calls.

We were ready to extract this logic into a microservice without having to refactor lots of controllers and other Presentation Layer code. We first replaced the implementation of conversations_service#conversations_for with a call to the service:

def conversations_for(user, offset = 0, limit = 50)
  response = @http_client.do_get(service_path(user), pagination(offset, limit))
  parse_response(user, response)
end

We avoided big-bang refactorings as much as we could, and this required us to have the microservices working together with the old Mothership code for as long as it took to completely extract the logic into the new microservice.

As described before, we did not want to use the Mothership’s database as the integration point for microservices. That database is an Application Database, and making it an Integration Database would cause problems because we would have to synchronize any change in the database across many different microservices that would now be coupled to it.

Although using the database as the integration point between systems was not planned, we had the new microservices accessing the Mothership’s database during the transition period.

This brought up two important issues. During the whole transition period, the new microservices could not change the relational model in MySQL—or, even worse, use a different storage engine. For extreme cases, like user-to-user messages where a threaded-based model was replaced by a chat-like one, we had cronjobs keep different databases synchronized.

The other issue was related to the Semantic Events system described in Part I. The way our architecture and infrastructure was designed requires events to be emitted where the state change happened, and this ought to be a single system. Because we could not have both the Mothership and the new microservice emitting events, we had to implement only the read-path endpoints until we were ready to make the full switch from the Mothership to the new microservice. This was less problematic than what we first thought, but nevertheless it did impact product prioritization because features to be delivered were constrained by this strategy.

By applying these principles we were able to extract most services from the Mothership. Currently we have only the most coupled part of our domain there, and products like the new user-to-user messaging system were built completely decoupled from the monolith.

In the next part, we will look at how we use Scala and Finagle to build our microservices.

Building Products at SoundCloud —Part I: Dealing with the Monolith

Wed, 11 Jun 2014 00:00:00 GMT

Most of SoundCloud’s products are written in Scala, Clojure, or JRuby. This wasn’t always the case. Like other start-ups, SoundCloud was created as a single, monolithic Ruby on Rails application running on the MRI, Ruby’s official interpreter, and backed by memcached and MySQL.

We affectionately call this system Mothership. Its architecture was a good solution for a new product used by several hundreds of thousands of artists to share their work, collaborate on tracks, and be discovered by the industry.

The Rails codebase contained both our Public API, used by thousands of third-party applications, and the user-facing web application. With the launch of the Next SoundCloud in 2012, our interface to the world became mostly the Public API —we built all of our client applications on top of the same API partners and developers used.

These days, we have about 12 hours of music and sound uploaded every minute, and hundreds of millions of people use the platform every day. SoundCloud combines the challenges of scaling both a very large social network with a media distribution powerhouse.

To scale our Rails application to this level, we developed, contributed to, and published several components and tools to help run database migrations at scale, be smarter about how Rails accesses databases, process a huge number of messages, and more. In the end we have decided to fundamentally change the way we build products, as we felt we were always patching the system and not resolving the fundamental scalability problem.

The first change was in our architecture. We decided to move towards what is now known as a microservices architecture. In this style, engineers separate domain logic into very small components. These components expose a well-defined API, and implement a Bounded Context —including its persistence layer and any other infrastructure needs.

Big-bang refactoring has bitten us in the past, so the team decided that the best approach to deal with the architecture changes would not be to split the Mothership immediately, but rather to not add anything new to it. All of our new features were built as microservices, and whenever a larger refactoring of a feature in the Mothership was required, we extract the code as part of this effort.

This started out very well, but soon enough we detected a problem. Because so much of our logic was still in the Rails monolith, pretty much all of our microservices had to talk to it somehow.

One option around this problem was to have the microservices accessing directly the Mothership database. This is a very common approach in some corporate settings, but because this database is a Public, but not Published Interface, it usually leads to many problems when we need to change the structure of shared tables.

Instead, we went for the only Published Interface we had, which was the Public API. Our internal microservices would behave exactly like the applications developed by third-party organizations integrate with the SoundCloud platform.

Soon enough, we realized that there was a big problem with this model; as our microservices needed to react to user activity. The push-notifications system, for example, needed to know whenever a track had received a new comment so that it could inform the artist about it. At our scale, polling was not an option. We needed to create a better model.

We were already using AMQP in general and RabbitMQ in specific — In a Rails application you often need a way to dispatch slow jobs to a worker process to avoid hogging the concurrency-weak Ruby interpreter. Sebastian Ohm and Tomás Senart presented the details of how we use AMQP, but over several iterations we developed a model called Semantic Events, where changes in the domain objects result in a message being dispatched to a broker and consumed by whichever microservice finds the message interesting.

This architecture enabled Event Sourcing, which is how many of our microservices deal with shared data, but it did not remove the need to query the Public API —for example, you might need all fans of an artist and their email addresses to notify them about a new track.

While most of the data was available through the Public API, we were constrained by the same rules we enforced on third-party applications. It was not possible, for example, for a microservice to notify users about activity on private tracks as users could only access public information.

We explored several possible solutions to the problem. One of the most popular alternatives was to extract all of the ActiveRecord models from the Mothership into a Ruby gem, effectively making the Rails model classes a Published Interface and a shared component. There were several important issues with this approach, including the overhead of versioning the component across so many microservices, and that it became clear that microservices would be implemented in languages other than Ruby. Therefore, we had to think about a different solution.

In the end, the team decided to use Rails’ features of engines (or plugins, depending on the framework’s version) to create an Internal API that is available only within our private network. To control what could be accessed internally, we used Oauth 2.0 when an application is acting on behalf of a user, with different authorisation scopes depending on which microservice needs the data.

Although we are constantly removing features from the Mothership, having both a push and pull interface to the old system makes sure that we do not couple our new microservices to the old architecture. The microservice architecture has proven itself crucial to developing production-ready features with much shorter feedback cycles. External-facing examples are the visual sounds, and the new stats system.

Roshi: a CRDT system for timestamped events

Fri, 09 May 2014 00:00:00 GMT

Let’s talk about the stream.

The SoundCloud stream represents stuff that’s relevant to you primarily via your social graph, arranged in time order, newest-first. The atom of that data model, an event, is a simple enough thing.

Timestamp
User who did the thing
Identifier of the thing that was done

For example,

At 2014-04-01T13:14:15.034Z,
a-trak
Reposted track skrillex/duck-sauce-nrg-skrillex-kill

If you followed A-Trak, you’d want to see that repost event in your stream. Easy. The difficult thing about time-ordered events is scale, and there are basically two strategies for building a large-scale time-ordered event system.

Data models

Fan out on write means everybody gets an inbox.

That’s how it works today: we use Cassandra, and give each user a row in a column family. When A-Trak reposts Skrillex, we fan-out that event to all of A-Trak’s followers, and make a bunch of inserts. Reads are fast, which is great. But writes carry perverse incentives: the more followers you have, the longer it takes to persist all of your updates. Storage requirements are also quadratic against user growth and follower count (i.e. affiliation density). And mutations, e.g. changes in the social graph, become costly or unfeasible to implement at the data layer. It works, but it’s unwieldy in a lot of dimensions.

At some point, those caveats and restrictions started affecting our ability to iterate on the stream. To keep up with product ideas, we needed to address the infrastructure. And rather than tackling each problem in isolation, we thought about changing the model.

The alternative is fan in on read.

When A-Trak reposts Skrillex, it’s a single append to A-Trak’s outbox. When users view their streams, the system will read the most recent events from the outboxes of everyone they follow, and perform a merge. Writes are fast, storage is minimal, and since streams are generated at read time, they naturally represent the present reality. (It also opens up a lot of possibilities for elegant implementations of product features and experiments.)

Of course, reads are difficult. If you follow thousands of users, making thousands of simultaneous reads, time-sorting, merging, and cutting within a typical request-response deadline isn’t trivial. As far as we know, nobody operating at our scale builds timelines via fan-in-on-read. And we presume that’s due at least in part to the challenges of reads.

Yet we saw potential here. Storage reduction was actually huge: we projected a complete fan-in-on-read data size for all users on the order of a hundred gigabytes. At that size, it’s feasible to keep the data set in memory, distributed among commodity servers. The problem then becomes coördination: how do you reliably and correctly populate that data system (writes), and materialize views from up to thousands of sources by hard deadlines (reads)?

Enter the CRDT

If you’re into so-called AP data systems, you’ve probably run into the term CRDT recently. CRDTs are conflict-free replicated data types: data structures for distributed systems. The tl;dr on CRDTs is that by constraining your operations to only those which are associative, commutative, and idempotent, you sidestep a lot of the complexity in distributed programming. (See: ACID 2.0 and/or CALM theorem.) That, in turn, makes it straightforward to guarantee eventual consistency in the face of failure.

With a bit of thinking, we were able to map a fan-in-on-read stream product to a data model that could be implemented with a specific type of CRDT. We were then able to focus on performance, optimizing our reads without becoming overwhelmed by incidental complexity imposed by the consistency model.

Roshi

The result of our work is Roshi, a distributed storage system for time-series events. It implements what we believe is a novel CRDT set type, closely resembling a LWW-element-set with inline garbage collection. At its core, it uses the Redis ZSET sorted set to store state, and orchestrates self-repairing reads and writes on top, in a stateless operational layer. We spent a long while optimizing the read path to support our latency and QPS requirements, and we’re confident that Roshi will accommodate our exponential growth for years. It took about six developer months to build, and we’re in the process of rolling it out now.

Roshi is fully open-source, and all the gory technical details are in the repository, so please do check it out. I hope it’s easy to grok: at the time of writing, it’s 5000 lines of Go, of which 2300 are tests. And we intend to keep the codebase lean, explicitly not adding features that are outside of the tightly defined problem domain.

Open-sourcing our work naturally serves the immediate goal of providing usable software to the community. We hope that Roshi may be a good fit for problems in your organizations, and we look forward to collaborating with anyone who’s interested in contributing. Open-sourcing also serves another, perhaps more interesting goal, which is advancing a broader discussion about software development. The obvious reaction to Roshi is to ask why we didn’t implement it with an existing, proven data system like Cassandra. But we too often underestimate the costs of doing that: costs like mapping your domain to the generic language of the system, learning the subtleties of the implementation, operating it at scale, and dealing with bugs that your likely novel use cases may reveal. There are even second-degree costs: when software engineering is reduced to plumbing together generic systems, software engineers lose their sense of ownership, which is the foundation of craftsmanship and software quality.

Given a well-defined problem, a specific solution may be far less costly than a generic version: there’s a smaller domain translation, a much smaller surface area, and less operational friction. We hope that Roshi stands in evidence for the case that the practice of software engineering can be a more thoughtful and crafted process. Software that is “invented here” can, in the right circumstances, deliver outstanding business value.

Roshi was a team effort. I’m deeply indebted to the amazing work of Tomás Senart, Björn Rabenstein, and Johan Uhle, without whom Roshi would have never been possible.

Introducing JavaScript SDK version 2

Thu, 01 May 2014 00:00:00 GMT

SoundCloud is pleased to introduce a new major version of the SoundCloud JavaScript SDK. In version 2, we’ve rewritten much of the internal code, resulting in better performance for your JavaScript applications and support for more streaming standards, such as HTTP Live Streaming.

You can test the new version by pointing your JavaScript applications to https://connect.soundcloud.com/sdk-2.0.0.js.

We’ve also created a guide to help you upgrade from version 1 to version 2.

JavaScript SDK version 1 is now deprecated and will be permanently replaced by version 2 on July 1, 2014.

On June 17, 2014, we will temporarily replace version 1 with version 2 between 10:00 and 11:00 UTC. We will do this again on June 24, 2014, between 18:00 and 19:00 UTC. These two upgrade tests will give you an opportunity to understand the impact of this change on your applications. To ensure a seamless transition for your users, we strongly encourage you to upgrade and perform internal tests in advance of these dates.

To receive notices before, during, and after these tests, follow @SoundCloudDev on Twitter.

If you have any questions about this upgrade, please feel free to email api-support@soundcloud.com.

Irrational Fun: Find Yourself at Berlin Buzzwords

Sun, 27 Apr 2014 00:00:00 GMT

We were counting down the days until Berlin Buzzwords on May 25, when we realised that it would be great if you came too! With that in mind, we’ve created a contest. One lucky winner will receive a free ticket to Berlin Buzzwords, including travel expenses and accommodation. Here are the details about how to apply.

The ratio of a circle’s circumference to its diameter, represented by the Greek letter π, is an irrational number—it never terminates or repeats. Your goal is to find the SoundCloud logo in π.

We have provided a 14 pixel by 6 pixel, greyscale reference image:

Here is the same image at 60X magnification:

Each of the 10 shades of grey in this image can be mapped to a number:

Color	RGB Hex	Number
	`ffffff`	0
	`f0f0f0`	1
	`ebebeb`	2
	`d0d0d0`	3
	`c1c1c1`	4
	`a8a8a8`	5
	`878787`	6
	`535353`	7
	`333333`	8
	`000000`	9

Applying this mapping to the reference image produces the following 84-digit bitmap:

0 0 0 0 0 3 3 9 9 9 6 0 0 0
0 0 0 3 4 9 5 9 9 9 9 4 0 0
0 1 2 7 5 9 5 9 9 9 9 7 1 0
3 9 5 9 5 9 5 9 9 9 9 9 9 3
6 9 5 9 5 9 5 9 9 9 9 9 9 6
1 8 5 9 5 9 5 9 9 9 9 9 8 1

Your challenge is to write an algorithm that finds the 10 sequences that most closely approximate the reference image. Each result should include the sequence and its position (after the decimal point) in π.

Here is an example result set:

Rank	Sequence	Offset
1	`082201638940102393659252475011295776958920 282336494898427768768699465405437965994582`	297,640,119
2	`310015049916890341096198545241549627773525 291969856827158758552799587406476458977970`	792,987,187
3	`212021479960265330123798820231693768599316 147634474729776147987653958935291919768971`	972,165,010
4	`000053743536020032496985553181128909983810 344939134894807349584729687746183109884672`	981,165,566
5	`142204297650171312983445842322141909755200 787408739757838589593329762648444919386594`	789,652,974
6	`300011495970664010077917573663456957498854 662995598898697947677549686339433357728071`	197,342,990
7	`313560984264300011495970664010077917573663 456957498854662995598898697947677549686339`	197,342,978
8	`870402479996214234001557832923050859979903 649788689695954439755933903629798966788984`	75,975,342
9	`208503759370245135490877462200175839969750 453766432680245845143285985661373828688970`	343,577,393
10	`300544295771128716836756973814286978997516 282647269986574856578306678421894769876141`	950,462,734

Entries will be judged against the follow criteria:

Code quality
Runtime performance
Visual closeness to the reference image (subjective)

You should run your algorithm against the following data set (approximately 1 GB), which contains the first 1,000,000,000 (billion) digits of π.

Please send your submission, including a link to the source-code repository, to e@soundcloud.com on or before May 5, 2014 23:59 UTC. Your repository should contain a README that includes instructions about how to set up and run your code. Entries are subject to the terms and conditions.

UPDATE

Congratulations to Tomasz Pewiński, who submitted the winning entry.

We’d also like to acknowledge entries by Dan Oved and Martin Kühl, which were also excellent. Thanks to everyone else who participated in the contest. We hope it was fun!

Welcome to SoundCloud's redesigned developer site

Fri, 11 Apr 2014 00:00:00 GMT

We’ve taken some time to bring all our developer resources together into a single site. In doing so, we’ve reorganized the layout to make things easier to find and also given the site a fresh new look.

We hope you like it!

If you have any feedback about the new design, follow @SoundCloudDev on Twitter and let us know.

Security update: Heartbleed vulnerability in OpenSSL

Fri, 11 Apr 2014 00:00:00 GMT

On Monday, April 7th, 2014, a major security vulnerability in OpenSSL was made public. The vulnerability was filed as CVE-2014-0160 and later dubbed “Heartbleed”, because the bug lies within OpenSSL’s heartbeat extension, which is used for keepalive monitoring. As a result of the bug, process memory can be read out remotely by an attacker—potentially including certificates, keys, credentials, tokens, or other sensitive data processed by the server.

OpenSSL works as a cryptographic library that allows for authenticity and confidentiality across the entire Internet. Because the reported Heartbleed bug affects a vast number of internet services using OpenSSL to secure their services (such as HTTPS, SMTP, IMAPS, and POP3), a patched OpenSSL version was released by the maintainers within hours. Linux and UNIX distributions, which depend on the OpenSSL implementation, received patches by their respective upstream maintainers.

SoundCloud too uses OpenSSL in many of our services to increase the security and privacy of our users. We therefore moved quickly to patch the vulnerability, and did so within hours of the patch being made available. We’ve also been in close communications with our vendors and service providers, to ensure that they have applied the appropriate fixes as well. We have confirmed that our implementations of OpenSSL are no longer vulnerable to this bug.

Because we consider our users’ security and privacy of the utmost importance, we have further taken the precautionary measures to rotate SSL certificates and keys, and expire authentication tokens, such as session cookies, remember tokens, and OAuth access and refresh tokens. This means that users will be signed out of their SoundCloud accounts. Along with top security researchers and responsible companies, we have also recommended to our users that they change their passwords on all accounts (not just SoundCloud) that they have signed-in with in the past week. Developers of API clients that check our SSL fingerprints will need to update them.

While the Heartbleed bug marks a sad day for the Internet as a whole, with SoundCloud’s rollout of Perfect Forward Secrecy (PFS) support last year, we ensured that the impact of an attack with the purpose of stealing private keys and reading previously encrypted traffic is minimized. In the same spirit we will also strive to find more such opportunities in the future and preemptively provide our users with the highest possible level of safety.

For more details about this bug, go to heartbleed.com. To use tools to check your services, go to Filippo.io or FiloSottile/Heartbleed on GitHub.

Sponsoring CocoaPods

Sat, 08 Mar 2014 00:00:00 GMT

I’m excited to announce that SoundCloud is sponsoring the development of CocoaPods through a Travis Foundation grant. CocoaPods is an open-source dependency manager for Objective-C projects. Travis Foundation is a non-profit that pairs corporate sponsors with open-source projects to make open source even better.

At SoundCloud, our iOS team uses CocoaPods every day to manage the dependencies of our mobile apps. We hope that this sponsorship will lead to improvements that benefit the entire Mac and iOS developer ecosystem.

Watch for updates in the coming weeks on the Travis Foundation blog and CocoaPods blog.

Smooth image loading by upscaling

Thu, 20 Feb 2014 00:00:00 GMT

The site soundcloud.com is a single-page application that displays a multitude of users’ images. At SoundCloud, we use a technique to make the loading of an image appear smooth and fast. When displaying an image on screen, we want it to display to the user as fast as possible. The images display in multiple locations from a tiny avatar on a waveform to a large profile image. For this reason, we create each image in several sizes. If you are using Gravatar, this technique also applies because you can fetch arbitrarily-sized images by passing the desired size in a query parameter (?s=).

The technique uses the browser’s cache of previously loaded images. When displaying a large avatar image, first display a smaller version that is stretched out to full size. When the larger image has loaded, it fades in over the top of the smaller version.

The HTML looks like this:

![](small.jpg)
![](large.jpg)

The CSS looks like this:

.fullImage {
  transition: opacity 0.2s linear;
}

For the sake of brevity, the positioning code is not included in the preceding snippet. However, the images should lie atop one another.

Finally, the JavaScript code looks like this:

var fullImage   = $('.fullImage'),
    placeholder = $('.placeholder');

fullImage
  .css('opacity', 0)
  .on('load', function () {
    this.style.opacity = 1;
    setTimeout(placeholder.remove.bind(placeholder), 500);
  });

Thus far, it’s not too complicated, and it gives a nice effect to the loading of images.

But there’s a problem: we don’t want to make a request to get the small image just to display it for a few milliseconds. The overhead of making HTTP requests means that loading the larger image will usually not take significantly longer than the small one. Therefore, it only makes sense to use this technique if a smaller image has already been loaded during a particular session and thus served from the browser’s cache. How do we know what images are in cache? Each time an avatar is loaded, we need to keep track of that. However over time, there could be many thousands of avatars loaded within one session, so it needs to be memory efficient. Instead of tracking the full URLs of loaded images, we extract the minimum amount of information to identify a image, and use a bitmask to store what sizes have been loaded:

// a simple map object, { identifier => loaded sizes }
var loadedImages = {},

  // Let's assume a basic url structure like this:
  // "http://somesite.com/{identifier}-{size}.jpg"
  imageRegex = /\/(\w+)-(\w+)\.jpg$/,

  // a list of the available sizes.
  // format is [pixel size, filename representation]
  sizes = [
    [ 20, "tiny"  ],
    [ 40, "small" ],
    [100, "medium"],
    [200, "large" ]
  ];

// extract the identifier and size.
function storeInfo(url) {
  var parts = imageRegex.exec(url),
      id    = parts[1]
      size  = parts[2],
      index;

  // find the index which contains this size
  sizes.some(function (info, index) {
    if (info[1] === size) {
      loadedImages[id] |= 1 << index;
      return true;
    }
  });
}

// once the image has loaded, then store it into the map
$('.fullImage').load(function () {
  storeInfo(this.src);
});

When the image loads, we extract the important parts from the URL: namely, the identifier and the size modifier. Each size is mapped to a number—its index in the sizes array—and the appropriate bit is turned on in the loadedImages map. The code on line 27 does this conversion and bit manipulation; 1 << index is essentially the same as Math.pow(2, index). By storing only a single number in the object, we save quite a bit of memory. A single-number object could contain many different flags. For example, assume there are four different sizes and 10,000 images in the following map:

asBools = {
  a: [true, true, false, true],
  b: [false, true, false, false],
  // etc...
};

asInts = {
  a: 11,  // 2^0 + 2^1 + 2^3 = 1 + 2 + 8
  b: 2,   // 2^1
  // etc...
}

The memory footprints of these objects differ by 30%: 1,372,432 bytes for the booleans, and 1,052,384 for the integers. The key names consume the largest portion of these objects’ memory. For this reason, it is important to compress the key names as much as possible. Numeric keys are stored particularly efficiently by the V8 JavaScript engine found in Chrome and Safari.

We now have a map that shows us what images have been loaded during this session, and you can use that information to choose a placeholder:

// find the largest image smaller than the requested one
function getPlaceholder(fullUrl) {
  var parts = imageRegex.exec(fullUrl),
      id = parts[1],
      targetSize = parts[2],
      targetIndex;

  sizes.some(function (info, index) {
    if (info[1] < targetSize) {
      targetIndex = index;
      return true;
    }
  });

  while (targetIndex >= 0) {
    if (loadedImages[id] & 1 << targetIndex) {
      return fullUrl.replace(
        /\w+\.jpg$/,
        sizes[targetIndex][1] + '.jpg'
      );
    }
    --targetIndex;
  }
}

// and in usage:
var placeholderUrl = getPlaceholder(fullSizeUrl);

if (placeholderUrl) {
  // there has been a smaller image loaded previously, so...
  addTheFadeInBehaviour();
} else {
  // no smaller image has been loaded so...
  loadFullSizeAsNormal();
}

Although, this technique is a bit involved and I’ve deliberately glossed over some of the details, it creates a nice visual effect, greatly reduces the perceived load time of the image, and it is especially effective for long-lived, single-page applications.

Hassle-free concurrency in Android using RxJava

Wed, 23 Oct 2013 00:00:00 GMT

Both our Android and iOS teams use the reactive programming paradigm to simplify asynchronous, concurrent code in our native mobile apps. For Android, we use Netflix’s RxJava. Matthias Käppler—a SoundCloud engineer and a contributor to the RxJava Android library—blogs about the HOWs and WHYs of RxJava on Android.

Tomorrow in London, Matthias will be talking about RxJava at Droidcon. You can grab a drink with him and other members of our Android Team at the SoundCloud Droidcon Drinkup.

Say hello to Sketchy the spam fighter

Mon, 07 Oct 2013 00:00:00 GMT

Sketchy is a spam-fighting, open-source software framework developed by SoundCloud engineers Matt Weiden, Rany Keddo, and Michael Brückner. Sketchy reduces malicious user activity on web applications. You can use it to address several common issues:

Detect when a user submits text that contains spam content.
Detect when a user submits multiple texts that are nearly identical.
Rate-limit malicious actions that users perform repeatedly.
Check user signatures such as IP addresses against external blacklist APIs.
Collect and consolidate reports of spam from users.

We created Sketchy so that we could fight spam around the clock. Any startup that reaches the scale of SoundCloud has to deal with spammers on a daily basis. We decided to release Sketchy as open-source software so other companies don’t have to reinvent the wheel. Since we’ve released it, Sketchy has been featured on GigaOM.

Sketchy has been deprecated and is no longer in use. It worked well for many years but has since been replaced by other services.

Building Clojure Services at Scale

Fri, 20 Sep 2013 00:00:00 GMT

SoundCloud has a service-oriented architecture, which allows us to use different languages for different services. With concurrency and scaling in mind, we started to build some services in Clojure due to its interoperability with the JVM, the availability of good quality libraries, and we just plain like it as a language.

How do you build distributed, robust, and scalable micro-services in Clojure? Read what Joseph Wilk, an engineer and Clojure enthusiast at SoundCloud, has to say.

Writing your own Karma adapter

Mon, 09 Sep 2013 00:00:00 GMT

Background

When we started to work on the new version of our mobile web app, we knew we wanted to run unit tests on a wide variety of clients, mobile devices, PhantomJS, and on Chrome when running locally. Because we practice continuous integration, we knew we also wanted Git hooks and proper results formatting.

We chose Karma runner, which is a project from the Angular JS team that provides developers with a “productive testing environment”. One of the advantages that Karma runner offers over other similar projects is its ability to use any testing framework. At SoundCloud, we aim to have the same toolset across various JavaScript projects, and our unit test framework of choice is Tyrtle.

Using Tyrtle

You can write your own Karma adapter by using the Tyrtle example that follows. The idea is to tie your tests to the Karma API. The pieces of information that you need are the number of tests, test suites or modules, the results of each test (with possible assertion or execution errors, or both), and a hook to let Karma know that the runner ran all of the tests.

You also need to provide a start function that configures the unit test framework, loads the test files, and starts the tests.

The basic template for an adapter is as follows:

(function (win) {
/**
 * Returned function is used to kick off tests
 */
function createStartFn(karma) {
  return function () {
  };
}

/**
 * Returned function is used for logging by Karma
 */
function createDumpFn(karma, serialize) {
  // inside you could use a custom `serialize` function
  // to modify or attach messages or hook into logging
  return function () {
    karma.info({ dump: [].slice.call(arguments) });
  };
}

win.__karma__.start = createStartFn(window.__karma__);
win.dump = createDumpFn(win.__karma__, function (value) {
  return value;
});
})(window);

Next, create a renderer/reporter for the unit test framework that will pass the data to Karma. Tyrtle has a renderer that can render HTML, XML for CI, or print to any other type of output.

To pass the data to Karma, implement the methods that follow:

/**
 * Tyrtle renderer
 * @interface
 */
function Renderer () {}
Renderer.prototype.beforeRun  = function (tyrtle) {};
Renderer.prototype.afterRun   = function (tyrtle) {};
Renderer.prototype.afterTest  = function (test, module) {};

The createStartFn function creates a renderer object, with a Karma runner instance available within the start-function’s scope.

Create a parameter named karma:

function TyrtleKarmaRenderer (karma) {
  this.karma = karma;
}

Tell karma what the total number of tests is:

/**
 * Invoked before all tests are run; reports complete number of tests
 * @param  {Object} tyrtle Instance of Tyrtle unit tests runner
 */
TyrtleKarmaRenderer.prototype.beforeRun = function (tyrtle) {
  this.karma.info({
    // count number of tests in each of the modules
    total: tyrtle.modules.reduce(function(memo, currentModule) {
      return memo + currentModule.tests.length;
    }, 0)
  });
};

After each test, pass the resulting data to Karma:

/**
 * Invoked after each test, used to provide Karma with feedback
 * for each of the tests
 * @param  {Object} test current test object
 * @param  {Object} module instance of Tyrtle module
 *                  to which this test belongs
 */
TyrtleKarmaRenderer.prototype.afterTest = function (test, module) {
  this.karma.result({
    description: test.name,
    suite: [module.name + "#"] || [],
    success: test.status === Tyrtle.PASS,
    log: [test.statusMessage] || [],
    time: test.runTime
  });
};

Next, inform Karma that the tests have all finished running:

/**
 * Invoked after all the tests are finished running
 * with unit tests runner as a first parameter.
 * `window.__coverage__` is provided by Karma.
 * This function notifies Karma that the unit tests runner is done.
 */
TyrtleKarmaRenderer.prototype.afterRun = function (/* tyrtle */) {
  this.karma.complete({
    coverage: window.__coverage__
  });
};

You now have a renderer constructor. Next, turn your attention to the createStartFn function. You need to configure and initialize the unit test framework that returns a function, which requires a list of test files that are served from the Karma server and starts the actual runner.

Karma serves the files that are required for testing from a path that Karma creates and timestamps the files to avoid caching issues in browsers. Karma makes each path available as a key in a hash named __karma__.files. This makes Karma a bit tricky to configure when using an AMD-loader such as require.js. To understand how to use AMD with Karma, go to:

http://karma-runner.github.io/0.8/plus/RequireJS.html

Here is the final createStartFn function:

/**
 * Creates instance of Tyrtle to run the tests.
 *
 * Returned start function is invoked by Karma runner when Karma is
 * ready (connected with a browser and loaded all the required files)
 *
 * When invoked, the start function will AMD require the list of test
 * files (saved by Karma in window.__karma__.files) and set them
 * as test modules for Tyrtle and then invoke Tyrtle runner to kick
 * off tests.
 *
 * @param  {Object} karma Karma runner instance
 * @return {Function}     start function
 */
function createStartFn(karma) {
  var runner = new Tyrtle({});

  Tyrtle.setRenderer(new TyrtleKarmaRenderer(karma));
  return function () {
    var testFiles = Object.keys(window.__karma__.files)
      .filter(function (file) {
        return (/-test\.js$/).test(file);
      })
      .map(function (testFile) {
        return testFile.replace('/base/public/', '').replace('.js', '');
      });
    require(testFiles, function (testModules) {
      // test files can return a single module, or an array of them.
      testFiles.forEach(function (testFile) {
        var testModule = require(testFile);
        if (!Array.isArray(testModule)) {
          testModule = [testModule];
        }
        testModule.forEach(function (aModule, index) {
          aModule.setAMDName(testFile, index);
          runner.module(aModule);
        });
      });
      runner.run();
    });
  };
}

To find more examples of how this all fits together, see the scripts test-main.js (the RequireJS configuration to work with Karma) and karma.conf.js. Also, there are many adapter implementations such as Mocha, NodeUnit, and QUnit on the Karma GitHub page.

Ursula Kallio contributed to the writing of this post.

Responsive Android applications with sane code

Mon, 12 Aug 2013 00:00:00 GMT

A common problem in Android development is that you need to jump off the main UI thread to retrieve data from an IO-based source. At SoundCloud, we use Netflix’s RxJava to simplify asynchronous flows in Android. In an interview given for the Google Developers series “root access Berlin”, Mustafa Sezgin—who heads up our Mobile Team—explains why we chose RxJava, what it gives us, and he walks through an example-usage pattern:

If you are interested in more details, be sure to catch Matthias Käppler—from our Android Team—give an in-depth talk at Droidcon London, October 24-27, 2013.

Win a trip to the Barcelona Ruby Conference

Fri, 09 Aug 2013 00:00:00 GMT

The lineup for BaRuCo 2013 looks amazing, with speakers such as Aaron Patterson, Katrina Owen, Sandi Metz, and Ruby’s inventor Yukihiro Matsumoto. The conference is currently SOLD OUT, but we have one extra ticket… and it could be yours!

If you win the ticket, SoundCloud will fly you from anywhere in the world to Barcelona, Spain and put you up in a nice Catalonian hotel.

How do you enter to win?

It’s simple. Just create a command-line interface in Ruby that uses the SoundCloud API. You can use the SoundCloud Ruby SDK, but this is not a requirement. The only constraints are that it is:

a command-line interface,
written in Ruby,
integrated with SoundCloud,
and totally awesome!

The author of the most creative CLI that uses SoundCloud will win the BaRuCo ticket.

To submit an entry, push your code to GitHub and send an email that includes a link to the repo to e@soundcloud.com by 12:59:59 UTC on August 26th, 2013. We will announce the winner on August 30th.

You must be at least 18 years old to enter and allowed to travel to Spain (sorry Edward Snowden).

Entries will be judged by a team of SoundCloud staff. If you have any questions, email me.

Music Hack Day Coming to Toronto!

Tue, 09 Jul 2013 00:00:00 GMT

The first Music Hack Day — organized by our very own Dave Haynes — was held in July 2009 in London, UK. Since then there have been over 30 Music Hack Day events all over the world in cities like San Francisco, New York, Paris, Barcelona and Reykjavik.

Music Hack Day events gather programmers, designers and artists to conceptualize, build and demo the future of music. Software, hardware, mobile, web, instruments, art — anything goes as long as it’s music related.

I’m happy to announce that SoundCloud is teaming up with Rdio, Unspace and The Echo Nest to organize the first ever Music Hack Day in Toronto. MHD Toronto happens on August 10th and 11th 2013.

If you are a Toronto developer, designer, or artist, come out and join us for what is sure to be an amazing event.

For more details and to register for the event, go to http://toronto.musichackday.org.

Berlin Geekettes All-Women Hackathon Roundup

Thu, 25 Apr 2013 00:00:00 GMT

Late last year, six women crowded in to a Mitte cafe booth and listened to Berlin Geekettes founder Jess Erickson share her idea: Berlin’s first all-women hackathon. With SoundCloud’s Amelie Anglade the then newly-appointed the Berlin Geekettes Tech Ambassador, we agreed that it was a great idea to produce the hackathon as a partnership between the Geekettes and the women developers of SoundCloud.

Fast forward to the first weekend of March, when the vision became reality: after 24 hours of hacking, 80 women demoed 29 projects across a range of different platforms, from a belt transformed into a game controller to an app aimed to help toddlers learn to do everyday tasks.

Hackathon with a twist

This hackathon invited all types of projects: web, iOS, Android, hardware, and so on. And the hackers rose to the challenge. One women even turned her belt into a game controller, and demo’d it by standing on a chair and playing Tetris by tapping her fingers on what she dubbed The WonderBelt.

Achievement of a strong collaborative energy was an important part of what we set out to do. With over 120 applications, we accepted as many women as we could possibly fit into the generous space provided free of charge by Deutsche Telekom, complete with internet infrastructure and an onsite technician. In addition to API representatives, we asked attendees to identify themselves as ”domain experts” if they were willing to offer help and answer questions. We wanted to make the event also appealing to new programmers, as well as non-programmers such as designers and people with product ideas. We partnered with OpenTech School to host a prep-school training session beforehand, with a presentation by SoundCloud’s very own Duana Stanley. The result? A wide range of women with different backgrounds and experience levels working together.

And the Berlin Geekettes wanted to create a fun, healthy atmosphere, which is why we had a yoga teacher leading relaxation exercise classes and fresh organic meals (shout out to the amazing chefs!). Etsy joined the fun by providing a craft corner with instructions and materials for making plush birds. Many hackers took a break for a mental refresh, with stretching and stitching.

Lots of actually working demos!

We got the usual API mash-ups, such as HomeThingy which suggests apartments on ImmobilienScout24 based on your Etsy purchase history, Eye-MOO which sends your EyeEM photos to be printed as postcards, and Mood Swings which serves music from SoundCloud and images from Tumblr to suit your mood.

There were not one but three mash-ups using the SoundCloud and Readmill APIs, combining books and sounds in various ways: Soundmill (share highlights from your SoundCloud audiobooks!), BOOKBEAT (which also one the Best Student Project prize), and our staff pick GetUp-ReadUp (a noisy alarm clock that uses book trivia to make you feel alert).

More than one team built apps aimed at small children, including the overall winner: Monkey See, Monkey Do helps your toddler understand daily routines using simple pictures. The team included an illustrator who made adorable drawings of a monkey brushing its teeth, having a snack, and other toddler activities.

More, please

Dive in to the whole list of projects list of submitted projects on the hackathon’s official wiki. We were blown away by the quality of the presentations from so many first-time hackathon-attendees!

Removing 'hotness' parameterhotness

Tue, 16 Apr 2013 00:00:00 GMT

The /tracks endpoint has traditionally accepted an order parameter for ordering results by either creation date or ‘hotness’. The method for calculating a tracks ‘hotness’ has never been clearly explained, but generally speaking is based on the number of likes and listens a track receives.

Recently we started to experience problems with the query that returns tracks ordered by hotness. In the past weeks, these problems started to effect and even cause outages for API users.

We have decided that the best way forward is to remove this parameter. Starting soon, GET requests to the /tracks endpoint will ignore the order parameter and default to ordering by creation date.

In the future, we look forward to releasing support for more stable and idiomatic search and order parameters. In the meantime, it is still possible to approximate the result sets previously returned by specifying order=hotness by manually sorting the returned tracks by a combination of favoritings\_count and playback\_count.

SoundCloud at PennApps

Wed, 23 Jan 2013 00:00:00 GMT

Last weekend, we sponsored and attended our first PennApps, the world’s largest student run hackathon held at the University of Pennsylvania in Philadelphia. Nearly 500 students participated from a variety of universities across the US and elsewhere. Students were given 36 hours to get into teams, hack on projects, then show them off to the judges. The results were astounding. In total, over 100 hacks were submitted. Here are some of my personal favourites.

Cloud Sequencer

The winners of the Best Audio Hack Prize, Cloud Sequencer is an app that allows you to hook up a midi keyboard to your browser, and map portions of SoundCloud tracks to the keys. The level of polish on this app was amazing and I’m happy to note that the team is working on making a public version available that will work with a midi keyboard or your computer’s keyboard. Here’s the team, Noah Klugman, Max Seiden, Gaurav Kulkarni and Yunxing Dai demoing their project.

SoundScultpr

This was fairly mind blowing. Four High School students, Jared Zoneraich, Kenny Song, Russell Kaplan, David Maginley built SoundSculptr. SoundSculptr uses a Leap Motion to enable users to create music by moving their hands. It’s a fantastic idea and they pulled it off brilliantly. You can even upload the result to SoundCloud!

Astoria

Ever checked out a neighbourhood using Google maps Street View? Astoria takes this to a new level by allowing you to use your phone to “steer” your way through a city Grand Theft Auto style, pulling in venues from Yelp as you pass them, showing property values, and letting you listen to local sounds on SoundCloud. Get the full neighbourhood experience with this app. Astoria was built by Joshua Ma, Anisha Pai, Max Kolysh and Thomas Georgiou. Here’s team member Anvisha Pai giving it a go.

LiveTale.us

I love apps that explore new ways of telling stories. LiveTale, built by Nathan Schloss, Yair Aviner, Ben Glickman and Tyler Cloutier lets users record snippets of a story or listen to previously recorded story tales plotted on a google map. Bonus points for deploying it live, so you can check it out.

Instagramaphone

This one made me laugh. Built by Dina Lamdany, Christopher Yan, Daria Jung and Nate Brennand, Instagramaphone allows you to record audio and then apply “Filters” before sharing it with the world. Such a brilliant concept and really well executed. Check it out and start playing with filters now. Oh, and they had a banana on their team. I applaud this.

I could go on, there were so many amazing hacks. Special mention to The Musical Toilet which lets users control SoundCloud with their, ahem, stream.

Architecture behind our new Search and Explore experience

Tue, 04 Dec 2012 00:00:00 GMT

Search is front-and-center in the new SoundCloud, key to the consumer experience. We’ve made the search box one of the first things you see, and beefed it up with suggestions that allow you to jump directly to people, sounds, groups, and sets of interest. We’ve also added a brand-new Explore section that guides you through the huge and dynamic landscape of sounds on SoundCloud. We’ve also completely overhauled our search infrastructure, which helps us provide more relevant results, scale with ease, and experiment quickly with new features and models.

In the beginning

In SoundCloud’s startup days, when we were growing our product at super speed, we spent just two days implementing a straightforward search feature, integrating it directly into our main app. We used Apache Solr, as was the fashion at the time, and built a master-slave cluster with the common semantics: writes to the master, reads from the slaves. Besides a separate component for processing deletes, our indexing logic was pull-based: the master Solr instance used a Solr DataImportHandler to poll our database for changes, and the slaves polled the master.

At first, this worked well. But as time went on, our data grew, our entities became more complex, and our business rules multiplied. We began to see problems.

The problems

The index was only getting updated on the read slaves about every fifteen minutes. Being real-time is crucial in the world of sound, as it’s important that our users’ sounds are searchable immediately after upload.
A complete re-index of the master node from the database took up to 24 hours, which made making a simple schema change for a new feature or bugfix a Sisyphean task.
Worse, during those complete-reindex periods, we couldn’t do incremental indexing because of limitations in the DataImportHandler. That meant the search index was essentially frozen in time: not a great user experience.
Because the search infrastructure was tightly coupled in the main application, low-level Solr or Lucene knowledge leaked everywhere. If we wanted to make a slight change, such as to a filter, we had to touch 25 parts of the codebase in the main application.
Because the application directly interacted with the master, if that single-point-of-failure failed, we lost information about updates, and our only reliable recovery was a complete reindex, which took 24 hours.
Since Solr uses segment or bulk replication, our network was flooded when the slaves pulled updates from the master, which caused additional operational issues.
Solr felt like it was built in a different age, for a different use-case. When we had performance problems, it was opaque and difficult to diagnose.

In early 2012, we knew we couldn’t go forward with many new features and enhancements on the existing infrastructure. We formed a new team and decided we should rebuild the infrastructure from scratch, learning from past lessons. In that sense, the Solr setup was an asset: it could continue to serve site traffic, while we were free to build and experiment with a parallel universe of search. Green-field development, unburdened by legacy constraints: every engineer’s dream.

The goals

We had a few explicit goals with the new infrastructure.

* Velocity: we want high velocity when working on schema-affecting bugs and features, so a complete reindex should take on the order of an hour. * Reliability: we want to grow to the next order of magnitude and beyond, so scaling the infrastructure—horizontally and vertically—should require minimum effort. * Maintainability: we don’t want to leak implementation details beyond our borders, so we should strictly control access through APIs that we design and control.

After a survey of the state of the art in search, we decided to abandon Solr in favor of ElasticSearch, which would theoretically address our reliability concerns in their entirety. We then just had to address our velocity and maintenance concerns with our integration with the rest of the SoundCloud universe. To the whiteboard!

The plan

On the read side, everything was pretty straightforward. We’d already built a simple API service between Solr and the main application, but had hit some roadblocks when integrating it. We pivoted and rewrote the backend of that API to make ElasticSearch queries, instead: an advantage of a lightweight service-oriented architecture is that refactoring services in this way is fast and satisfying.

On the write side—conceptually, at least—we had a straightforward task. Search is relatively decoupled from other pieces of business and application logic. To maintain a consistent index, we only need an authoritative source of events, ie. entity creates, updates, and deletes, and some method of enriching those events with their searchable metadata. We had that in the form of our message broker. Every event that search cared about was being broadcast on a well-defined set of exchanges: all we had to do was listen in.

But depending on broker events to materialize a search index is dangerous. We would be susceptible to schema changes: the producers of those events could change the message content, and thereby break search without knowing, potentially in very subtle ways. The simplest, most robust solution was to ignore the content of the messages, and perform our own hydration. That carries a runtime cost, but it would let us better exercise control over our domain of responsibility. We believed the benefits would outweigh the costs.

Using the broker as our event source, and the database as our ground truth, we sketched an ingestion pipeline for the common case. Soon, we realized we had to deal with the problem of synchronization: what should we do if an event hadn’t yet been propagated to the database slave we were querying? It turned out that if we used the timestamp of the event as the “external” version of the document in ElasticSearch, we could lean on ElasticSearch’s consistency guarantees to detect, and resolve, problems of sync.

Once it hit the whiteboard, it became clear we could re-use the common-case pipeline for the bulk-index workflow; we just had to synthesize creation events for every entity in the database. But could we make it fast enough? It seemed at least possible. The final box-diagram looked like this:

Implementation and optimization began. Over a couple of weeks, we iterated over several designs for the indexing service. We eked out a bit more performance in each round, and in the end, we got where we wanted: indexing our entire corpus against a live cluster took around 2 hours, we had no coupling in the application, and ElasticSearch itself gave us a path for scaling.

The benefits

We rolled out the new infrastructure in a dark launch on our beta site. Positive feedback on the time-to-searchability was immediate: newly-posted sounds were discoverable in about 3 seconds (post some sounds and try it out for yourself!). But the true tests came when we started implementing features that had been sitting in our backlog. We would start support for a new feature in the morning, make schema changes into a new index over lunch, and run A/B tests in the afternoon. If all lights were green, we could make an immediate live swap. And if we had any problems, we could rollback to the previous index just as fast.

There’s no onerous, manual work involved in bootstrapping or maintaining nodes: everything is set up with ElasticSearch’s REST API. Our index definitions are specified in flat JSON files. And we have a single dashboard with all the metrics we need to know how search is performing, both in terms of load and search quality.

In short, the new infrastructure has been an unqualified success. Our velocity is through the roof, we have a clear path for orders-of-magnitude growth, and we have a stable platform for a long time to come.

Enter DiscoRank

It’s not the number of degrees of separation between you and John Travolta: DiscoRank is our modification of the PageRank algorithm to provide more relevant search results: short for Discovery Rank. We match your search against our corpus and use DiscoRank to rank the results.

The DiscoRank algorithm involves creating a graph of searchable entities, in which each activity on SoundCloud is an edge connecting two nodes. It then iteratively computes the ranking of each node using the weighted graph of entities and activities. Our graph has millions and millions of nodes and a lot more edges. So, we did a lot of optimizations to adapt PageRank to our use case, to be able to keep the graph in memory, and to recalculate the DiscoRank quickly, using results from previous runs of the algorithm as priors. We keep versioned copies of the DiscoRank so that we can swap between them when testing things out.

How do we know we’re doing better?

Evaluating the relevance of search results is a challenge. You need to know what people are looking for (which is not always apparent from their query) and you need to get a sample that’s representative enough to judge improvement Before we started work on the new infrastructure, we did research and user testing to better understand how and why people were using search on SoundCloud.

Our evaluation baseline is a set of manually-tagged queries. To gather the first iteration of that data, we built an internal evaluation framework for SoundCloud employees: thumbs up, thumbs down on results. In addition, we have metrics like click positions, click engagement, precision, and recall, all constantly updating in our live dashboard. This allows us to compare ranking algorithms and other changes to the way we construct and execute queries. And we have mountains of click log data that we need to analyse.

We have some good improvements so far, but there’s still a lot of tuning that can be done.

Search as navigation

The new Suggest feature lets you jump straight to the sound, profile, group, or set you’re looking for. We’ll trigger your memory if you don’t remember how something is spelled, or exactly what it’s called. Since we know the types of the results returned, we can send you straight to the source. This ability to make assumptions and customizations is a consequence of knowing the structure and semantics of our data, and gives a huge advantage in applications like this.

Architecture-wise, we decided to separate the suggest infrastructure from the main search cluster, and build a custom engine based on Apache Lucene’s Finite State Transducers. As ever, delivering results fast is crucial. Suggestions need to show up right after the keystroke. It turned out we are competing with the speed of light for this particular use-case.

The fact that ElasticSearch doesn’t come with a suggest engine turned out to be a non-issue and rather forced us to build this feature in isolation. This separation proved to be a wise decision, since update-rate, request patterns and customizations are totally different, and would have made a built-in solution hard to maintain.

Search as a crystal ball

Similar to suggest, Explore is based on our new search infrastructure, but with a twist: our main goal for Explore is to showcase the sounds in our community at this very moment. This led to a time-sensitive ranking, putting more emphasis on the newness of a sound.

So far, so good

We hope you enjoy using the new search features of SoundCloud as much as we enjoyed building them. Staying focused on a re-architecture is painful when you see your existing infrastructure failing more and more each day, but we haven’t looked back—especially since our new infrastructure allows us to deliver a better user experience much faster. We now can roll out new functionality with ease, and the new Suggest and Explore along with an improved Search itself are just the first results.

Keep an eye out for more improvements to come!

Apache Lucene and Solr are trademarks of the Apache Software Foundation

Coding for GOOD - SoundCloud API

Fri, 02 Nov 2012 00:00:00 GMT

The web has come a long way. We have APIs that allow developers to create amazing applications quickly, and browser technologies have advanced to the point where JavaScript / HTML and CSS are necessary tools for creating apps. Despite this, there’s still a skills gap and companies are having an increasingly hard time hiring for technical positions.

This is the main reason the folks at GOOD teamed up with the Apollo Group to launch Coding for GOOD. A series of coding lessons and final project. Participants with the top three submissions will be flown to Los Angeles to participate in a hackathon. The best of the three hacks may be offered a job at GOOD.

We were excited when GOOD approached us about contributing a lesson on the SoundCloud API, so I sat down with them in their New York office and did a quick walkthrough on using the JavaScript SDK to create a simple SoundCloud app. Check it out, and if you know of anyone new to using web APIs, have them take a look too.

Story Hack Boston Recap

Tue, 02 Oct 2012 00:00:00 GMT

On Saturday, we hosted Story Hack Boston along with P2PU and Mashery at the MIT Media Lab. About 50 people from content and technical backgrounds joined us to create new story telling experiences. The crowd was pretty evenly split, which made for a lot of awesome collaboration.

This event was the first of it’s kind for us, and therefore a bit of an experiment. Hack days that bring together people from a variety of disciplines around a specific topic can be really interesting and present their own set of challenges. Judging from the tone of the day and the outcome, I think it turned out really well.

There were 10 hacks demo’d in total. A couple of my favorites:

SoundStep - An iOS application that uses geolocation to tell you about stories that have been recorded in your hood! Amazing to see a working iOS app built and demo’d in a single day.
Story Prompter - A web application that pulls headlines from USA Today using Mashery’s API and asks you to record a story about the headline.

We had some really great content hacks as well. See the full list on Hacker League.

Some reports on the day from happy hackers:

A huge thanks to our sponsors, P2PU and Mashery for making this event possible.

P2PU did a great write-up about lessons learned during Story Hack Day. We also got some nice coverage in BostInno before and after the event.

The Next App Gallery Update

Tue, 18 Sep 2012 00:00:00 GMT

We’re making some changes to how we manage our App Gallery and wanted to take some time to explain them to you, our developer community.

The App Gallery is where we highlight interesting and useful SoundCloud powered apps and services for our users. As our developer community continues to grow, it’s even more important that we keep a high bar for apps found in App Gallery. Having a high standard protects the value of being featured in App Gallery for all of our developers while giving our users a sense of confidence in the caliber of content we’re showcasing there.

When evaluating a newly submitted app for App Gallery, we consider the following:

Is the app both high quality and adhering to our API Terms of Use?
Does it enable new and valuable experiences for SoundCloud users?
Has it shown notable traction in gaining connected users?

Moving forward, we’ll be proactively reaching out to folks who meet this criteria about adding their apps to App Gallery instead of accepting mass submissions like we have in the past. Our long term hope is to completely rebuild App Gallery to work in conjunction with Next SoundCloud in a way that better serves our entire developer community. That includes ideas like better attribution of content within SoundCloud products and easier, relevant discovery within App Gallery for SoundCloud users.

If you’re confident your app meets our criteria and we haven’t gotten in touch with you, feel free to drop us a line at api@soundcloud.com.

How to Make a SoundCloud Powered Children's Toy

Thu, 30 Aug 2012 00:00:00 GMT

Just over a week ago we had our first internal hackathon at SoundCloud. You can read (and listen!) about it on our community blog or read some of the awesome press coverage the event received.

We had over 60 people attend and over 20 projects were demoed. I joined a team organized by Josh Devins to build ToyBox, a children’s toy that plays sounds from SoundCloud in response to physical events. The team consisted of myself, Horaci Cuevas, Josh Devins, and Oliver Hookins.

The first task was to come up with a design for our project. An Arduino fitted with gyro, accelerometer and motion sensors provided the interface between real world events and the rest of the system. A Raspberry Pi would then run the following pieces of software:

toybox:downloader - The downloader polls a webapp for data about what sounds from SoundCloud should be played in response to certain events. In order to make the toy more responsive, toybox:downloader downloads the sounds from SoundCloud and stores them on disk (The Raspberry Pi’s SD card).
toybox:playback - An asynchronous wrapper around mplayer.
toybox:serial - An app that listens for data on the Raspberry Pi’s serial port. Event data from the Arduino is printed out on stdout for other processes to consume. toybox:serial also uses 0MQ to push event data to a web interface so you can monitor a toy’s activity from a browser. The 0MQ consumer also responds to events by playing the appropriate sounds using toybox:playback.

To provide an interface for toy owners to register a toy (id’d by the Raspberry Pi’s MAC address) and associate SoundCloud sounds with events, we built a simple Sinatra application that we hosted on Heroku. toybox:webapp also provides a couple of endpoints that serve JSON describing what sounds should be played when for a particular toy. This provided the service endpoint needed by toybox:downloader.

Knowing what needed to be built, we each started hacking on a piece. Horaci worked on toybox:serial, Oliver worked on toybox:downloader and toybox:playback and I worked on toybox:webapp. Josh focused on the Arduino soldering and coding and Oliver and Horaci helped get the Pi set up. You can see the Arduino code here.

There was a bit of frantic integration work at the end, but it did all come together nicely. It was great to see sounds being played from SoundCloud whenever a button was pushed, or the toybox was shaken, tilted or spoken to.

Making it pretty is a job for another hackathon :-)

Evolution of SoundCloud’s Architecture

Thu, 30 Aug 2012 00:00:00 GMT

This is a story of how we adapted our architecture over time to accomodate growth.

Scaling is a luxury problem and surprisingly has more to do with organization than implementation. For each change we addressed the next order of magnitude of users we needed to support, starting in the thousands and now we’re designing for the hundreds of millions. We identify our bottlenecks and addressed them as simply as possible by introducing clear integration points in our infrastructure to divide and conquer each problem individually.

By identifying and extracting points of scale into smaller problems and having well defined integration points when the time arrived, we are able to grow organically.

Product conception

From day one, we had the simple need of getting each idea out of our heads and in front of eyeballs as quickly as possible. During this phase, we used a very simple setup:

Apache was serving our image/style/behavior resources, and Rails backed by MySQL provided an environment where almost all of our product could be modeled, routed and rendered quickly. Most of our team understood this model and could work well together, delivering a product that is very similar to what we have today.

We consciously chose not to implement high availability at this point, knowing what it would take when that time hopefully arrived. At this point we left our private beta, revealing SoundCloud to the public.

Our primary cost optimization was for opportunity, and anything that got in the way of us developing the concepts behind SoundCloud were avoided. For example, when a new comment was posted, we blocked until all followers were notified knowing that we could make that asynchronous later.

In the early stages we were conscious to ensure we were not only building a product, but also a platform. Our Public API was developed alongside our website from the very beginning. We’re now driving the website with the same API we were offering to 3rd party integrations.

Growing out of Apache

Apache served us well, but we were running Rails app servers on multiple hosts, and the routing and virtual host configuration in Apache was cumbersome to keep in sync between development and production.

The Web tier’s primary responsibility is to manage and dispatch incoming web requests, as well as buffering outbound responses so to free up an application server for the next request as quickly as possible. This meant the better connection pooling and content based routing configuration we had, the stronger this tier would be.

At this point we replaced Apache with Nginx and reduced our web tier’s configuration complexity, but our architecture didn’t change.

Load distribution and a little queue theory

Nginx worked great, but as we were growing, we found that some workloads took significantly more time compared to others (in the order of hundreds of milliseconds).

When you’re working on a slow request when a fast request arrives, the fast request will have to wait until the slow request finishes, called “head of the line blocking problem”. When we had multiple applications servers each with its own listen socket backlog, analogous to a grocery store, where you inevitably stand at one register and watch all the other registers move faster than your own.

Around 2008 when we first developed the architecture, concurrent request processing in Rails and ActiveRecord was fairly immature. Even though we felt confident that we could audit and prepare our code for concurrent request processing, we did not want to invest the time to audit our dependencies. So we stuck with the model of a single concurrency per application server process and ran multiple processes per host.

In Kendall’s notation once we’ve sent a request from the web server to the application server, the request processing can be modeled by a M/M/1 queue. The response time of such a queue depends on all prior requests, so if we drastically increase the average work time of one request the average response time also drastically increases.

Of course, the right thing to do is to make sure our work times are consistently low for any web request, but we were still in the period of optimizing for opportunity, so we decided to continue with product development and solve this problem with better request dispatching.

We looked at the Phusion passenger approach of using multiple child processes per host but felt that we could easily fill each child with long-running requests. This is like having many queues with a few workers on each queue, simulating concurrent request processing on a single listen socket.

This changed the queue model from M/M/1 to M/M/c where c is the number of child processes for every dispatched request. This is like the queue system found in a post office, or a “take a number, the next available worker will help you” kind of queue. This model reduces the response time by a factor of c for any job that was waiting in the queue which is better, but assuming we had 5 children, we would just be able to accept an average of 5 times as many slow requests. We were already seeing a factor of 10 growth in the upcoming months, and had limited capacity per host, so adding only 5 to 10 workers was not enough address the head of the line blocking problem.

We wanted a system that never queued, but if it did queue, the wait time in the queue was minimal. Taking the M/M/c model to the extreme, we asked ourselves “how can we make c as large as possible?”

To do this, we needed to make sure that a single Rails application server never received more than one request at a time. This ruled out TCP load balancing because TCP has no notion of an HTTP request/response. We also needed to make sure that if all application servers were busy, the request would be queued for the next available application server. This meant we must maintain complete statelessness between our servers. We had the latter, but didn’t have former.

We added HAProxy into our infrastructure, configuring each backend with a maximum connection count of 1 and added our backend processes across all hosts, to get that wonderful M/M/c reduction in resident wait time by queuing the HTTP request until any backend process on any host becomes available. HAProxy entered as our queuing load balancer that would buffer any temporary back-pressure by queuing requests from the application or dependent backend services so we could defer designing sophisticated queuing in other components in our request pipeline.

I heartily recommend Neil J. Gunther’s work Analyzing Computer System Performance with Perl::PDQ to brush up on queue theory and strengthen your intuition on how to model and measure queuing systems from HTTP requests all the way down to your disk controllers.

Going asynchronous

One class of request that took a long time was the fan-out of notifications from social activity. For example, when you upload a sound to SoundCloud, everyone that follows you will be notified. For people with many followers, if we were to do this synchronously, the request times would exceed the tens of seconds. We needed to queue a job that would be handled later.

Around the same time we were considering how to manage our storage growth for sounds and images, and had chosen to offload storage to Amazon S3 keeping transcoding compute in Amazon EC2.

Coordinating these subsystems, we needed some middleware that would reliably queue, acknowledge and re-deliver job tickets on failure. We went through a few systems, but in the end settled on AMQP because of having a programmable topology, implemented by RabbitMQ.

To keep the same domain logic that we had in the website, we loaded up the Rails environment and built a lightweight dispatcher class with one queue per concern. The queues had a namespace that describes estimated work times. This created a priority system in our asynchronous workers without requiring adding the complexity of message priorities to the broker by starting one dispatcher process for each class of work that bound to multiple queues in that work class. Most of our queues for asynchronous work performed by the application are namespaced with either “interactive” (under 250ms work time) or “batch” (any work time). Other namespaces were used specific to each application.

Caching

When we approached the hundreds of thousands user mark, we saw we were burning too much CPU in the application tier, mostly spent in the rendering engine and Ruby runtime.

Instead of introducing Memcached to alleviate IO contention in the database like most applications, we aggressively cached partial DOM fragments and full pages. This turned into an invalidation problem which we solved by maintaining the reverse index of cache keys that also needed invalidation on model changes in memcached.

Our highest volume request was one specific endpoint that was delivering data for the widget. We created a special route for that endpoint in nginx and added proxy caching to that stack, but wanted to generalize caching to the point where any end point could produce proper HTTP/1.1 cache control headers and would be treated well by an intermediary we control. Now our widget content is served entirely from our public API.

We added Memcached and much later Varnish to our stack to handle backend partially rendered template caching and mostly read-only API responses.

Generalization

Our worker pools grew, handling more asynchronous tasks. The programming model was similar for all of them: take a domain model and schedule a continuation with that model state to be processed at a later state.

Generalizing this pattern, we leveraged the after-save hooks in ActiveRecord models in a way we call ModelBroadcast. The principle is that when the business domain changes, events are dropped on the AMQP bus with that change for any asynchronous client that is interested in that class of change. This technique of decoupling the write path from the readers enables the next evolution of growth by accommodating integrations we hadn’t foreseen.

after_create do |r|
  broker.publish("models", "create.#{r.class.name}", r.attributes.to_json)
end

after_save do |r|
  broker.publish("models", "save.#{r.class.name}", r.changes.to_json)
end

after_destroy do |r|
  broker.publish("models", "destroy.#{r.class.name}", r.attributes.to_json)
end

This isn’t perfect, but it added a much needed non-disruptive, generalized, out-of-app integration point in the course of a day.

Dashboard

Our most rapid data growth was the result of our Dashboard. The Dashboard is a personalized materialized index of activities inside of your social graph and the primary place to personalize your incoming sounds from the people you follow.

We have always had a storage and access problem with this component. Looking at the read and write paths separately, the read path needs to be optimized for sequential access per user over a time range. The write path needs to be optimized for random access where one event may affect millions of users’ indexes.

The solution required a system that could reorder writes from random to sequential and store in sequential format for read that could be grown to multiple hosts. Sorted string tables are a perfect fit for the persistence format, and add the promise of free partitioning and scaling in the mix, we chose Cassandra as the storage system for the Dashboard index.

The intermediary steps started with the model broadcast and used RabbitMQ as a queue for staged processing, in three major steps: fan-out, personalization, and serialization of foreign key references to our domain models.

Fan-out finds the areas of the social graph where an activity should propagate.
Personalization looks at the relationship between the originator and destination users as well as other signals to annotate or filter the index entry.
Serialization persists the index entry in Cassandra for later lookup and joining against our domain models for display or API representations.

Search

Our search is conceptually a back-end service that exposes a subset of data store operations over an HTTP interface for queries. Updating of the index is handled similarly to the dashboard via ModelBroadcast with some enhancement from database replicas with index storage managed by Elastic Search.

Notifications and Stats

To make sure users are properly notified when their dashboard updates, whether this is over iOS/Android push notifications, email or other social networks we simply added another stage in the Dashboard workflow that receives messages when a dashboard index is updated. Agents can get that completion event routed to their own AMQP queues via the message bus to initiate their own logic. Reliable messages at the completion of persistence is part of the eventual consistency we work with throughout our system.

Our statistics offered to logged in users at https://soundcloud.com/you/stats also integrates via the broker, but instead of using ModelBroadcast, we emit special domain events that are queued up in a log then rolled up into a separate database cluster for fast access across the various time ranges.

What’s next

We have established some clear integration points in the broker for asynchronous write paths and in the application for synchronous read and write paths to backend services.

Over time, the application server’s codebase has collected both integration and functional responsibilities. As the product development settles, we have much more confidence now to decouple the function from the integration to be moved into backend services that can be consumed à la carte by not only the application but by other backend services, each with a private namespace in the persistence layer.

The way we develop SoundCloud is to identify the points of scale then isolate and optimize the read and write paths individually, in anticipation of the next magnitude of growth.

At the beginning of the product, our read and write scaling limitations were consumer eyeballs and developer hours. Today, we’re engineering for the realities of limited IO, network and CPU. We have the integration points set up in our architecture, all ready for the continued evolution of SoundCloud!

Shoot yourself in the foot with iptables and kmod auto-loading

Tue, 14 Aug 2012 00:00:00 GMT

As some of you might know, we had an outage yesterday. We believe that in every mistake there is something to learn from, so after each outage we are writing post-mortems. Usually we do this internally because the issues we run into are very specific to our infrastructure.

This time we ran into a quite nasty issue which could affect everyone running a linux system with a lot sessions on it and we thought you might be interested to know about that pitfall.

What happened?

At 4:40pm CEST, we got reports about Yikes (503/504 errors) on SoundCloud. Around the same time, our monitoring alerted for a high amount of 503s at our caching layer and right after that one of our L7 routing nginx instances was reported down.

We were still able to log into that system. dmesg showed:

Aug 13 14:46:52 ams-mid006.int.s-cloud.net kernel: [8623919.136122] nf_conntrack: table full, dropping packet.
Aug 13 14:46:52 ams-mid006.int.s-cloud.net kernel: [8623919.136138] nf_conntrack: table full, dropping packet.

N.B.: Our systems are set to UTC timezone

That wasn’t expected. The first thought was: “Someone must have changed the sysctl tunings for that”. Then we realized that this system has no need for connection tracking, so nf_conntrack shouldn’t be loaded at all. As a quick contermeasure we raised net.ipv4.netfilter.ip_conntrack_max. This fixed that situation and brought the service back up.

Why did it happen?

After bringing the site back up, we investigated what caused the kernel to enable connection tracking. Doing a lsmod showed that connection tracking and iptables modules were actually loaded. Another look into dmesg revealed that right before the outage the ip_tables netfilter module was loaded:

Aug 13 14:38:27 ams-mid006.int.s-cloud.net kernel: [8623415.007818] ip_tables: (C) 2000-2006 Netfilter Core Team
Aug 13 14:38:35 ams-mid006.int.s-cloud.net kernel: [8623422.444931] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)

So what happened? One of our enginners was doing some preparations for scaling that layer of our infrastructure. To verify we don’t use any specific iptable rules on that system, he did:

iptables -L iptables -t nat -L

Those commands themself are pretty harmless. They will just list configured iptables rules. The first one rules in the filter table, the second one in the nat table. Nothing which should change any system configuration, right? Nope. Let’s try to reproduce it. Just boot up some system (I’ve tried it on my Ubuntu Laptop). No iptables module should be loaded:

root@apollon:~# lsmod|grep ipt

Now just list your iptable rules:

iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

And check again for loaded modules:

root@apollon:~# lsmod|grep ipt
iptable_filter 12810 0
ip_tables      27473 1 iptable_filter
x_tables       29846 2 iptable_filter,ip_tables

Okay, that loaded some iptables module to make it possible to add filter rules via iptables. This shouldn’t cause any problems, since without any actual rules the impact on the kernel is negligible. But now check your nat table:

root@apollon:~# iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination

Completely empty as well. But now look at your kernel modules:

 root@apollon:~# lsmod|grep ipt
 iptable_nat 13229 0
 nf_nat 25891 1 iptable_nat
 nf_conntrack_ipv4 19716 3 iptable_nat,nf_nat
 nf_conntrack 81926 3 iptable_nat,nf_nat,nf_conntrack_ipv4
 iptable_filter 12810 0
 ip_tables 27473 2 iptable_nat,iptable_filter
 x_tables 29846 3 iptable_nat,iptable_filter,ip_tables

By just listing your iptable rules for the nat table, the kernel loaded nf_conntrack which enabled connection tracking. See dmesg:

[75024.007681] nf_conntrack version 0.5.0 (16384 buckets, 65536 max

On your Laptop you probably don’t care – it’s even quite convenient. On a production server that handles a large number of connections the fairly small default nf_conntrack table will overflow quite fast and cause dropped connections.

How do we prevent it?

iptables doesn’t load the nf_conntrack itself, it only loads ip_tables which again loads modules it depends on via the kernel’s kmod facility.

But since that module loader uses the modprobe user-space helpers like modprobe, the auto-loading process will honour modprobe.d/ settings. Unfortunatelly there is no easy way to disable loading of a module altogether, but there is a workaround for that.

Since we don’t need iptables at all on that system, we’ve created a /etc/modprobe.d/netfilter.conf like this:

 alias ip_tables off
 alias iptable off
 alias iptable_nat off
 alias iptable_filter off
 alias x_tables off
 alias nf_nat off
 alias nf_conntrack_ipv4 off
 alias nf_conntrack off

This will make modprobe load off instead of the actual kernel module.

Trying to run any iptables command now, should now give you now:

iptables -t nat -L
FATAL: Module off not found. iptables v1.4.12: can't initialize iptables table `nat': Table does not exist (do you need to insmod?) Perhaps iptables or your kernel needs to be upgraded.

Introducing the CloudSeeder Devkit

Mon, 13 Aug 2012 00:00:00 GMT

Today we’re featuring a guest post from our friends at Retronyms. They’ve built some amazing community features into their app Tabletop using the SoundCloud API and have open sourced their CloudSeeder Devkit. This post was written for us by David Shu. David is a software engineer at the Retronyms and has worked on a number of iOS apps, including Tabletop and Dokobots. He currently resides in San Francisco, CA.

We recently built a SoundCloud-powered community into our app Tabletop, a modular audio environment for the iPad, using the CocoaSoundCloudAPI. The project, CloudSeeder, lets users browse, stream, favorite, and comment on Tabletop tracks without ever leaving the app.

As developers, we discovered tons of talented users in our Tabletop community. At the same time, our users found inspiration from each other and a new showcase for their creations. To share in the excitement of community creation with all developers, today we’re releasing the CloudSeeder Devkit as open source on Google Code.

What does CloudSeeder do?

Let’s run through a typical scenario using Tabletop as an example. When you start up an app with CloudSeeder integrated, you’ll get an at-a-glance view of the latest community happenings including app news, latest track posted, and likes and comments on tracks you’ve uploaded. When someone makes a cool new track, everyone who launches the app gets notified immediately in their feed.

Tabletop displaying the Headline view in the upper left hand corner. All the headline items are cycled through with a nice animation and you can size this view to suit your UI.

Tapping on the Headline will bring you to the CloudSeeder community. The Main view will slide up, showing you News, Featured Tracks, Popular Tracks, Latest Tracks, My Uploads, My Activity, and Following. CloudSeeder filters all the tracks for you (using the "created_with" property, for those of you familiar with the SoundCloud API), so all the tracks and activities you see will be specific to your application.

The CloudSeeder Main view showing the Featured Tracks tab.

Curating Featured Tracks can be done in the app. When you point CloudSeeder to a SoundCloud user in the config file, all you have to do is “like” a track as that user and it automatically shows up in the Featured Tracks list! If you’re a user, nothing feels better than when the app creator features your track!

Integrating CloudSeeder into your app

If you’re ready to poke around CloudSeeder, you can head to the Google Code project. We had fast integration and easy customization in mind when developing CloudSeeder, so the repo contains source code and images for UI (including Retina!) that you can just drop into Xcode. You should be ready to go after adding CocoaSoundCloudAPI, filling out the config header file with a few SoundCloud details, and adding the CloudSeeder resources to your project.

We also added a couple new features to the upload process. If you register for the upload notification, you can add tags, description, and BPM to the track metadata. For example:

// Register for the share notification
[[NSNotificationCenter defaultCenter]
    addObserver:self
       selector:@selector(soundCloudShareWillShow:)
           name:kCSShareViewControllerWillShowNotification
         object:nil];
...

- (void)soundCloudShareWillShow:(NSNotification *)notification {
    CSShareViewController *shareVC = (CSShareViewController *)[notification object];

    // Add tags, description, bpm to track metadata
    [shareVC setTags:[NSArray arrayWithObjects:@"TABLETOP", @"CloudSeeder", @"bunnies", nil]];
    [shareVC setDescription:@"Created with Tabletop for iPad"];
    [shareVC setBPM:getTempo()];
}

Be a part of CloudSeeder Select

CloudSeeder has one more trick up its sleeve to benefit developers: cross promotion. Apps that are a part of CloudSeeder Select get featured across all other CloudSeeder Select apps. When users check out their Featured Tracks list, one track out of that list will be a featured track from another fellow CloudSeeder app. Tapping on that track will show the usual track view, along with a section promoting that app.

CloudSeeder Select: Cross promote your app to users in other CloudSeeder apps! A banner on top of the Track view shows the cross promoted app’s details.

A simple email with an icon image and some information on your app will get you in and ready for cross promotion. Find out the details in the documentation.

Get CloudSeeder!

So if you’re the proud developer of an iPad audio app, get the devkit from the Google Code project, see CloudSeeder in action with Tabletop, and check out the CloudSeeder SoundCloud group. Thanks!

Go at SoundCloud

Tue, 24 Jul 2012 00:00:00 GMT

SoundCloud is a polyglot company, and while we’ve always operated with Ruby on Rails at the top of our stack, we’ve got quite a wide variety of languages represented in our backend. I’d like to describe a bit about how—and why—we use Go, an open-source language that recently hit version 1.

It’s in our company DNA that our engineers are generalists, rather than specialists. We hope that everyone will be at least conversant about every part of our infrastructure. Even more, we encourage engineers to change teams, and even form new ones, with as little friction as possible. An environment of shared code ownership is a perfect match for expressive, productive languages with low barriers to entry, and Go has proven to be exactly that.

Go has been described by several engineers here as a WYSIWYG language. That is, the code does exactly what it says on the page. It’s difficult to overemphasize how helpful this property is toward the unambiguous understanding and maintenance of software. Go explicitly rejects “helper” idioms and features like the Uniform Access Principle, operator overloading, default parameters, and even exceptions, on the basis that they create more problems through ambiguity than they solve in expressivity. There’s no question that these decisions carry a cost of keystrokes—especially, as most new engineers on Go projects lament, during error handling—but the payoff is that those same new engineers can easily and immediately build a complete mental model of the application. I feel confident in saying that time from zero to productive commits is faster in Go than any other language we use; sometimes, dramatically so.

Go’s strict formatting rules and its “only one way to do things” philosophy mean we don’t waste much time bikeshedding about style. Code reviews on a Go codebase tend to be more about the problem domain than the intricacies of the language, which everyone appreciates.

Further, once an engineer has a working knowledge of Effective Go, there seems to be very little friction in moving from “how the application behaves today” to “how the application should behave in the ideal case.” Should a slow response from this backend abort the entire request? Should we retry exactly once, and then serve partial results? This agent has been acting strangely: can we install a 250ms timeout? Every high-level scenario in the behavior of a system can be expressed in a straightforward and idiomatic implementation, without the need for libraries or frameworks. Removing layers of abstraction reduces complexity; plainly stated, simpler code is better code.

Go has some other nice properties that we’ve taken advantage of. Static typing and fast compilation enable us to do near-realtime static analysis and unit testing during development. It also means that building, testing and rolling out Go applications through our deployment system is as fast as it gets.

In fact, fast builds, fast tests, fast peer-reviews and fast deployment means that some ideas can go from the whiteboard to running in production in less than an hour. For example, the search infrastructure on Next is driven by Elastic Search, but managed and interfaced with the rest of SoundCloud almost exclusively through Go services. During validation testing, we realized that we needed the ability to mark indexes as read-only in certain circumstances, and needed the indexing applications to detect and respect this new dimension of index-state. Adding the abstraction in the code, polling a new endpoint to reliably detect the state, changing the relevant indexing behaviors, and writing tests for them, all took half an afternoon. By the evening, the changes had been deployed and running under load for hours. That kind of velocity, especially in a statically-typed, natively-compiled language, is exhilarating.

I mentioned our build and deployment system. It’s called Bazooka, and it’s designed to be a platform for managing the deployment of internal services. (We’ll be open-sourcing it pretty soon; stay tuned!) Scaling 12-Factor apps over a heterogeneous network can be thought of as one large, complex state machine, full of opportunities for inconsistency and race conditions. Go was a natural choice for this kind of job. Idiomatic Go is safely concurrent by default; Bazooka developers can reason about the complexity of their problem without being distracted by the complexity of their tools. And Bazooka makes use of Doozer to coordinate its shared state, which—in addition to being the only open-source implementation of Paxos in the wild (that we’re aware of)—is also written in Go.

All together, SoundCloud maintains about half a dozen services and over a dozen repositories written entirely in Go. And we’re increasingly turning to Go when spinning up new backend projects.

Interested in writing Go to solve real problems and build real products? We’d love to hear from you!

Recording and Sharing Phone Calls

Thu, 12 Jul 2012 00:00:00 GMT

A few weeks ago, I attended News Hack Day in San Francisco. News Hack Days are events that bring together journalists, developers and designers for multi day creative coding and brainstorming sessions.

I really like the idea of hack days that bring together people from different backgrounds. After chatting with a few journalists, it became obvious to me that recording interviews on the phone is a real pain. I saw this as an opportunity to build a fun app that would make this easier for people.

I spent most of my time during the two hacking days building Phone Recorder. It’s a simple node.js application that uses Twilio and SoundCloud to record conference calls and upload them to the user’s SoundCloud account as either public or private sounds. You can give it a whirl here or check out the source code on GitHub. I demoed the application at the end of the event and got some really good feedback.

The idea is pretty simple. A user signs into the app using their SoundCloud account and creates a new call. They’re given a number to call, and two sets of codes: one for the moderator and another for participants. When they call the number, they’re prompted to enter a code. Once the call is finished (the moderator hangs up), the audio recording of the call is downloaded from Twilio and uploaded to the moderator’s SoundCloud account.

One tricky bit was associating the moderator’s SoundCloud account (and access token) with a specific call. I decided to use Redis for this. You could use any server-side data store, Redis is just nice and simple, which is great when you’re at a hack day rushing to get something done.

Once that bit was solved, the code that uploads to SoundCloud was simple to write:

exports.postTrack = function(file, title, sharing, token, callback) {
  var stat = fs.statSync(file);
  var mime = 'application/octet-stream';

  var req = rest.post('https://api.soundcloud.com/me/tracks.json', {
    multipart: true,
    data: {
      'track[title]': title,
      'track[sharing]': sharing,
      'track[asset_data]': rest.file(file, null, stat.size, null, mime),
      'oauth_token': token
    }
  });

  return req.on('complete', function(data) {
    return callback(data);
  });
};

// download the audio recording from Twilio and post to SoundCloud
return util.downloadFile(recordingUrl, function(file) {
  var scope = call.private ? "private" : "public";
  return soundcloud.postTrack(path, title, scope, token, function(r) {
    return console.log("Posted: " + r.title);
  });
});

Attending the Hack Day was a good opportunity to build something useful that uses sound on the web in a new, interesting way.

Built an awesome hack using our API? Let me know about it in the comments.

SocialFolders Releases SoundCloud Ruby Gem

Fri, 06 Jul 2012 00:00:00 GMT

There are many approaches to building libraries that wrap HTTP APIs. For many of our officially supported SDKs we chose to build light wrappers around HTTP client libraries with a few added features to make it easier to work with the SoundCloud API. This approach has a few benefits. It guarantees a certain consistency and is relatively easy to maintain. It’s also fairly future proof. Changes in the HTTP API do not typically require updates to client libraries.

Sometimes however, you might be looking for something a bit more feature-full, or with more abstraction from our HTTP API. That’s why I was really happy to see that the great folks at SocialFolders built an alternative SoundCloud Ruby gem and released it to the public. You can check out their blog post about it or go straight to the source on GitHub.

You can install the library from source or by running:

$ gem install soundcloud-client

The library is based on the excellent Faraday HTTP client library and can therefore be used really easily with libraries like EventMachine or Typhoeus. It also provides wrapper methods for all SoundCloud API endpoints. This means that instead of calling a method named for an HTTP verb and providing an endpoint path, you can write code that looks like this:

require 'soundcloud-client'

client = Soundcloud::API.new('ACCESS_TOKEN')
tracks = client.tracks(:order => 'hotness', :limit => 10)
tracks.each do |track|
  puts "Track: #{track.title}"
end

It’s great to see tools like this coming out of our developer community. Huge shout out to Saverio Trioni and the rest of the SocialFolders team.

Music Hack Day Barcelona 2012

Fri, 29 Jun 2012 00:00:00 GMT

SoundCloud loves hack days. Our latest hack day adventure brought us to Music Hack Day in Barcelona and we thought we’d share a bit of the great experience we had there.

Photo by Thomas Bonte

MHD Barcelona was organized by the Music Technology Group of Universitat Pompeu Fabra. The first event on the program was a pre-meeting where all hack day attendees were invited to the university to enter the research labs and see what is under development. Several great projects were demoed including the Reactable (pictured below), and a melody extraction algorithm that when ran extracted a main melody out of a song with several instruments. Fed back into a vocalizer the computer generated singing was eerily, impressively human.

Photo by Thomas Bonte

The MTG had arranged that Music Hack Day would happen within the Sonar music festival and so the hacking venue was surrounded by music-goers of all kinds and the thumping bass of some of the world’s best new artists. Working on your hack while Flying Lotus or Jacques Greene are playing several steps away proved to be one of the most challenging parts of the event but when the clock ran down to zero and all hacks had to be in 41 projects had made it to the wiki and were “ready” for demoing.

Photo by Music Technology Group

The quality of projects completed in such a short time span was really impressive - a sentiment repeated in other articles like The Next Web’s review of the event. Our favorites:

Jamming Invasion - Winners of the SoundCloud prize of 400 Euros towards travel to the next Music Hack Day this hack was a Space Invaders style of game that used SoundCloud tracks and frequency analysis to generate the enemies. The demo was great and you could really see the effect the levels of the music had on the gameplay.

[Free(CC)it!](http://wiki.musichackday.org/index.php?title=Free(CC\)it!) - This was an incredibly clever hack that took a normal music track, analysed it and then built another replacement track out of the closest matching samples from the FreeSound archive. The idea and analysis behind this was really inspiring.

Legalize It! - This hack was an app inside of Spotify that used the Music Metric API to gather the latest quite literally “hot” tracks from torrent sites and then mapped them to available tracks on Spotify so that you can listen to the latest and greatest legally

AriHits - A tilt-controlled iPhone beat machine on that mashed up an instrumental track and a vocal track from SoundCloud. An inspired performance that had heads nodding to the beat.

Tunemap - A very impressive music visualisation that mapped Echnoest and Deezer music info. Incredibly polished for the short time frame the hackers had to work on it.

So to wrap up this post we would like to again thank the amazing organizers of this event. We are crossing our fingers that they decide to organize the same again next year, and if so, we will see you all back in beautiful Barcelona.

Waveforms, Let's Talk About Them

Tue, 26 Jun 2012 00:00:00 GMT

I’ve worked at SoundCloud for over two years now, and if there’s one thing I do a lot, it’s color waveforms. Tons of them. And, I’ve done it several different ways. Today, Johannes and I are pumped to announce a new JavaScript library called Waveform.js that will assist you in your coloring efforts. But first, let’s take some time to look back and learn from past techniques.

Element Stacking

If you’ve used our API before, you’ll know that each track resource contains a waveform_url that points to a 1800 x 280 pixel PNG of that particular sound. The classic way of coloring this image was to not color it at all. Instead, you would simply stack it on top of two or three absolutely positioned DIVs whose size was adjusted given the loading or playing state of that sound.

So, what’s wrong with this technique? Well, you can’t adjust the outer color of the waveform image. So unless the area around the waveform is #E5E5E5, things are going to look subpar.

Webkit Mask

I saw the future, and it was Webkit

Webkit browsers can overcome the outer color issue with a new css function called webkit-mask-box-image. This function colors a div to the shape of a provided mask. In this case, our waveform image. Check out the example below:

.waveform {
    -webkit-mask-box-image: url(https://w1.sndcdn.com/cWHNerOLlkUq_m.png);
    background: #81D8D0;
    height: 100px;
    width: 500px;
}

<div class="waveform"</div>

It’s simple to code and damn powerful. You can combine this function with the element stacking technique above to generate nice waveforms in most environments. But what about non-webkit browsers? HTML 5 to the rescue!

Canvas

Canvas adoption is rapidly increasing and for good reason - it fucking rules. With canvas we can dynamically manipulate an area of pixels with basic JavaScript. By placing the waveform image mask onto the canvas and looping through each pixel, we can adjust the outer color accordingly by simply asking “Is this pixel non-transparent?” and changing it to a specified color.

However, there is one caveat: the Cross-Domain Origin Policy. If you’ve developed with canvas before, you know that accessing and manipulating images from different domains or origins is prohibited. [1] But, we can get around this by accessing the image via the server, and I wrote a service that does exactly that.

Wave64 exists as an endpoint you pass a SoundCloud waveform_url to from the client. It will then read that image on the server and return a Base64 encoded text blob [2] which can be manipulated by canvas. I combined this endpoint with some jQuery and wrote the $.wave64 plugin which takes a waveform url, height, width, and color and returns the colored waveform.

Problem solved? Never. Programming is a math problem you can only make harder on yourself, and as such Johannes and I sought out to simplify this even more.

Waveform.js

Damn Good Waveforms - Eric Wahlforss, CTO of SoundCloud

Johannes had the bright idea to analyze the waveform image’s peaks and store their values in an array of floating points. HUH..? Take a look at the image below:

If every column was considered a point, it’s value would be a floating decimal from 0 to 1 depending on it’s height. So, we wrote a basic server-side script using RMagick to analyze the columns, convert them into 1800 floating points, and cache the result.

Don’t believe me? Here’s Forss - Flickermood.

In addition, Johannes also wrote the new Waveform.js JavaScript library to convert this data into a canvas generated (and interpolated) waveform. Simply provide the container where it will live, an innerColor &&|| outerColor, and the provided waveform data array. Like so:

new Waveform({
  container: document.getElementById("waveform"),
  innerColor: "#81D8D0",
  data: WAVEFORM_DATA,
})

Psyched? Yeh, me too. And, not only can you pass colors to the inner and outer color, you can also provide functions. Check out a few examples of this technique on waveformjs.org, including the ability to render both the loading and playing progress right into a single Waveform.

Waveform.js is open-sourced on Github so go ahead and push, pull, and fork the hell out of it. You can report bugs and discuss features on the Github issues page.

The End?

Probably not. It would rock if the SoundCloud API provided the waveform array data right from the API so we wouldn’t even need the endpoint. I’ll keep complaining internally until this happens.

One More Thing

Johannes and I would like to thank our fellow co-workers Tomas and Robb for their wisdom when building Waveform.js. Make sure to throw some virtual high-fives their way.

[1] However, there are a few workarounds.

[2] Thanks to Max Novakovic for the technique.

OceaniaSmashing Pumpkins use Premiere 1.5 BETA to debut "Oceania"

Mon, 18 Jun 2012 00:00:00 GMT

Smashing Pumpkins are using a BETA of the next version of Premiere to debut their new record “Oceania.” Here’s what’s new about the app.

Moving Stacks

I converted the app from a basic Sinatra app over to my Hipster Stack of Node.js, Jade, and Stylus for rapid development. I chose to use Backbone.js similarly to how I utilized it in Stratus 2 to define a basic player structure. It’s really nice to be able to define a single model for tracks and use it for multiple views. In this case, the model is used to create a track list item and each waveform sound element.

Meta Data

In an effort to make deploying the app even quicker, I’m pulling as much data from your SoundCloud account, album, and track as possible. Simply adjust the single setup line and everything will populate automatically.

The purchase link is pulled from the set. You can even give it a custom title now from SoundCloud.com.
Your Official Site, Twitter, and Facebook links are pulled directly from your account profile.
And, of course, the album and track data is used to generate all the tracks and artwork.

What I’m trying to say is be sure to fill out those SoundCloud profiles completely, and this app will make your life easier.

Waveform

I was going to write an entire section on waveform coloring (exciting, I know) here, but I’ve decided to wait until next week because Johannes and I have written a new library called Waveform.js to handle this. More on that soon. ;-)

I’ve used it in Premiere 1.5 to construct the sound navigator which is heavily inspired by our team’s work on SoundCloud Next.

Permalinks with Pushstate

Each album track is given its own permalink mirrored from the its SoundCloud permalink url. This permalink also drives the URL system of Premiere 1.5 which will change automatically via history.pushState. This allows for deep linking and individual tracks to be shared.

What’s next?

I’m going to build more of the JavaScript SDK’s authenticated functions like commenting, liking, and following right into the app. Once that’s done, I’ll document the app and make it available via open-source on Github.

Premiere 1.5 represents a stepping stone to 2.0 which will see the app being hosted for the first time. This will make it as easy to deploy an album premiere as it is to launch an email for download campaign with Email Unlock.

Building The Next SoundCloud

Thu, 14 Jun 2012 00:00:00 GMT

This article is also available in:

Serbo-Croatian: Pravljenje novog SoundCloud

Armenian: SoundCloud (ձայնամպ) ծրագրավորողների համար

The front-end team at SoundCloud has been building upon our experiences with the HTML5 widget to make the recently-released Next SoundCloud beta as solid as possible. Part of any learning also includes sharing your experiences, so here we outline the front-end architecture of the new site.

Building a single-page application

One of the core features of Next SoundCloud is continuous playback, which allows users to start listening to a sound and continue exploring without ever breaking the experience. Since this really encourages lots of navigation around the site, we also wanted to make that as fast and smooth as possible. These two factors were enough in themselves for us to decide to build Next as a single-page Javascript application. Data is drawn from our public API, but all rendering and navigation happens in the browser for near-instant navigation without the need to make round-trip requests to the server.

As a basis for this style of application, we have used the massively popular Backbone.js. What attracted us to Backbone (apart from the fact that we’re already using it for our Mobile site and the Widget), is that it doesn’t prescribe too much about how it should be used. Backbone still provides a solid basis for working with views, data models and collections, but leaves lots of questions unanswered, and this is where its flexibility and strength really lies.

For rendering the views on the front end, we use the Handlebars templating system. We evaluated several other templating engines, but settled on Handlebars for a few reasons:

No logic is performed inside the templates, which enforces good separation of concerns.
It can be precompiled before deployment which results both in faster rendering and a smaller payload that needs to be sent to clients (the runtime library is only 3.3kb even before gzip).
It allows for custom helpers to be defined.

Modular code

One technique we used with the Widget which ended up being a great success was to write our code in modules and declare all dependencies explicitly.

When we write code, we write in CommonJS-style modules which are converted to AMD modules when they’re executed in the browser. There are some reasons we decided to have this conversion step, possibly best explained by seeing what each style looks like:

// CommonJS module ////////////////
var View = require('lib/view'),
    Sound = require('models/sound'),
    MyView;

MyView = module.exports = View.extend({
    // ...
});

// Equivalent AMD module //////////
define(['require', 'exports', 'module', 'lib/view', 'models/sound'],
  function () {
    var View = require('lib/view'),
      Sound = require('models/sound'),
    MyView;

  MyView = module.exports = View.extend({
      // ...
    });
  }
);

The extra define boilerplate is tedious to write
Duplication of module dependencies is also tedious and error-prone
Conversion from CommonJS to AMD is easily automated, so why not?

During local development, we convert to AMD modules on-the-fly and use RequireJS to load them individually. This makes development quite frictionless as we can just save and refresh to see the updates, however it’s not so great for production since this method creates hundreds of HTTP requests. Instead, the modules are concatenated into several packages, and we drop RequireJS for the super-lightweight module loader AlmondJS.

CSS and Templates as dependencies

Since we’re already including all of our code by defining explicit dependencies, we thought it made sense to also include the CSS and template for a view in the same way. Doing this for templates was rather straight-forward since Handlebars compiles the templates into a Javascript function anyway. For the CSS, it was a new paradigm:

Views define which CSS files are needed for it to display properly, just the same as you would define the javascript modules you need to execute. Only when the view is rendered do we insert its CSS to the page. Of course, there are some common global styles, but mostly, each view has its own small CSS file which just defines the styles for that view.

When writing the code, we write in plain vanilla CSS (without help of preprocessors such as SCSS or LESS), but since they are being included by Require/Almond they need to be converted into modules as well. This is done with a build step which wraps the styles into a function which returns a <style> DOM element. Here’s an example of how it looks in essence:

Input is plain CSS

.myView {
  padding: 5px;
  color: #f0f;
}
.myView__foo {
  border: 1px solid #0f0;
}

Result is an AMD module

define("views/myView.css", [...], function (...) {
  var style = module.exports = document.createElement('style');
  style.appendChild(
    document.createTextNode('.myView { padding: 5px; color ... }');
  );
})

Views as components

A central concept in developing Next is that of treating views as independent and reusable components. Each view can include other ‘sub’ views, which can themselves include subviews and so on. The effect of this is that some views are merely composites of other views and cover an entire page, whereas others can be as small as a button, or even a label in some cases.

Keeping these views independent is very important. Each view is responsible for its own setup, events, data, and clean up. Views are ‘banned’ from modifying the behaviour or appearance of their subviews, or even making assumptions about how or where this view itself is being included. By removing these external factors, it means that each view can be included in any context with absolute minimum fuss, and we can be sure that it will work as it is supposed to.

As an example, the ‘play’ button on Next is a view. To include one anywhere on the site, all that we need to do is create an instance of the button, and tell it the id of the sound it should play. Everything else is handled internally by the button itself.

To actually create these subviews, most of the time they are created inside the template of the parent view. This is done by use of a custom Handlebars helper. Here is a snippet from a template which uses the view helper:

<div class="listenNetwork__creator"> {{view "views/user/user-badge" resource_id=user.id}}
</div>

As you can see, adding a subview is as simple as specifying the module name of the view and passing some minimal instantiation variables. What actually happens behind the scenes goes like this:

When a view is rendered, the template must return a string. When the view helper is invoked, it pushes the attributes passed to it, plus a reference to the requested view class, into a temporary object with an id, and outputs a placeholder element (we use <view data-id="xxxx">). The id is just a unique, incrementing number. After a template is rendered, the output would be a string which might look something like:

<div class="foo">
    <view data-id="123"></view>
</div>

Then we find the placeholders and replace those elements with the subview’s element which it automatically creates for itself. In essence, the code does this:

parentView.$('view').each(function () {
  var id = this.getAttribute('data-id'),
        attrs = theTemporaryObject[id],
      SubView = attrs.ViewClass,
      subView = new SubView(attrs);
  subView.render(); // repeat the process again
  $(this).replaceWith(subView.el);
});

Sharing Models between Views

So, we now have a system where there will be dozens of views on the screen at one time, many of which will be views of the same model. Take, for example, a “listen” page:

There would be a view for the play button, the title of the sound, the waveform, the time since the sound was uploaded (this dynamically updates itself, which is why it is a view), and so on. Each of these views are of the same sound model, but we wouldn’t want each to duplicate the data. Instead we need to find a way to share the model.

Also remember that each of these views has to handle the case where there is no data yet. Almost all views are instantiated only with the id of its model, so it’s quite possible that the data for that model hasn’t been loaded yet.

To solve this, we use a construct we call the instance store. This store is an object which is implicitly accessed and modified each time a constructor for a model is called. When a model is constructed for the first time, it injects itself into the store, using its id as a unique key. If the same model constructor is called with the same id, then the original instance is returned.

var s1 = new Sound({id: 123}),
    s2 = new Sound({id: 123});

s1 === s2; // true, these are the exact same object.

This works because of a surprisingly little-known feature of Javascript. If a constructor returns an object, then that is the value assigned. Therefore, if we return a reference to the instance created earlier, we get the desired behaviour. Behind the scenes, the constructor is basically doing this:

var store = {};

function Sound(attributes) {
    var id = attributes.id;

    // check if this model has already been created
    if (store[id]) {
        // if yes, return that
        return store[id];
    }
    // otherwise, store this instance
    store[id] = this;
}

This is not a particularly new pattern: it’s simply just the Factory Method Pattern wrapped up into the constructor. It could have been written as Sound.create({id: 123}), but since Javascript gives us this expressive ability, it makes sense to use it.

So, this feature means that it’s completely simple for views to share the same instance of a model without knowing anything about the other views, simply by calling the constructor with a single id. We can then use this shared instance as a localised ‘event bus’ to facilitate communication and synchronisation of the views. Usually this is in the form of listening to changes to the model’s data. If the views subscribe to the ‘change’ events which affect it, then they will be notified immediately upon change and the page can be kept up-to-date with very little effort required by the developer.

This also is how we solve the issue of there being no data on the model. On the first pass, several views might have a reference to a model which only contains an id and no other attributes. When the first view is rendered, it can detect that the view does not have enough information and so it would ask the model to fetch its data from the API. The model keeps track of this request, so that when the other views also ask it to fetch, we do nothing and avoid duplicate requests. When the data comes back from the server, the attributes of the model will be updated, causing ‘change’ events which then notify all the views.

Making full use of data

A common feature of many APIs is that when one particular resource is requested, other related resources are included in the response. For example, on SoundCloud, when you request information about a sound, included in the response is a representation of the user who created that sound.

/* api.soundcloud.com/tracks/49931 */
{
  "id": 49931,
  "title": "Hobnotropic",
  ...
  "user": {
    "id": 1433,
    "permalink": "matas",
    "username": "matas",
    "avatar_url": "http://i1.soundc..."
  }
}

Rather than let this extra data go to waste, each model is aware of which ‘sub-resources’ it can expect in its responses. These sub-resources are inserted into the instance store in case any views need to use the data. This means we can save a lot of extra trips to the API and display the views much faster.

So, for our example above, the Sound model would know that in its property “user” it has a representation of a User model. When that data is fetched, then two models are created and populated on the client side:

var sound = new Sound({id: 49931 });

sound
  .fetch()             // get the data
  .done(function () {  // and when it's done
    var user = new User({id: sound.user.id });
    user.get('username'); // 'matas' -- we already have the model data
  });

What’s important to remember is that because there’s only ever one instance of each model, even pre-existing instances are updated. Here’s the same example from above, but note when the User is created.

var sound = new Sound({id: 49931 }),
    user = new User({id: 1433 });

user.get('username'); // undefined -- we haven't fetched anything yet.
sound
  .fetch()
  .done(function () {
    user.get('username'); // 'matas' -- the previous instance is updated
  });

Letting go

Holding on to every instance of every model forever isn’t feasible, especially for the Next SoundCloud. Because of the nature of the site, it’s quite possible that a user might go for several hours without ever performing a page load. During this time, the memory consumption of the application would just continue to grow and grow. Therefore, at some point we need to flush some models out of the instance store. To decide when it is safe to do this, the instance store increments a usage counter each time an instance has been requested, and views can ‘release’ a model when it is no longer needed and the count is decremented.

Periodically, we check the store to see if there are any models with a count of zero, and they’re purged from the store, allowing the browser’s garbage collector to free up the memory. This usage count is encapsulated in the store object, but in essence it’s something like this:

var store = {},
    counts = {};

function Sound(attributes) {
  var id = attributes.id;
  if (store[id]) {
    counts[id]++;
    return store[id];
  }
  store[id] = this;
  counts[id] = 1;
}

Sound.prototype.release = function () {
  counts[this.id]--;
}

The reason for performing the cleanup on a timer, rather than whenever a usage count hits zero is so that the model stays in the store when you switch views. If you navigate to another page, there will be a single moment between cleaning up the existing views and setting up the new ones when every single model’s count is zero. The new page might actually contain views of one or more of these models, so it’d be quite wasteful to remove them instantly.

A long journey…

This has been a brief introduction into some of the methods and concepts we’re using to create Next SoundCloud, but it’s just the beginning. There are plenty more features which we have yet to build and therefore plenty more challenges to tackle. If you want join us along the way, remember that we’re always hiring!

SoundCloud at HackTO

Tue, 17 Apr 2012 00:00:00 GMT

On Saturday, I joined over a hundred other hackers at HackTO. This has become a regular event in the Toronto tech scene, thanks to excellent organizing by Leila Boujnane and Corey Reid. SoundCloud joined several other API providers, including Atomic Reach, Context.IO, FreshBooks, Shopify, TinEye, Twilio, Trendspottr, WordPress and YellowAPI.

The idea behind the event is simple: find a team (or go solo) and build an app using one or more of the APIs presented by sponsors. All apps have to be demoable by 5pm at which time you pitch to the judges who select the 1st, 2nd and 3rd place winners.

For a one day hackathon, there were an impressive number of hacks demoed — 23 in total. Nearly half used SoundCloud, which was awesome. One thing that struck me was how much sound enhanced the demo process. Having an app that produces audio content makes for a much more engaging demonstration. It’s hard to ignore a demo when speakers are blaring.

Here’s a few of my favourites from the event:

HackPad

Built by Alan Hietala, HackPad allows you to create loops and triggers using the SoundCloud API. Alan gave an impressive demo that had the crowd moving (not an easy task after a long day of hacking!). He’s got a write up with an overview of the app and a link to Github.

80s Joke Line

This Twilio and SoundCloud mashup lets users call up a Twilio provisioned number and record or listen to jokes. Jokes are all stored and served by SoundCloud. Ram, Max, Houssam and Sam did a great job demoing. Protip: If your app is based on jokes, your demo will be pretty funny! Check out the app to post your own jokes and listen to others.

Podcasting Robots

Feeling too lazy to read a blog? That’s cool, Mo built this WordPress, SoundCloud and Twilio mashup so you can have robots read it to you. Despite the fun angle, this kind of thing could be a great accessibility app. Could really use some different voices though :-)

It was great to see so many people finish their app in just under 6 hours. Thanks to everyone who participated and special thanks to FreshBooks for hosting!

Create rich media experiences using timed comments

Wed, 11 Apr 2012 00:00:00 GMT

In the last weeks we got pretty excited about the idea of using timed comments to create and script rich media experiences. Imagine being able to trigger all kinds of visualizations & interactions for a timed comment while playing a track.

We’ve already built a few experiments around this concept:

The onTimedComments hack is a simple example showing off the full power of this idea. It takes any kind of link in the comments and translates it into a rich media embed using Embed.ly.
WaveRaid lets you literally play a track. Collect all the timed comments using the play button as your space ship.
With StoryWheel you can create a slideshow from your Instagram pictures and record your story around them. It’s using the timed comments to store where each picture has to show up. Alex & Eric used this to tell the story of SoundCloud.

With the latest version of our Javascript SDK we’ve made it very simple for everyone to use this technique using the new ontimedcomments event. Here is a basic example (press play to try it):

We are looking forward to all the innovative apps & hacks you can come up with using this! As always, we are happy to hear your ideas & thoughts about this.

Desktop Sharing Kits for OS X and Windows

Tue, 10 Apr 2012 00:00:00 GMT

Last week we announced a new integration with Ableton Live 8, that lets you easily share your sounds from within Ableton Live to SoundCloud. Today we’re making the technology behind that integration available to everyone through our new Desktop Sharing Kits.

Similar to our iOS & Android Sharing Kits, our Desktop Sharing Kits let you easily add “Share to SoundCloud” functionality to your Windows and OS X applications.

They come as seperate executables for Microsoft Windows and Mac OS that you include in your application and invoke to let users share a sound to SoundCloud. The sharing kits take care of everything from there by providing a nice UI and handling the entire upload process.

The kits are open sourced and available for download on GitHub (Windows and Mac. We hope to enable some great integrations with this. Let us know what you think!

Wave Raid powered by Timed Comments

Thu, 05 Apr 2012 00:00:00 GMT

One of the SoundCloud API’s most powerful features is the timed comment. At its core it seems simple enough: a piece of text associated with a point in time for a particular track. But where you see timed opinions, I see a light-weight game scripting engine.

So when Johannes Wagener told me he was adding an ontimedcomments event to our JS SDK, I immediately began work on Wave Raid: Quest for the Timed Comment.

Pre Game

I’m fairly new to gaming, html5, canvas… pretty much everything used in building this hack. So Daniel X Moore’s No Tears Guide to HTML5 Games was an absolute godsend. It gives you an extremely solid base to develop HTML5 games while simultaneously explaining the basics of canvas. I wish more tutorials were written like this.

OnTimedComments

So, how does this ontimedcomments event work? Just add it as a function to your SC.stream event and tell your app what to do with the comment chunks as they come in. In my case, I’m creating coins (comments) to collect in game.

Disclaimer: CoffeeScript:

SC.stream TRACK_ID,
    ontimedcomments: (comments) ->
        for comment in comments
            coin = Coin(comment: comment)
            coins.push(coin)

Simple? Yes, indeed. Just loop through the comments and do whatever you want with them. [nerdrant]I may even want to look at the comment user’s details and vary the coin’s properties. Maybe if the comment came from the track owner, it would be a power up or obstacle.[/nerdrant]

Post Game

“That’s great Lee. but I’m not building a game.”

OKAY :( … You can use ontimedcomments in all sorts of ways:

Display comments in time with a playing track visualized in interesting ways. 3D, perhaps?
Maybe your track has an accompanied transcript of timed comments. Wouldn’t it be nice to display those while the track is playing? Yes, it would!
How about rich media content? Add some photos as timed comments to your track and then build an app that displays those in realtime. We used a similar technique on Storywheel.

But don’t take my word for it. Get in there, build something awesome, and let us know about it. We’ll be waiting.

Developer Contest: SoundCloud + Acapela Group Mashup

Wed, 04 Apr 2012 00:00:00 GMT

SoundCloud is teaming up with Acapela Group for our first Developer Contest. Acapela Group offers amazing text to speech solutions and have a variety of SDKs so you can write apps that create sound files using one of their voices (including hip-hop and country!). Take a listen to some sample voices.

We’re calling on our developer community to mashup SoundCloud and Acapela. Show us what you can do using a text to speech service with the best audio content platform on the web. Maybe you want to have your email read (or sung!) to you layered on some instrumental or drum tracks. We think there are loads of options for making interesting and accessible apps using these two services.

Deadline: May 15th

The deadline for submissions is ~~May 1st~~ May 15th. That gives you just under a month to check out our API docs and get started with a Acapela SDK. Sign up for a free Acapela evaluation account here.

When you’re app is ready to show off, send it to us using this form.

Prizes

This is a chance to get creative and show us something useful and fun. We’ve got some great prizes lined up for the top 3 picks:

3rd place — A pair of Monster iSport headphones.
2nd place — A pair of Monster Purity headphones.
1st place — A special prize from Monster (to be announced) and a 1 year subscription to Acapela VaaS

All of our top picks will be showcased on this blog. Special thanks to our contest sponsor Monster and our partner Acapela!

Update: We’ve moved the contest deadline to May 15th. You have a couple more weeks to get those entries in!

HTML5 Widget API

Thu, 22 Mar 2012 00:00:00 GMT

Today we’re officially announcing our JavaScript API for the new HTML5 SoundCloud Widget. To use it, just insert the script tag on a page where you’re using our HTML5 Widget.

<script src="https://w.soundcloud.com/player/api.js" type="text/javascript"></script>

What does it permit?

Access and control the properties of HTML5 Widget

<iframe id="sc-widget" src="https://w.soundcloud.com/player/?url=https://api.soundcloud.com/users/1539950/favorites" width="100%" height="465" scrolling="no" frameborder="no"></iframe>
<script src="https://w.soundcloud.com/player/api.js" type="text/javascript"></script>
<script type="text/javascript">
  (function(){
    var widgetIframe = document.getElementById('sc-widget'),
        widget       = SC.Widget(widgetIframe);

    widget.bind(SC.Widget.Events.READY, function() {
      widget.bind(SC.Widget.Events.PLAY, function() {
        // get information about currently playing sound
        widget.getCurrentSound(function(currentSound) {
          console.log('sound ' + currentSound.get('') + 'began to play');
        });
      });
      // get current level of volume
      widget.getVolume(function(volume) {
        console.log('current volume value is ' + volume);
      });
      // set new volume level
      widget.setVolume(50);
      // get the value of the current position
    });

  }());
</script>

Load widgets

<iframe id="sc-widget" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F1848538&show_artwork=true"></iframe>
<script src="https://w.soundcloud.com/player/api.js" type="text/javascript"></script>
<script type="text/javascript">
  (function(){
    var widgetIframe = document.getElementById('sc-widget'),
        widget       = SC.Widget(widgetIframe),
        newSoundUrl = 'https://api.soundcloud.com/tracks/13692671';

    widget.bind(SC.Widget.Events.READY, function() {
      // load new widget
      widget.bind(SC.Widget.Events.FINISH, function() {
        widget.load(newSoundUrl, {
          show_artwork: false
        });
      });
    });

  }());
</script>

Resources

take a look at our documentation for more details and technical descriptions
check out our playground to get an overview of functionalities provided by the API

Feedback

If you’ve got a suggestion, found a bug, please let us know by leaving a comment below.

Hacker Time – the first 3 months

Mon, 19 Mar 2012 00:00:00 GMT

Over 3 months ago we started the “Hacker Time” initiative (see here) and now it is time for a recap what’s happened so far. From the outset, people outside of the engineering department were very forthcoming in suggesting ideas for Hacker Time projects. While it was great to see so much interest, the essence of Hacker Time is that engineers create the projects that they’re personally motivated to develop. (It’s absolutely key to keep things separate from the general backlog!). Once that clicked, our engineers started initiating and working on their passion projects. Hacker Time has found momentum; currently around 50% of our engineers (who have been in the company longer than 6 months) have a project they are currently working on. These projects range from doing remote online courses to participating in competitions to cool hacks (which sometimes start at “Hack Days” and are then continued within Hacker Time). Some of the highlights are:

Instasound lets you record your sounds, apply a range of filters in real time, and instantly share to SoundCloud – video
Large Hadron Migrator – a tool to migrate database tables online without downtime
HyperSpec – HyperSpec provides a Ruby DSL for testing HTTP APIs from the “outside”
Story Wheel lets you record a story around your Instagram pictures and share it on the web as a nostalgic slideshow.
Massive Site Map – a ruby gem to build painfree google sitemaps for webpages with millions of pages. Differential updates keep generation time short and reduces load on DB.
KDD Cup 2012 – a team of 5 SoundCloud engineers is participating
Some engineers are taking online courses from Stanford, e.g. the machine learning online course

We’ve decided not to track the aggregate hours spent on Hacker Time projects, but to leave it to the teams to organize. It’s an approach that’s working well, because it reinforces the spontaneity and informality that this kind of initiative needs in order to be credible.

So what about the 50% of engineers that haven’t yet worked on their own Hacker Time project? Well, no big surprises here, there are two main reasons: Either they feel they have don’t have enough time or they haven’t found a topic yet.

Overall we’ve seen the initial concept successfully initiated and it’s begun to get into full swing. We’re really pleased with the output so far and will have more updates soon – some engineers will present their projects in more detail!

Stratus 2 BETA

Mon, 12 Mar 2012 00:00:00 GMT

Stratus is a jQuery powered SoundCloud player that lives at the bottom (or top) of your website or blog. In version 2, we’ve rebuilt Stratus from the ground up to include many requested features, a nicer aesthetic, and a much easier installation.

Pumped? Yeh, me too. Here’s how to participate in the BETA:

Installation

See installation guides for: Wordpress Tumblr Blogger

Installing Stratus is easy. All you need to do is add some code to the head section of your HTML.

First, make sure you have jQuery installed.

Next, include the main Stratus library.

<script type="text/javascript" src="https://stratus.sc/stratus.js"></script>

Finally, configure and load the Stratus plugin.

<script type="text/javascript">
  $(document).ready(function(){
    $('body').stratus(function(){
      links: 'https://soundcloud.com/foofighters/sets/wasting-light'
    });
  });
</script>

That’s it! For a list of all configuration options, go here.

Feedback

All good BETA tests require good feedback. If you’ve got a suggestion, found a bug, or deployed the player on your site: Please, let me know @leemartin or at lee@soundcloud.com. We really appreciate the help!

Thanks and happy BETA testing!

Moving API Support to Stack Overflow

Wed, 07 Mar 2012 00:00:00 GMT

The SoundCloud Developer Community has grown immensely. We have over ten thousand registered applications and over three hundred showcased in our App Gallery. Our goal as the Platform Team is to provide the best API tools possible, while also providing support and inspiration. So far, our primary channels have been this blog, our Twitter account, and our mailing list, hosted on Google Groups.

In the coming months, you can expect some changes as we retool to accommodate the growth of our developer community. We are revisiting everything and doubling down on our efforts to provide the best, most accessible platform possible for building amazing applications that use sound in exciting new ways on the web.

Our first major change is that starting today, we are moving our primary support channel from the mailing list to Stack Overflow. Stack Overflow is a hugely popular Q&A site, powered by Stack Exchange. Developers from all communities have been using Stack Overflow to support each other and provide guidance on a great variety of programming topics.

We encourage you to start using the ‘soundcloud’ tag on Stack Overflow from now on for all questions related to the SoundCloud Platform. You can set up email notifications for new questions that show up with our tag, or you can subscribe to an RSS feed. We are monitoring questions, and will chime in when we can be of assistance. Our hope is that, using this new format, we will be able to build up a knowledge base, and will be able to take advantage of the expertise of our platform users as well as members of the broader programming community.

On April 2nd, we will be turning the Google Group into a announcement only list. You can stay subscribed and expect regular updates on new features to the platform. Our team will continue to be active on the mailing list during the transition phase.

See you all on Stack Overflow!

Announcing new Python SDK

Tue, 06 Mar 2012 00:00:00 GMT

It’s no secret that SoundCloud is a Ruby shop, but that doesn’t stop us from giving some love to the Pythonistas in our community.

Our old Python API wrapper has been neglected. It doesn’t support OAuth 2 or all of the resources made available by our API. It’s old and crufty and we’re sorry for letting it get that way. In order to make it up to you, we wrote a new one and made it much better.

Our new Python SDK has an interface that should be very familiar to anyone who has used our Ruby or JS SDKs:

import soundcloud

# Create a new client that uses the user credentials oauth flow
client = soundcloud.Client(client_id='YOUR_CLIENT_ID',
                           client_secret='YOUR_CLIENT_SECRET',
                           username='YOUR_USERNAME',
                           password='YOUR_PASSWORD')

# print the username of the authorized user
print client.get('/me').username

It is available on PyPi, so you can install it using easy_install, or better yet, pip. The source code is available on GitHub, contributions of all kind are very welcome.

If you want to give it a whirl, you can install it from PyPi and take a look at the examples in the README file. As always, let us know if you’re building something cool, we’d love to hear about it!