The Pinterest backend consists of hundreds of microservices deployed across tens of thousands of machines. A simple request to the Pinterest home feed can make hundreds of network calls to dozens of backend services. While we know how long a request takes to execute, previously, we didn’t know why a request was slow. To collect this data, we built Pintrace, a distributed tracing pipeline that tracks a request as it’s processed across our Python and Java backend services. We also built Pintrace collector to help us debug performance issues. Pintrace collector is a Spark job that reads spans from Kafka, aggregates them into traces and stores them in an Elasticsearch backend. Today, we’re open sourcing Pintrace collector on Github, and contributing the code to the OpenZipkin community. By releasing it to the community, we hope others can benefit from the project as much as we do.

Background

Tracking down the source of latency can be very challenging because the slowness can be caused by any one of our backend services. Our metrics (OpenTSDB) and logging (ELK) infrastructure provide a coarse-grained view of how the requests are processed on the backend, but before Pintrace, it was hard to pinpoint why a request was slow. 

To solve this challenge, we turned to distributed tracing, which tracks latency issues and shows where a request spends its time. It provides fine-grained visibility into request execution by capturing request causality information (i.e., which downstream requests were executed as part of a request) in addition to request latency information. Pintrace is our in-house distributed tracing pipeline that tracks a request as it’s processed across our Python and Java backend services. 

Before we look into the details, let’s define some terminology. Annotations are events occurring during a request. For example, sending a request to the server is an annotation. A span represents a logical operation and contains a set of annotations that belong to that operation. For example, a RPC call consisting of "client send" and "client receive" annotations is represented as a span. A trace is a graph of spans. A request execution is usually captured as a trace. All spans in a trace have the same trace ID. For more details on how traces are structured, check out OpenTracing's introduction.

Pintrace: a tracing pipeline

Building a distributed tracing infrastructure involves setting up a tracing pipeline. A tracing pipeline consists of two main components–source code instrumentation and trace processing backend.

The source code instrumentation is responsible for tracking the request across services and generating events (spans) as the request is processed along various services. The trace processing backend is responsible for aggregating the generated spans, processing them into traces and storing them. The stored traces can be searched and visualized in a UI. 

The Pintrace tracing pipeline consists of several components shown in the following diagram. The components in blue are built in-house.

Source code instrumentation

Source code instrumentation is the bulk of the work when setting up a tracing pipeline. The instrumentation is responsible for generating spans that belong to the same request with the same trace ID. To achieve this, the instrumentation does three tasks: (a) propagating the trace ID along with every request (b) generating the spans for every logical operation performed by that component and (c) sending the generated spans to the backend. The first two responsibilities are implemented by a tracer. A logger implements the third responsibility. At Pinterest, we log the spans to Kafka using an analytics tool called Singer. 

Pintrace backend consists of Thrift services written in Python, Java, Go, Node and C++, each using different frameworks. To trace a request across these services, we implemented a tracer for each language framework combination. Pinterest has a common framework per language, and most services we’re interested in are written in Python and Java, so we only had to instrument Python and Java frameworks. When we first started building Pintrace, the initial goal was to track down network request latency, so we focused solely on capturing the network activity of a request as spans in Python and Java applications. 

Python instrumentation

Our Java services are fronted by a monolithic Python application called ngapi, a HTTP service that handles all incoming requests to our backend. To serve a request, ngapi talks to several Java services over Thrift and implements a custom web framework based on the Python gevent library. Since it’s a custom framework, we implemented our own tracer to trace the requests passing through it. 

We wrote a tracer for our web framework using the open tracing API. The tracer propagated the trace ID from the HTTP interface to the Java Thrift service using a custom Thrift wrapper provided by Finagle. In addition, the tracer generated spans for all Thrift, Memcache and HTTP requests. To report these spans, we wrote a logger that took the generated spans, converted them to Zipkin format and logged them to Kafka using Singer.

Since ngapi is managed in a monorepo and deployed as a monolithic application, the change to enable tracing is deployed in a single step across all frontend services.
Java tracer and logger

Our Java services use an in-house framework called “service framework,” a wrapper around Twitter’s Finagle library. Since Finagle ships with a Zipkin tracer, we didn’t build a new tracer for our Java services. By default, Finagle tracer writes the gathered spans to scribe or a log file, but since we wanted to write our spans to Kafka using Singer, we wrote a span logger that logs the spans to Singer.

Once the tracer and logger are in place, we enabled them in our service framework. However, our service framework lacked a way to enable tracing in all apps at once. To avoid duplicating this logic in more than 100 applications, we updated the service framework to have a global config and enabled tracing in it. Then we updated our Java apps to use this global config from the updated service framework. The hardest parts of enabling tracing in Java was making the change in hundreds of  apps and working to deploy the changes across the organization.

Sampler

Even though our client instrumentation has really small latency overhead, there’s double digit computation overhead added by capturing the spans and logging them. To reduce this overhead, as well as the cost of storing the generated traces, we only trace a small set of the overall requests. 

A sampler component decides which requests should be traced and randomly selects between 0 to 1 percent (typically 0.3 percent) of all of our backend requests. This rate can be dynamically adjusted using the decider framework. We also sample all requests on development machines so developers can use traces as part of their day-to-day activities.

Trace processing backend

The trace processing backend, as shown in the diagram above, is responsible for aggregating the spans from thousands of machines in our cluster, processing them into traces and storing and visualizing them. 

Singer-Kafka pipeline

The Singer-Kafka pipeline is responsible for aggregating the spans generated by several processes across thousands of machines and writing them to a Kafka topic. We use Singer as our span aggregation pipeline. The Python and Java loggers log the spans to local files. The Singer daemon installed on each host tails these files and writes the spans to a Kafka topic.

Pintrace collector

We used the OpenZipkin backend to ingest the spans, but the captured traces revealed a few issues. For example, some of our instrumentation can be faulty. In those cases, we’d like to fix the faulty span or drop the span until we fix the instrumentation. Even with careful sampling, at our scale, we see a lot of similar traces. Instead of storing those traces, it’d be more cost-effective to further sample the traces at ingest time and only store high-value traces.

These features were challenging to add to our backend, so we built Pintrace collector, a Spark job that reads spans from Kafka, aggregates them into traces and stores them in an Elasticsearch backend. We chose Spark for implementing the pipeline since it’s flexible and scalable enough to implement the needed filtering, grouping spans by trace ID and aggregating spans over time windows for analytics purposes. As an added bonus, the Spark job allows us to run real-time analytics on the spans without storing them. 

The image below shows the internal architecture of Pintrace collector. StreamFactory is a extensible interface that ingests data from Kafka or any other transport. The ingested spans are then processed to ensure they’re properly formed using a data quality checker. Optionally, the filtering step filters spans based on criteria like service name or annotations on the span. The aggregation phase groups the spans by time or trace ID. The final consumer stage persists the data to a storage system like ElasticSearch service. 

Zipkin UI

As shown in the architectural diagram, we use Zipkin UI to search and view the traces stored in the ElasticSearch cluster. We’ve also contributed a few bug fixes to the Zipkin UI.

Open sourcing Pintrace collector

Over the last year, we’ve noticed others in the OpenZipkin community looking for the same solutions we implemented in Pintrace, such as more advanced sampling and stream processing to support Vizceral visualizations. We wanted to share our work with the community while also leveraging its collective expertise. In order to be as open as possible, we’re contributing our code directly to OpenZipkin under zipkin-sparkstreaming code base. We encourage you to watch or collaborate with us in building Zipkin-spark. We hope you find the Pintrace collector as useful as we do. We can’t wait to see new ideas from the community.

Acknowledgements: The contributors to this project are Naoman Abbas, Phoebe Tse, Ashley Huynh, Alejandro Garcia Salas, Emmanuel Udotong, Brian Overstreet, Xiaoqiao Meng and Suman Karumuri. We would also like to thank Adrian Cole for his feedback and advice during the project.

Tags: 

This week we announced Shop the Look, a new way to shop and buy products inside home and fashion Pins. With more than 150 million people discovering billions of ideas every month on Pinterest, and fifty-five percent using the app to shop and plan purchases, we wanted to make it easier than ever to go from inspiration and action. Shop the Look combines our computer vision technology with human curation to recommend a variety of related products and styles you can bring to life in just a tap. In this post, we’ll share how we engineered Shop the Look. 

Overview 

We built two different technologies to make products inside Pins shoppable. First, we’ll explain how we combined computer vision technology with human curation to help people shop on Pinterest. The second approach ingests content from content networks and automatically makes products inside Pins shoppable.

Approach #1: combining computer vision with human curation 

Leveraging our object detection technology, we built new machine-assisted curation tools that help marry human taste with computer vision. Given a fashion Pin, we use object detection to infer areas of the image that contain products and apply our visual search technology to quickly find and parse through millions of products in order to generate a set of candidate products. 

Human curators with fashion expertise confirm the most relevant products within the candidate. We can then recommend these products in Shop the Look. 

Approach #2: Ingesting third party data

To make Shop the Look more scalable, we built a feed ingestion system for content networks. Now we can easily ingest external feeds containing image links, product links and XY coordinates.

Our daily workflow uploads feed files to S3 parses and validates the feed. We save any errors back to S3 to notify brands and networks. With the processed feed, the system then creates Pins on behalf of the content network and matches their products with Pins people can buy on Pinterest. After, products are tagged inside the Pin using the XY coordinates within the feed. Our jobs are idempotent and can be rerun at any point without leading to duplicate content or causing the feature to break.

On occasion brands or networks don’t provide XY coordinates. This was a challenge we solved using our object detection technology. First, we detect the objects and coordinates within the source image. We then cross-match the annotations with the same object detection performed on the product and assign the product an XY coordinate on the source image.

Data model

We built Shop The Look using a graph-based model. To represent the tag relationship, we created a node called PinTag which contains the XY coordinate and an edge with a given fashion or home Pin. We also created edges from PinTag to different Pins with similar products. Here’s an example:  

The graph-based solution allows us to build even more useful features: 

• How to wear it: Different inspirational Pins often share common products. For any Buyable Pin, we can easily query the reverse to find inspirational images containing the same product.  This helps Pinners see more ways to style a scarf, wear the latest jean trend or make a product they already own work for them in new ways.
  
• Pin co-occurrence: Shop the Look and How to Wear it both traverse the width of the graph once. If we traverse the graph several times, we start to draw relationships between products to take advantage of the accurate data human curation provides. In the future, we’ll be able to infer products that are similar or complementary. This means we can recommend looks composed of items that go well together.

Infrastructure

We built this on top of a specialized Zen cluster we call Zen Canonical Pin. Zen is our in-house graph-based storage system, and a Canonical Pin is a data model representing Pins that share the same link and image hash (like the Pins above). Zen Canonical Pin gives us two key benefits.

• Performance: The cluster is specifically designed to deliver low read latency, accomplished by high cache availability. The requests on this cluster surpass 400K QPS, while delivering p99 read latency of ~20ms. This is critical as Shop the Look content often lives on our most popular Pins.
• Coverage: Storing this data on the canonical Pin representation amplifies Shop the Look across our entire product. Whenever a Pin shares the same image signature and link, we’ll light up that Pin with Shop the Look content. This makes more Pins shoppable, so it's easier to discover a Shop the Look Pin. 

Once the data is stored, it’s ready for serving. We built Shop the Look for all our platforms–iOS, Android and web.

What’s next

Early tests show Pinners engage with Shop the Look Pins 3-4x more than Pins without Shop the Look, save Shop the Look Pins 5x more and visit a brand’s website 2-3x more. Now that we’re rolling out this experience to everyone, we have a few upcoming initiatives:

• Unify Shop the Look with visual search: As our computer vision accuracy improves, we can scale Shop the Look exponentially. We hope to unify our visual search experience with the content in Shop The Look.
• Increase value: We hope to bring more content networks and brands onto the platform to help Pinners discover more products they don’t even need to know the names of to find and buy.

Shop the Look is now available to Pinners in the U.S. We hope you give it a try! 

Acknowledgements: This project was a cross functional effort in collaboration with Rocir Santiago, Joyce Zha, Rahul Pandey, Casey Aylward, Coralie Sabin, Cheryl Chepusova, Cherie Yagi, Kim O’Rourke, Anne Purves, Tiffany Chao, Aimee Bidlack, Aya Nakanishi, Tim Weingarten, Adam Barton and Yuan Tian. People across the whole company helped launch this feature with their insights and feedback. 

Tags: 

Our mission at Pinterest is to help you discover and do things you love. Under the hood, we’re powering a visual discovery engine with 100B ideas saved by 150M people around the world. Today we’re introducing three new visual discovery products–Lens, Instant Ideas and Shop the Look–that turn any image into an entry point to finding more ideas. 

Visual search at Pinterest

In 2014 we started investing heavily in computer vision and created a small team focused on reinventing the ways people find images. Less than a year later we launched visual search, a new way to search for ideas without text queries. For the first time, visual search gave people a way to get results even when they can’t find the right words to describe what they’re looking for. Last summer, visual search evolved as we introduced object detection, which finds all the objects in a Pin’s image in real-time and serves related results. Today, visual search has become one of our most-used features, with hundreds of millions of visual searches every month, and billions of objects detected. Now, we’re introducing three new products on top of our visual discovery infrastructure. 

Pinterest Lens

Lens is a new way to discover ideas with your phone’s camera inspired by what you see in the world around you. Just take a photo of anything that interests you–a pair of shoes, seasonal produce, a chair you’d like to see in your home–and Lens will give you recommendations for similar objects or ways to bring the idea, recipe or style to life. If you’ve ever wanted an easy way to find a jacket like something you spot on the street, or get ideas for decorating your house with the same style as your designer friend with a single tap of button, Lens is for you.

Lens builds on the advancements we’ve made in computer vision and machine learning over the last year and goes beyond visual similarity. Lens tries to understand the objects you’re looking at and how they could be useful to you. For example, if you see strawberries on sale at the grocery store, just point Lens at them, take a picture and get creative recipe ideas (like chocolate strawberry waffle balls...on a stick!?), not just strawberry images. If you point Lens at an awesome pair of sneakers you find, Lens won’t just find visually similar sneakers, but will return outfit ideas with the same sneaker style. You can even point Lens at the night sky to find ideas related to constellations, galaxies and UFOs. 

We’ll share an in-depth technical blog post on how we built Lens in the coming days. In the meantime, we’re busy rolling out Lens in BETA on iOS and Android. Ultimately, Lens will get better as more people use it. If you have feedback as you use Lens, please let us know!

Inspired by the world Lens has revealed, we’re also introducing new ways to find ideas on Pinterest in just a tap. 

Shop the Look 

Shop the Look is a new way to shop and buy products you see inside Pins. It combines our computer vision technology with human curation to recommend a variety of related products and styles you can buy on Pinterest or from a brand in just a tap. For Buyable Pins results, we’ll also show you ideas for bringing a look to life. For example, if you see a scarf you like, now you can see different ways to style it right on the Pin. We’ll share an in-depth technical blog post on how we engineered Shop the Look soon, and for now, you can check it out on iOS, Android and the web. 

Instant Ideas 

Building on the machine learning that powers 10B recommendations on Pinterest every day, today we’re launching Instant Ideas, a new way to transform your home feed with similar ideas in just a tap. So as you’re browsing through your home feed and find something interesting, now you can get more ideas like it by tapping the circle in the bottom right corner of any Pin. We then use these signals to personalize your home feed in real-time, making recommendations instantly more relevant to your tastes and interest as you scroll through your feed. Check Instant Ideas out on iOS, Android and web starting today. 

With these new products, it’s now easier than ever to find ideas across Pinterest and from the world around you. We’re excited to see how people use these technologies. Look for technical deep dives on these products in the coming days! 

Tags: 

This post follows a previous one, Evolving MySQL Compression.

Pinterest’s main data source–Pin data–is stored as medium-sized (~1.2kb) JSON blobs in our MySQL cluster. These blobs are very compressible, but the existing compression system in MySQL was less than optimal and only resulted in 2:1 compression. In a previous post, we discussed why column compression is a more ideal compression system. In order to use column compression and have significant savings, we need to use a compression and optional predefined compression dictionary (i.e. lookback window). Here we’ll cover how we increased the compression ratio of Pin data from around 3:1 to 3.47:1.  

Background

First, let’s take a look at the underlying compression library used by InnoDB, Zlib, a compression library initially used for the PNG file format. Zlib’s compression follows the DEFLATE compression algorithm which consists of two main stages. First, LZ77 replaces recurring strings with a pointer to a previous instance of that string as well as the length of the string to repeat. Then the resulting data is Huffman encoded. We optimize the LZ77 stage by using a predefined dictionary that preloads the lookback window with common substrings. This works especially well since each data object compressed is much smaller than the sliding window, allowing the entirety of each Pin object to look back into the majority of the predefined window. We were inspired by similar work done by Cloudflare. 

Initial tests

Before starting this project, the predefined dictionary was populated with a single Pin object. The compression ratio was roughly 3:1, which is good given all Pin objects share roughly the same scheme, allowing keys to be reused. However, we knew we could do better. 

It’s common for Pins to share certain JSON fields or substrings of those fields. By using only a single Pin, we weren’t taking advantage of additional savings accessible through those JSON values. Examples of such fields are links which often start with “http://” or fields expecting boolean values. I ran a few quick compressions to ensure there were savings to be had. I took roughly 10,000 Pins, concatenated them and compressed them using the largest window size (32kb) and highest compression level (Z_BEST_COMPRESSION, 9). This allowed each Pin to look back into the previous 25 or so Pins for common strings while compressing the data as much as possible. I then did the same thing, but, shuffled the bytes of each Pin before concatenating. Because the relative frequency of each byte is still the same (disregarding strings that are replaced with the LZ77 lookback tuple), the Huffman encoding portion of DEFLATE yielded roughly the same savings. The difference in compression ratios would provide an approximation of expected savings. Another option was to populate the predefined dictionary with a handful of Pins using deflateSetDictionary, but the first method allowed roughly the same outcome without having to write code. Regardless, the results showed there were savings to be had, so I continued on with the project. 

The process

Initially, I tried modifying the Zlib source to determine which common strings the LZ77 portion compressed for use in the pre-defined dictionary. Programming in C isn’t my fastest coding language, so after a week or so I decided it’d be less painful to write to write my own utilities to generate a decent pre-defined dictionary for Pin objects than to spend my internship trying to become proficient with highly optimized K&R style C. 

I took a large number of Pins (~200k) from two different shard DBs and two different generation eras. We shard based on userID, so earlier shard DBs contain a lot of older Pins with somewhat different schemas than more recently added Pins. 

In order to generate a more general pre-defined dictionary for all generations of Pins, I shuffled the input data in three different ways:

1. Random Pin pairs from the older shard.
2. Random Pin pairs from the newer shard.
3. Random Pins from the older shard paired with random Pins from the newer shard. 

I then batched the Pin pairs and concurrently looked for common substrings (within the lookback window and lookahead buffer, all larger than 3 bytes) between each Pin pair in the batch. 

Because this is a long running process, I maintained persistent state at the end of each completed batch which allowed the script to start from the last completed batch in the event of failure. The substring frequencies from each batch were then aggregated and passed onto the next step, dictionary generation. 

Dictionary generation

After batching, the substring are scored based on the length of substring and the occurring frequency. In order to avoid repeating substrings in the output dictionary, we swallow fully contained substrings and incorporate the score into the swallowing substring’s score. The highest n scoring strings that fit in the defined dictionary size are concatenated such that higher scoring strings are at the end of the dictionary (as per the zlib manual). Then, any overhang is truncated and the results are handed to the user as the pre-defined dictionary. 

Although the implementation was slow, we found the frequencies of common substrings between 100,000 pairs of Pins in slightly more than 400 CPU days. We didn’t devote time to improving performance, because generating the dictionary itself isn’t a frequently run task. Furthermore, as much as I would’ve liked to come up with an efficient shortest common supersequence solution to fit more in the pre-defined dictionary, it’s NP-complete.

These tools have been added to our open source tools repository in the hope they might be useful to others.

Benchmarks

Benchmarks using this predefined dictionary looked promising. In terms of space savings, the computed dictionary saves more than 10 percent versus using a single Pin as the predefined dictionary, in addition to 40 percent savings over the existing InnoDB Page Compression. We considered using an 8kb or 16KB dictionary over a ~32KB (32506B) dictionary as a tradeoff to increase performance, but since it wasn’t a requirement, we didn’t increase the zlib compression level to maximize the compression savings.

Next steps

When the first blog post on column compression was written, we hadn’t deployed this change to production. Since, we’ve pushed column compression out without incident.

In the future, we’re planning to test a Pin-curated static Huffman tree as a potential pathway to additional savings. Currently, zlib doesn’t support this functionality and will either use a general static code or a dynamic code, depending on size. 

Acknowledgements: Thanks to Rob Wultsch for his guidance and mentorship. Thanks to Pinterest for making my internship a memorable experience.

Tags: 

As video content becomes more ubiquitous on the internet, it’s an increasingly important part of Pinterest. Pinterest is one of the largest visual platforms in the world, where people have saved billions of pieces of rich media, making it ripe for further video disruption. As part of our progress with video, we launched native video Pins last year to help Pinners more seamlessly watch videos right on Pinterest. Here we’ll cover how and why we built a video platform behind it. Read on!

The benefits of native video

There are a number of factors we considered before making the decision to build our own video platform. For several years, Pinterest provided third party video support by embedding YouTube clips in Pins. This simple solution worked, but had limits. By introducing a native video infrastructure, we made the following gains:

• More content variety. WIth the launch of a native video player, we’re able to give partners the opportunity to create diversified and unique content for their audience on Pinterest. 
• Improved understanding of user needs. With native videos, we can better understand how Pinners interact with clips and run experiments easily in comparison to embedding. Collecting signals from the video player also allows us to improve our infrastructure by monitoring how many times a video stalled, what are the hotspots (e.g. particularly interesting part of TED talk can be in the middle of the clip) and more. 
• Customized playback features. Since the video player is a Pinterest product, we have control over its progress. For example, we can deliver some of our animated GIFs as videos, and customizing the player allows us to hide controls and make it look like a regular GIF file.
• Better recommendations. Each time a video is uploaded to Pinterest, we analyze its content using machine learning, identify objects using computer vision (e.g. broccoli, shoes, abstract art) and categorize it by topic (e.g. food, products and home decor). These signals are used to improve the relevancy of the 10 billion recommendations we make every day, as well as help us identify content that shouldn’t be included on the platform (such as spam and illegal content).
• Steps toward a monetization strategy. We now have flexibility in bringing value to publishers in the form of injecting ads (pre-, mid- or post-roll), sponsored content or adopting other models most suitable for Pinners and publishers.
• More room for technical innovation. As the amount of content grew we ran into new technical challenges, like providing efficient access to video corpus for batch processing, training our deep networks, cost-effective re-encoding and more. 

Under the hood 

Our video platform has multiple moving pieces that work together to make our video experience great. We’ll talk about each of them in more detail in subsequent posts, but here’s a brief summary.

Ingestion pipeline + upload tools

We wanted to make the video experience great for both Pinners and content producers. Our video platform provides multiple ways for uploading media files, including APIs, a dedicated uploader, MRSS feed ingestion and the Pin Collective, a group of expert creators who support video publishers. All of this makes a producer’s life easier.

Content analysis

Once a video is uploaded we analyze its content, feed recommendation and search engines with extra signals.

Video delivery & playback

Pinners access video in different circumstances. For example, some watch videos during their daily commute, on 3G connections and tiny screens, while others watch it on 4K displays with fast broadband access. We optimize video streams for multiple scenarios by using Adaptive Bitrate Streaming and collecting information about playback performance from both video players and content delivery networks.

Review tools

Customized review tools allow us to quickly identify unwanted content (both automatically and by hand), support scheduled publication for large partners and review automated classification results.

Visibility & experimentation

This includes documentation, examples, analytics and more. Pinterest is experimentation-driven company and our systems allow to easily try new features and learn how people respond to them. Once the feature is ready, we document it and ship it.

Challenges and lessons learned

Pinterest is made up of billions of pieces of uniquely dynamic content, and it’s key to understand the differences between images and video. 

High-quality video content is scarce. There’s a significant cost to video production and users’ expectations are higher. It also results in relatively more material being subject to copyright violations and takedown requests. Additionally, vdeo infrastructure can be much more expensive than static images because of their size and the large amount of processing power and bandwidth they require.

Acquiring video content also takes time. We worked closely with several large content providers to understand the engineering solutions that would be most useful to them. For instance, we developed a MRSS feed ingestion service that's customized to handle each partner’s pre-existing feed. We also invested in visibility tools to ensure partners could track success metrics.

One of the most valuable lessons we learned was the importance of understanding consumer needs, including: 

• How much time are they willing to dedicate to watching content, and how long will they wait for video to load before they abandon it? It could be less than a second for products like Pinterest or several seconds in the case of VoD platforms.
• What type of content are they after? Is it DIY tutorials, food recipies, movie trailers or full movies? We learned starting with versatile types of content and running experiments is the most effective long term solution.
• Are there any technical limitations? If most of views are coming from mobile devices, costs associated with mobile bandwidth, battery drain and muting sound by default should be considered. If it’s in-house UHD displays, quality may be the number one concern.

What’s next

We put a lot of work into building a video platform at Pinterest. This post only briefly summarizes some of our findings. Stay tuned for more posts on building and ranking video!

If you’re interested in helping us build the best video experience for millions of Pinners and partners, reach out!

Tags: 

One of the most popular ways people find ideas on Pinterest is through Related Pins, an item-to-item recommendations system that uses collaborative filtering. Previously, candidates were generated using board co-occurrence, signals from all the boards a Pin is saved to. Now, for the first time, we’re applying deep learning to make Related Pins even more relevant. Ultimately, we developed a scalable system that evolves with our product and people’s interests, so we can surface the most relevant recommendations through Related Pins. In this post, we’ll cover how we use deep learning to generate recommendation candidates, which, in testing, has increased engagement with Related Pins by 5 percent. 

Board co-occurrence 

For years, Related Pins have been generated using board co-occurrence. For example, Pin X is saved to N boards by many people. Related Pins for Pin X are generated from all the other Pins saved to the N boards. Board co-occurrence provides infinite candidates to recommend: every Pin has been saved to a board, therefore every Pin has at least some Pins we can generate recommendations from. 

However, board co-occurrence has a few disadvantages:

• Board segmentation. Pinners often have multiple boards for one interest. Someone may save a wine-related idea to a wine board, but cocktails made with wine to a separate cocktails board. This gap in boards makes it challenging to recommend related drinks for the wine Pin.
• Board granularity. Boards are usually created for a broad topic, so Related Pins surfaced using board co-occurrence can be tangentially related. Figure 2 shows a query Pin of a lion couple cuddling and was saved to the boards “animals” and “wild animals.” As a result, the Related Pins are not exactly cuddling lions, but all kinds of wild animals.
• Board drifting. The topic of a board tends to change over time as an interest evolves. For example, a “healthy” board could start with fitness ideas and eventually evolve into other areas like recipes.  

Board co-occurrence can lose the context of a Pin, so we needed a way to better understand Pins and their relative relationships to Pinners. Inspired by the Word2vec approach for creating word embeddings in the context of human language, we developed Pin2vec to embed Pins in the context of Pinners' activity.

Using deep learning to generate Related Pins 

We built Pin2Vec to embed all the Pins in a 128-dimension space. First, we label a Pin with all the other Pins someone has saved in his/her activity session, each as a Pin tuple. Pin tuples are used in supervised training to train the embedding matrix for each of the tens of millions of Pins of the vocabulary. We use TensorFlow as the trainer. At serving time, a set of nearest neighbors are found as Related Pins in the space for each of the Pins. 

Training data is collected from recent engagement, such as saving or clicking, and a sliding window is applied. Low quality Pins and those not engaged with are removed from training. Then, each Pin is assigned with a unique Pin ID. Within the sliding window, training pairs are extracted such that the first Pin is the example and each of the following Pins is its label. Figure 3 illustrates an example session and training pairs. In our case, you can imagine each user session is a sentence with Pins as words.

We used a feedforward neural network with a hidden layer of 128 dimensions. Figure 4 shows the architecture. The network is inspired by word2vec. The input vector is a one-hot vector with a size of vocabulary and, in our case, is tens of millions of Pins. The vector is reduced to the 128-dimension vector by multiplying with the hidden layer weight matrix. An eLu activation function is applied after hidden layer. At last the hidden layer output is multiplied with the softmax matrix and a cross-entropy is used to calculate the loss. We sampled 64 negative Pins in loss optimization in lieu of iterating on tens of millions of Pins. We trained the Pin2Vec embedding on machines with 32 cores and 244GB memory. 

The learned Pin2Vec groups Pins with respect to a Pinner’s recent engagement. The Related Pins are closer to each in the high dimensional space. Figure 5 shows the 3D presentation of the Pins, and each is depicted as a point.

Next, we look up the N nearest Pins for each of the Pins by their distances. We found the results are more relevant than using board co-occurrence signals. Figure 6 shows the Pin2Vec-found Related Pins for the cuddling lions. Every Pin is an image of a couple of female and male lions snuggling up, suggesting Pinners were looking for the exact topic when they saved these Pins. Figure 7 shows the Related Pins for a bottle of wine. Board co-occurrence only found images of bottled wines, whereas Pin2Vec found recommendations for drinks made with wine. This suggests Pinners actually saved the bottled wine Pin and the wine cocktail Pins in the same time series. Pin2Vec helps bridge the gap.

Pin2Vec has become an important source for generating candidates, but it doesn’t replace board co-occurrence. In our tests, we’ve found the board co-occurrence is better performing for long tail Pins that are sparse in engagement data. Both board co-occurrence and Pin2Vec are used to generate candidates, while a separate rerank system sorts the candidates based on richer features (we’ll discuss reranking in a separate post). The euclidean distance of Pin2Vec neighbors will be one of the features.

Pin2Vec is now an important candidate source powering Related Pins. It’s a big step in improving the relevancy of Related Pins. Looking ahead, we're already making our models faster and analyzing more signals to better personalize recommendations for Pinners around the world.

If you’re interested in building neural network learned embedding like Pin2Vec, join us!

Acknowledgements: This technology was built in collaboration with Diveesh Singh, David Liu, Raymond Shiau, Mukund Narasimhan, Andrew Zhai, Frances Haugen, Zhigang Zhong, Stephenie Rogers, Derek Cheng, Vanja Josifovski, Dmitry Chechik, Xin Liu, Dmitry Kislyuk and Jenny Liu. People across the whole company helped launch this feature with their insights and feedback. 

Tags: 

Pinterest is a catalog of ideas people use to discover and do what they love. With more than half of Pinners using our app outside of the U.S., we’ve been making significant improvements to show people content that matches their unique tastes, no matter where they live or what language they speak. Before, when we talked to Pinners around the world, one common piece of feedback was that Pinterest felt foreign to them from the moment they signed up. In this post we’ll cover how we overhauled the new user experience for people outside the U.S., made it more personalized and increased new user activation and retention by 5-10 percent. 

Revamping the new user experience

Our first step in improving the signup experience for users outside the U.S. was understanding what people look for when they come to Pinterest. Conceptually, Pinterest organizes interests and ideas into what we call topics. When someone first signs up, we ask them to identify their interests from a set of topics we then turn into a personalized home feed. Previously, international Pinners chose from the same set of topics as those in the U.S. but since interests differ around the world, we wanted to personalize this experience.

Figure 1: Previous new user topic picker for men in Germany

Identifying local interests

To find the most relevant topics, we analyzed what people find most interesting on Pinterest and how they engage with that content. We looked at the most common board topics from country to country because saving a Pin to a board is an important signal for a user’s interests. We also analyzed a massive search query log to identify relevant topics using top search queries by country. After combining these signals, we sent topic candidates to our in-country teams for review. This approach leveraged both data signals from Pinner engagement and domain knowledge from local experts. 

Supporting content country by country

After identifying a set of topics for a country, the next step was finding the best content. With increasing international usage, we now have Pins in more languages than ever. From previous A/B experiments, we know Pinners are much more likely to save Pins in their language, so for candidate topics, we only show content that’s locally relevant. One of the best ways to do this is to show Pins that other local users are saving. Recently, we developed a new way to support topics for different countries by showing Pins from relevant Boards.

A simple way to find boards related to a given topic is by matching the topic’s name with a board’s title. Instead of using a strict title match, we used fuzzy matching with synonyms in order to increase the number of candidates. In our matching strategy, we ignored whitespace and non-alphabetical characters. For example, when matching “dogs” we also looked at similarly named boards like “dog”, “d o g s”, and “puppies!! ♥”.

Next, we ranked the candidates using a rich set of signals including Pinner engagement, board relevancy and recent board activity. To boost local content, we scored boards higher if they had a large number of Pins that were relevant to a Pinner’s language. After ranking, we further vetted the boards by sending them to our in-country teams to help ensure high relevance and quality.

Figure 2: Overhauled new user topic picker for men in Germany

Impact

When we rolled out these changes abroad, we saw a huge increase in new user activation, and those who signed up were 5-10 percent more likely to come back and use Pinterest. Our experiments confirmed that personalization is critical for helping Pinners outside the U.S. get more value from Pinterest. This new user onboarding experience is available for anyone who speaks French, German, Japanese, Portuguese or Spanish, and we’re planning to roll it out to more languages soon. 

Next steps

While this overhaul is a big improvement, it’s just another step towards delivering a more personalized experience for Pinners around the world. Looking ahead, we’re actively working on further improvements to the experience by providing even better content to Pinners in more countries and more languages.

Tags: 

Pinterest Search handles billions of queries every month. Every day, we help millions of Pinners discover useful ideas by delivering results among billions of Pins saved by people with overlapping tastes. In the early days, we built our first search system on top of Solr and Lucene. Over past few years, we’ve evolved our search stack by adding new layers, designing services and experimenting with ranking functions. Advancements in our search product have resulted in product evolutions over the years and across platforms that range from the search guides that have become industry standard, to improved results based on signals like interests and location, to visual search that uses the latest in computer vision. In this post, we’ll provide an overview of our search serving and ranking stack, and look ahead to future improvements.

Life of a search query

The following diagram shows the life of a search query on Pinterest.

When a Pinner searches on Pinterest, the query goes from our API layer to our search backend. In the search backend, an Anticlimax service understands the query, Obelix machines find the most relevant Pins for the query and an Asterix service coordinates the returned results.

Asterix

Asterix serves as the super root of our search system. It first talks to Anticlimax to understand the query intent and then rewrites the query. With the query rewrite result, Asterix constructs a personalized ranking function that may favor personalized results, such as local content, fresh content or partial matches.

Inside Asterix, there are three major components: cluster client, rerankers and blenders. The cluster client is a scatter/gather service that distributes search requests to Obelix nodes, waits for results and then merges results together. (The cluster client also retries on outliers and handles partial results and other failures.) The merged search result is then reranked based on different business logic. For example, a machine learned reranker generates a new ranking score for each Pin based on context features, while a local reranker boosts Pins in a Pinner’s language.

The search results from different clusters are blended using both simple and complex blending logic. For instance, we can use proportional blending to insert 10 percent fresh Pins into results. More complex blending logic allows us to, for example, surface Buyable Pins in results based on query intent, such as “men’s black sneakers”.

Anticlimax

Anticlimax is our query understanding and rewrite service. It has a pluggable interface so engineers can plug in their rewriters and datasets. Each query rewriter takes in a structured query and user information from the previous worker and rewrites it into another structured query.

All query rewriters are chained in a sequence, and we execute them one by one. Spell correction and query segmentation must be executed before other rewriters. Query expansion, query category prediction and other workers can switch order or execute in parallel.

We support different data sources for each query rewriter. For example, the spell correction model is stored in memory, larger dictionaries are stored as HFile on disk and query category prediction data is stored in a different service (which we query for every search).

Obelix

Obelix is a single leaf search server. It receives search requests from Asterix, retrieves and scores matching Pins from the index and returns top results to Asterix.

An Obelix server may have multiple index segments. The Pins inside each index segment are ranked according to query independent score. This score measures Pin quality, which is an important factor of the final ranking. With the static rank, we’re able to score first few retrieved Pins for most search queries and guarantee the best Pins are scored. Static rank also enables us to have complex functions for scoring.

The searcher in Obelix scans and scores multiple index segments in parallel. It maximize our CPU usage during non-peak hours and improves latency.

Search Ranking

Search ranking is the process of choosing the most relevant, useful and personalized Pins for a search query. We solve this unique problem from four different aspects.

1. Query. Similar to other search engines, Pinterest query rewrite does spell correction, query segmentation, category prediction and other rewrites. However, a unique aspect of Pinterest search is what Pinners are searching for. On Pinterest, people issue exploratory queries for ideas versus asking objective questions. To provide diverse results, we developed a context-based query expansion. By analyzing query logs and engagement data, we extract query pairs that have similar keyword context and engaged result, and use them to construct term expansions. For example, “relief” can expand to “remedies stress” under context “anxiety”. After expanding our result, we provide guides to help user drill down specific interests.
2. Content. Pinterest has a unique, human curated dataset constructed of Pins, boards and Pinners. We explore different signals from our content, some human readable (e.g. board titles) and some not (e.g. embedding vectors for Pins, boards and Pinners).

Let’s look at an example of work we’ve done to better understand board titles. Let’s say a Pinner saves a Pin about “Daisy Ridley at the 2016 Academy Awards” to a board called “Oscar gowns.” We extract dozens of signals, such as board title frequency, Pin topic distribution and board quality, and can then understand this Pin is not only about “Daisy Ridley” but also related to “oscar gowns” and, more specifically, “2016 oscar gowns.”

Personalization

Now that 150 million people use Pinterest every month, and more than half of Pinners are outside the U.S., the ideas and interests on the platform are more diverse than ever. This presents a huge technical challenge–showing the right idea to the right person at the right time. For example, if a Pinner searches “football” in the UK, he or she is likely not looking for American football content. To make search on Pinterest more personal for every Pinner, we  boost results based on certain preferences, such as gender, location and language.

There’s a lot more we can do to personalize the search experience for Pinners, and we’re exploring strategies and building plans for the future of personalized search.

Ranking

We use a machine learning model to score search results. The model aims to optimize user engagement with results after issuing the query, such as a Pinner saving a result or clicking through depending on the query’s intent. As mentioned, model features are from query, content and Pinners, like the text matching score between a rewritten query and a specific text source (e.g. board title), the match between search query category and Pin category or the match between Pin description language and a Pinner’s language. In the end, all features are combined in a linear function, and we can still understand and tweak certain parameters.

We’re also experimenting with different models including neural networks and gradient boosting decision trees. There’s a lot of ongoing work,both in terms of search quality and the infrastructure side so we can better understand content and apply complicated scoring functions online.

Conclusion

Building a personalized search experience is a huge technical challenge. Our search infrastructure and ranking are still very young, and there are endless opportunities for engineers to make a big impact. We’re building our next generation serving system with capabilities for better personalization. We’re also investigating different signals, ranking models and verticals that provide better user experiences. If you’re passionate about helping millions of people find personalized, useful ideas, join our team!  

We would like to acknowledge the tremendous contributions of the search team Keita Fujii, Kevin Ma, Laksh Bhasin, Maesen Churchill, Matthew Fong, Roger Wang, Rui Jiang, Timothy Koh, Ying Huang, Yuliang Yin, Zhao Zheng, Zheng Liu, and Zhongxian Chen.

Tags: 

In 2015, we made the decision to migrate our legacy web experience to React to keep up with our fast growth and perform better with increased developer velocity. Ultimately, we found React rendered faster than our previous template engine, had fewer obstacles to iterating on features and had a large developer community. Expanding on a previous post which covered migrating Pinner profiles to React, here we’ll dive deeper into migrating the web infrastructure to serve React pages, which required moving a huge amount of code without breaking the site.

The roadmap to React

When we began this project, Pinterest.com was humming along on the existing architecture for some time. On the server, Django, a Python web application framework, served our web requests and Jinja was rendering our templates. The server response to the browser included all the markup, assets and data the browser needed to fetch our JavaScript, images and CSS, and initialize our client-side application. Nunjucks, a JavaScript template rendering engine that uses the same template syntax as Jinja, did all subsequent template renders on the client side.

The template syntax and the stack looked like this:

This architecture worked, since the template syntax was (almost) the same between Jinja and Nunjucks. However, the template rendering utilities and libraries had to be duplicated, as illustrated above, meaning for every Jinja Python utility we needed to add, we had to write a JavaScript version for Nunjucks. This was cumbersome, resulted in a lot of bugs and was yet another reason to move to a world where template renders would happen using the same language and engine both client-side and server-side.

Here's a diagram of what our end-goal was for consolidating template rendering in React:

It looks pretty good: we can share utilities and libraries between client and server, and we have one engine, React, rendering templates on both. But how do we get there? If we switch our client-side rendering engine from Nunjucks to React, we’d also have to switch our server-side rendering, so they could share the same template syntax. Halting development so we could switch all of our templates to React wasn’t an option.

We needed a solution that would allow us to iteratively convert the hundreds of Pinterest components without interrupting the work of product teams or the experience of Pinners. That solution looks like this:

The first step was to consolidate to a single template rendering engine between client and server before we could replace that engine with something else. If the server could interpret JavaScript, use Nunjucks to render templates and share our client-side code, we could then move forward with an iterative migration to React.

Server-side Nunjucks architecture

When we first considered how we’d interpret JavaScript on the server-side, there were two main choices: PyV8 and Node. PyV8 had the advantage of giving us a quick way to prototype and not worry too much about standing up a separate service, but it wasn’t well maintained and the memory footprint of the package was significant.

Node was a more natural choice, despite the overhead of standing up a new service and that we'd be communicating with this service via a network interface with its own complexities (described more in the next section). There was a large community supporting and using Node and we’d have better control over tuning and optimizing the service. 

In the end, we went with standing up Node processes behind an Nginx proxy layer and architected the interface in such a way that each network request would be a stateless render. This allowed us to farm requests out to the process group and scale the number of processes as needed.

We also refactored our client-side rendering JavaScript so it could be used by both the client and the server. This resulted in a synchronous rendering module with an API that took environment-agnostic requests and returned final markup using Nunjucks. Both Node and the browser called this module to get HTML.

On the web server, we short-circuited template rendering so instead of calling Jinja, it made network requests to farm the template render out to our Node workers.

Before: Jinja renders an entire module tree in one pass

Pinterest templates are structured as trees. A root module calls children modules, which also have children modules, etc., and a render pass traverses these modules to generate the resulting HTML which makes up the final result.

Each module can either render based on the data it receives from its parent, or it can request a network call be made to acquire more data in order for rendering to continue. These data requests are necessarily blocking, since we don't know the render path until we hit the node. This means module tree rendering is blocked by downstream data requests that can initiate at any time.

Because Python is doing all the rendering on a single thread, renders block the thread and are essentially serial.

The purple circle that appears when the user agent makes a request represents a module render request with no data. The API is called to get the data, filling the circle and readying it for a render pass. Rendering materializes the children and stops when it reaches children that need data. Subsequent calls to the API fulfill these data requests and rendering continues. 

After: Nunjucks requests over the wire

As before, a user agent makes a request which results in a latent module render request that needs data. Data is obtained again by making a call to the API, but another network call is made to a co-located Node process to render the template as far as it can go with the data that it has.

Then, Node sends back a response with the rendered templates, and also a "holes" array indicating the modules the worker was unable to render because they still need data. Our Python webapp then provides the data they need by calling the API, and each module is sent back to Node as completely independent module requests in parallel. This is repeated until the entire tree is rendered and all requests return with no holes.

Rollout

Confidence in the new system was key to rolling it out. Developers were still building with Jinja and creating and modifying new Python utilities, and we had to be sure the new system didn’t introduce latency to page loads for Pinners. We also had to build error handling, service monitoring, alerts and a runbook to scale maintenance and troubleshooting of the new Node processes.

There were many dependencies for ensuring a smooth transition, and two tools were essential to the project's success.

Linters and tests. Jinja and Nunjucks syntax is close to the same, but not identical. The difference in what each template engine supported as well as the language differences in Python and Nunjucks forced us to keep tight restrictions on what engineers could do with templates. Ultimately, we needed to ensure templates rendered on the server would render identically on the client, and templates rendered by Jinja would render identically when rendered by Nunjucks.

At Pinterest, we rely heavily on build-time linters that prevent developers from doing things that would break the site as they develop, which assisted in making sure all templates being developed only used the subset of features supported by both Jinja and Nunjucks. We even wrote a special extensible Nunjucks extension that takes custom rules we write, written in an ESLint-style fashion, and applies them to all the Nunjucks templates during every build. We also implemented a special all-encompassing unit test suite called "render all tests" that literally rendered every single template and ensured they rendered identically between Jinja and Nunjucks, and between client and Node. This helped safeguard our releases from crazy bugs that would’ve been extremely difficult to track down.

Pinterest experiment framework. We rolled out the new architecture to employees only at first, and then to a very small percentage of Pinners. We kept an eye on the metrics that track user activity and performance via our experiment dashboard. A gradual rollout allowed us to track down tricky render bugs, Python/JavaScript discrepancies and performance issues before the majority of Pinners were  exposed to the new system.

One example of a bug caught by the experiment dashboard was a nuanced client-side-only render bug that only affected a tiny percentage of users on a specific browser doing a very specific action. Tracking this action allowed us to narrow in on the bug and verify when it got fixed:

Performance

Server-side rendering plays an important role in serving rich content on Pinterest to Pinners. We rely on performant server response times in order to provide a faster experience and maintain good SEO standing.

During early iterations, the Nunjucks architecture was slower than our existing Jinja setup on the server side. Making multiple network calls introduced extra overhead (preparing the request, serializing and deserializing data), and the roundtrips added nontrivial milliseconds to our render time.

We did two things that helped bring down the delta and allow us to launch.

Parallelization. With Jinja, we didn't need to call a sidecar process over a network protocol in order to render a template. However, because of the CPU-bound nature of template rendering, this also meant Jinja template renders couldn’t be meaningfully parallelized. This wasn’t the case with our Nunjucks render calls. By parallelizing our calls with gevent, we were able to kick off simultaneous network connections to our proxying nginx layer which farmed the requests out to available workers very efficiently.

Avoid unnecessary data serialization. There were several hotspots in our template rendering where we were simply embedding large amounts of data in the markup in order to send to the browser. These were located mainly in the static head and around the body end tags, and were consistent for every web request. A big slowdown was the serialization and deserialization of these huge JSON blobs of data to our workers. Avoiding this helped us gain another performance edge that finally got us to parity.

Here's a graph of the results (Nunjucks in red, Jinja in green):

React is happening

Once the Nunjucks engine was in place and serving 100 percent of Pinterest.com's templates, it was open season for developers to start converting their modules to React components. Today, with Nunjucks code quickly being replaced by React conversions all over the codebase, we’re deprecating our old framework and happily tackling the challenges of building a complete React application while seeing many performance and developer productivity gains from the migration.

Tags: 

Billions of videos are viewed across the internet every day, but video on Pinterest is unique. On Pinterest, you’ve always been able to save videos from around the web, and in 2013, we made it possible to play them right on the Pin when you tap. Videos help people visualize how to bring ideas to life, which make them the perfect medium for Pinterest. In fact, we found that people who watch videos on their smartphones are more than twice as likely to take action when they come across a video on Pinterest than on another app. 

Today we’re making watching videos on Pinterest even more seamless with the launch of our first native video player. Starting today, people can watch native videos from experts and tastemakers on Pinterest in the newly launched Explore, as well as in their home feed as those they follow save videos.

In this post, we’ll dive into how we built video into our iOS app including best practices we learned while working with AVFoundation and building a native video experience from scratch.

AVFoundation 101

Here we’ll introduce a few key concepts in AVFoundation as context for the rest of the post. For those of you who have been working with AVFoundation or AVKit, feel free to skip ahead. 

We built our video experience using AVFoundation, the standard framework provided by Apple that sits on top of Core Media, Core Audio and Core Animation. It provides an interface to a powerful framework which allows developers to create time-based audiovisual media.   

Terms

• AVPlayer: The engine of video playing on iOS that provides control interfaces for video playback.
  

  /* Examples of Provided APIs */
  - (AVPlayerStatus)status;
  - play();
  - pause();
  - (void)seekToTime:(CMTime)time;
  
  
  - addPeriodicTimeObserverForInterval;
  - addBoundaryTimeObserverForTimes;

• AVPlayerLayer:  A subclass of CALayer to which an AVPlayer directs its visual output. Think of it as a monitor: it renders content for the connected AVPlayer. This is how you create one:
  

  + (AVPlayerLayer *)playerLayerWithPlayer:(AVPlayer *)player;

  One of our favorite features built into AVFoundation enables developers to create multiple AVPlayerLayers against a single AVPlayer so all AVPlayerLayers play the output of that AVPlayer in sync. This made modularizing player container views and transitions between ViewControllers easy to implement.

• AVPlayerItem: An AVPlayerItem is initialized with an AVAsset and represents a playable resource. 

Goals

We had three main goals while architecting the video framework:

1. Meet current and potential long-term product needs. Since we’re building a client-side infrastructure to accommodate future product needs, we made the video components modular, portable and robust.
2. Be resource savvy since videos can be extremely resource intensive and can cause OOM crashes or out-of-disk-space shutdowns. Additionally, if we weren’t careful, we could drain a user’s data or their battery quickly. 
3. Prioritize performance and minimize frame drops in scrolling and in transitions. Playing videos is inherently main thread intensive, so we had to balance the video playback experience and its potential UI performance impacts.

A User Scenario

Before we dive into the architecture, let’s consider the following user scenario: 

1. User finds a video Pin autoplay-ing in the home feed.
2. User taps on it and the video zooms up into Pin Closeup and continue to play. 
3. User taps on the full-screen button and the video zooms into full-screen view.

In this scenario, there are three ViewControllers involved, Feed, Pin close up and Full Screen. Each is complex and modular, but they all play the same video continuously and are connected by two different interactive transitions so the video plays continuously and performantly during transitions. In addition, each has different playback control interfaces.

Overview

From the user scenario described above, there are a few baseline requirements we wanted to build:  

1. Ensure all ViewControllers that play the same video share one AVPlayer-AVPlayerItem pair to avoid each loading separate buffers and maintaining separate playback states. 
2. Because of #1, the ViewControllers shouldn’t own the AVPlayer-AVPlayerItem pair. 
3. Each ViewController should manage their own view hierarchy. As ViewControllers are heavily used throughout the app, managing hand-off could become challenging if they share the same view. 

We built our first iteration of the architecture with the following key components: 

• PlayerController (a data structure) holds the AVPlayer-AVPlayerItem pair and provides APIs to control playback and buffering. 
• PlayerControllerManager (a singleton manager) holds strong references to PlayerControllers through a dictionary keyed by video identifiers.
• PlayerLayerContainer (a container view class) is used in all ViewControllers and contains an AVPlayerLayer created off the AVPlayer from the PlayerController. 

All this terminology can be confusing, so the below diagram of this base architecture should help.

At the top of the diagram is the singleton manager that holds strong references to the PlayerControllers (only weak references are allowed anywhere else in the app). To access the PlayerController, you just need the video ID. The central manager design enables us to prefetch or clear when needed. 

The gray boxes in the ViewControllers are the PlayerLayerContainers. Each creates an AVPlayerLayer through the shared AVPlayer and puts the AVPlayerLayer in its view hierarchy. This helps with making these ViewControllers modular, however there’s a caveat we’ll get to later. 

Controls

The control interfaces can be generalized into two categories: pushers and pullers. The pushers are ones Pinners can take action on, such as play/pause, mute/unmute, seek, etc. The pullers control elements that listen for playback state changes and update themselves accordingly (e.g. progress indicator, buffering indicator, timestamps, etc.). 

The pusher controls are easy, but the pullers are slightly trickier. In order to reflect the accurate playback state, you have to listen to many different broadcasted notifications as well as setting up Key-Value-Observations on various property values. To highlight a couple:

/* Notifications names */
AVPlayerItemDidPlayToEndTimeNotification
AVPlayerItemPlaybackStalledNotification 
AVPlayerItemNewAccessLogEntryNotification


/* Properties to KVO */
BOOL playbackLikelyToKeepUp;
NSArray<NSValue *> *loadedTimeRanges;

One thing that can often trip us iOS developers is setting up and removing observers. What can be particularly troublesome is seeing a bunch of observer-related code sprinkled everywhere. We decided to wrap all of this logic in an EventObserver object which handles removing listeners and observers based on its own object lifecycle. It only listens for specific events based on its delegate flags, because handling event callbacks can be pretty main thread intensive.

Caveats

We don’t live in a perfect world and there are always hidden caveats in the SDK that can be tricky. 

• Problem: Creating an AVPlayerLayer against an AVPlayer while a video is playing caused a noticeable skip in playback. Because of this, we couldn’t allow each VC to have its own AVPlayerLayer. Instead they'd have to share the same player layer to avoid the skip in playback.
• Solution: We extended the PlayerController to own the shared instance of AVPlayerLayer together with the AVPlayer-AVPlayerItem pair while at the same time refactoring the way PlayerLayerContainer worked. We came up with a solution inspired by the First Responder analogy from UIKit, where the PlayerLayerContainer would claim itself as the current presenter of the PlayerController whenever it became visible and then return the AVPlayerLayer so the container could add it to its view hierarchy.
   
• Problem: AVPlayerLayer holds strong references to its AVPlayer, so it can’t clear out the buffer when needed.
• Solution: The refactor in problem #1 helped resolve this issue. When PlayerLayerContainer becomes invisible, it resigns current presenter and removes its reference to the AVPlayerLayer, allowing the PlayerController to remove all references to the AVPlayer.
   
• Problem: Error code: “Cannot decode” due to the limitation of up to four “render pipelines” allowed in an app (reference Stackoverflow). We noticed an app can create more than four AVPlayer instances, but can’t connect more than four AVPlayers with AVPlayerItems.
• Solution: We do not set an AVPlayer’s currentItem to the AVPlayerItem until we need to start playback. Then, we remove the currentItem as soon as the playback is no longer needed.

Other tips 

1. The AVPlayerItemNewAccessLogEntryNotification carries the latest entry of AVPlayerItemAccessLogEvent which contains a lot of useful information and has proved to be quite helpful for getting insights on video performance.
2. The built-in buffering strategy for video playback favors avoiding stalls over time to begin playback. On poor network connections, such as a 3G network, the player would sometimes buffer more than 10 seconds of data before beginning to play. Apple introduced a new API in iOS10, preferredForwardBufferDuration, that allows developers to adjust this built-in behavior. Unfortunately, there’s nothing we can do for iOS 9 and below.
3. Audio is important for a great video experience, yet it can also be unwanted at times, especially for mobile users. iOS wrapped complex sound behaviors into a few standard categories, and we currently use a mix of the two most-used categories: 
   • AVAudioSessionCategoryPlayback: Two important traits of this audio category:
     • Your app cuts off any music/video apps playing in the background.
     • Audio streams of your app will play sound regardless of your iPhone’s hardware ringer mute control.
   • AVAudioSessionCategoryAmbient: When your app’s audio session is set to this category, audio streams will respect the hardware ringer mute switch, meaning you won’t be able to play any sound when the user’s hardware mute is ON.
     Note that your app doesn’t need to stick to one of these provided categories. We use different categories for different user scenarios.
4. The playbackLikelyToKeepUp property on AVPlayerItem is your best friend if you need to display loading spinners when the videos are buffering.
5. If you’re using HLS protocol for video streams, make good use of preferredPeakBitRate property. We set it to be a lower value when the user is in cellular so the first few segments loads faster (resulting in shorter wait times) and so we don’t drain a user’s cellular data by loading an unnecessarily high resolution video stream.

Forward thinking product development

When we build new product features into the codebase, we have to think ahead in terms of product progression so we make the right architectural decisions. For example, we wanted to build a way for native videos to play upon close ups of Pins, and we anticipated next building an immersive full-screen view that would play the same video continuously. After that, we could consider adding a custom transition in between, and the video could be play continuously during the transition. The list goes on.

Thinking ahead benefited us in so many ways. In particular, because we had most of the components built, we were able to build an immersive iOS video experience for Explore in just a few weeks.

Try it out the new framework for yourself by visiting the new Pinterest Explore today.

Acknowledgements: Huge shoutouts to Ricky Cancro who worked closely and tireless with me on designing and implementing all of the video components from ground up. Thanks to Steven Ramkumar, Scott Goodson and Max Gu for providing invaluable feedback and suggestions along the way!

Tags: