Staticpulse Writings

Open Sourcing Screwdriver, Yahoo’s Continuous Delivery Build System for Dynamic Infrastructure

Thu, 12 Jan 2017 12:57:52 -0500

By James Collins, Sr. Director, Developer Platforms and Services, and St. John Johnson, Principal Engineer
Continuous Delivery enables software development teams to move faster and adapt to users’ needs quicker by reducing the inherent friction associated with releasing software changes. Yahoo’s engineering has modernized as it has embraced Continuous Delivery as a strategy for improving product quality and engineering agility. All our active products deliver from commit to production with full automation and this has greatly improved Yahoo’s ability to deliver products.
Part of what enabled Yahoo to make Continuous Delivery at scale a reality was our improved build and release tooling. Now, we are open sourcing an adaptation of our code as Screwdriver.cd, a new streamlined build system designed to enable Continuous Delivery to production at scale for dynamic infrastructure.
Some of the key design features of Screwdriver have helped Yahoo achieve Continuous Delivery at scale. At a high level these are:
Making deployment pipelines easy
Optimizing for trunk development
Making rolling back easy
Easy deployment pipelines: Deployment pipelines that continuously test, integrate, and deploy code to production greatly reduce the risk of errors and reduce the time to get feedback to developers. The challenge for many groups had been that pipelines were cumbersome to setup and maintain. We designed a solution that made pipelines easy to configure and completely self-service for any developer. By managing the pipeline configuration in the code repository Screwdriver allows developers to configure pipelines in a manner familiar to them, and as a bonus, to easily code review pipeline changes too.
Trunk development: Internally, we encourage workflows where the trunk is always shippable. Our teams use a modified GitHub flow for their workflows. Pull Requests (PRs) are the entry point for running tests and ensuring code that entered the repository has been sufficiently tested. Insisting on formal PRs also improves the quality of our code reviews.
To ensure trunks are shippable, we enable functional testing of code in the PRs. Internally, this is a configuration baked into pipelines that dynamically allocates compute resources, deploys the code, and runs tests. These tests include web testing using tools like Selenium. These dynamically-allocated resources are also available for a period after the PR build, allowing engineers to interact with the system and review visual aspects of their changes.
Easy rollbacks: To allow for easy code rollbacks, we allow phases of the pipeline to be re-run at a previously-saved state. We leverage features in our PaaS to handle the deployment, but we store and pass metadata to enable us to re-run from a specific git SHA with the same deployment data. This allows us to roll back to a previous state in production. This design makes rolling back as easy as selecting a version from a dropdown menu and clicking “deploy.” Anyone with write access to the project can make this change. This helped us move teams to a DevOps model where developers were responsible for the production state.
The successful growth of Screwdriver over the past 5 years at Yahoo has today led to Screwdriver being synonymous with Continuous Delivery within the company. Screwdriver handles over 25,000+ builds per day and 12,000+ daily git commits as a single shared entrypoint for Yahoo. It supports multiple languages and handles both virtual machine and container-based builds and deployment.
Screwdriver.cd’s architecture is comprised of four main components: a frontend for serving content to the user, a stateless API that orchestrates between user interactions and build operations, the execution engines (Docker Swarm, Kubernetes, etc.) that checkout source code and execute in containers, and the launcher that executes and monitors commands inside the container.
The diagram below shows this architecture overlaid with a typical developer flow.
To give some context around our execution engines, internal Screwdriver started as an abstraction layer on top of Jenkins and used Docker to provide isolation, common build containers, etc. We used features provided by Jenkins plugins to leverage existing work around coverage and test reports. However, as Screwdriver usage continued to climb, it outgrew a single Jenkins cluster. So in order to grow to our needs, we added capabilities in Screwdriver that allowed us to scale horizontally while also adding capabilities to schedule pipelines across a number of Jenkins clusters. As we scaled Screwdriver, we used less from Jenkins and built more supporting services utilizing our cloud infrastructure. The open-source version is focused on Kubernetes and Docker Swarm as our primary supported execution engines.
In the coming months we will expand our offering to match many of the features we have internally, including:
Mechanism to store structured build data for later use (last deployed version, test coverage, etc.)
Built-in metric collecting
System-wide templates to enable getting started quickly
Log analysis to provide insights to developers
Please join us on the path to making Continuous Delivery easy. Visit http://screwdriver.cd to get started.

I use this every single day. Now you can too!

My niece is learning.

Sat, 26 Nov 2016 11:13:39 -0500

My niece is learning.

Empowering the Minds of Tomorrow: Yahoo Makes Large Donation to Hampton University

Fri, 28 Oct 2016 14:29:56 -0400

lifeatyahoo:

By LaShanti Jenkins, Snr. Talent Acquisition Mgr.

‘Empowering the Mind of Tomorrow’ are the words that now sit inside of a window in Hampton University’s Computer Science (CS) department in Virginia. Yahoo’s Office of Inclusive Diversity sponsored the branding behind a significant donation of 80 servers to the school.

To celebrate the donation, Yahoo’s University Recruiting hosted a series of events where students got the opportunity to speak with Yahoo engineers, present their resumes, receive tips on interviewing in the tech space and explore options for internships and full-time employment at Yahoo. Additionally, Yahoo’s Jeffrey Scholz facilitated a tech talk to introduce students to the Yahoo Video Guide and the technology behind the app, including how it was developed at scale.

A big thank you to Yahoo’s LaShanti Jenkins, also a Hampton University alum, for making this happen!

I was a part of this. So proud.

icphoto: Garry Winogrand’s photographs are sophisticated,...

Sun, 28 Aug 2016 01:18:34 -0400

icphoto:

Garry Winogrand’s photographs are sophisticated, chance observations of daily life that demonstrate his mastery of the 35-millimeter camera.
📷Garry Winogrand, 1981 http://buff.ly/2bsp8Js

icphoto: #WednesdayWisdom from Alfred Eisenstaedt: “When I have...

Wed, 24 Aug 2016 22:00:24 -0400

icphoto:

#WednesdayWisdom from Alfred Eisenstaedt: “When I have a camera in my hand, I know no fear.”

life: From the March 23, 1959 feature article: HAWAII—BEAUTY,...

Sun, 21 Aug 2016 12:26:10 -0400

life:

From the March 23, 1959 feature article: HAWAII—BEAUTY, WEALTH, AMIABLE PEOPLE - After long years of trying, the idyllic islands at last stand on the brink of statehood. Pictured here: “Cattle graze under volcanic cliffs on Oahu Island. Tunnels through cliffs link back country to Honolulu.” (Ralph Crane—The LIFE Picture Collection/Getty Images) #1950s #Oahu #Hawaii

leicanews: Anémones du Japon.Leica II (D) Couplex, 50 mm Elmar,...

Mon, 15 Aug 2016 07:50:41 -0400

leicanews:

Anémones du Japon.
Leica II (D) Couplex, 50 mm Elmar, Kodak Plus-x.
Photo by Stéphane Marco‎ in Leica Screw Group.

Yahoo Sports Introduces Your Guide to Live Sports

Fri, 12 Aug 2016 13:24:25 -0400

yahoo:

By Parker Emmott, Director, Product Management

Although the sports we love haven’t changed, the way we watch them definitely has. We are excited to say that the days of scouring the internet to find where your live sports are streaming are a thing of the past. Starting today with MLB, our completely redesigned Yahoo Sports app now features a guide to find and stream live baseball games.

Now you can find streaming options available for a specific baseball game, and we’ll link you directly to the right app for your viewing pleasure! We manage the complex rights and blackout restrictions, so we only show options available to you.

Wondering how it works?

When you’re on the Schedule page, a ‘Watch’ button will now be displayed alongside games with streaming options.

Simply tap through to the ‘Game Details’ page, and tap the ‘Watch Now’ button to see your streaming options.

And if you see that a game is “Live on Yahoo!”, like our MLB Free Game of the Day, you can watch the game directly on the ‘Game Details’ page.

Make sure to update your Yahoo Sports app today to take advantage of this new live sports guide, or download Yahoo Sports for iOS and Android if you haven’t already! And if you find yourself wishing a similar guide existed for your favorite streaming TV shows and movies, you can download Yahoo Video Guide for iOS and Android. Happy streaming!

My team of crack Video engineers made this happen. First big release for some of them!

tamarkincamera: A very very early and all original Leica I...

Wed, 10 Aug 2016 21:39:28 -0400

tamarkincamera:

A very very early and all original Leica I (model A) with unusual original leatherette, ratchet mechanism and early fittings! The earliest one we’ve seen in a while #1817 made in 1926! #nofilter #raritydujour #rareasscamera #tamarkincamera #tamarkinrarecameraauction #leica #leicacamera #leicacraft #leica_world (at Tamarkin Camera)

Very tempted to add a Leica screwmount without a built-in rangefinder (like this one here) to my collection.

leicacraft: A top view of a very special one of one M for...

Wed, 04 May 2016 18:19:53 -0400

leicacraft:

A top view of a very special one of one M for @kantocamera #leicacraft #kameracraft #leica #leicacamera #camera #rangefinder #cameraporn

Having taken apart cameras and lenses and usually not being able to figure out how to put them back together, I appreciate a company like Kanto and the incredible service they provide to photographers.

japancamerahunter: Desert Camo Leica M3 Betriebsk custom. This...

Wed, 10 Feb 2016 22:40:33 -0500

japancamerahunter:

Desert Camo Leica M3 Betriebsk custom. This was a very special customer order. We worked together to make this camera completely unique. #leica #leicacraft #lensporn #camera #cameraporn #filmcamera #shootfilm #leicacamera (at Musashino, Tokyo)

Such a beautiful customization. I may do a lighter-shade repaint one day.

Controlling audio output on iOS with AVAudioSession

Wed, 02 Dec 2015 11:12:21 -0500

yahooeng:

Nano Anderson, Engineering Mgr @ Yahoo Video NYC

If your app plays sound of any kind, you have probably asked yourself “why is/isn’t sound playing now?”. You toggle the ringer switch, which sometimes solves the problem. But sometimes it doesn’t. You copy and paste some code from Stack Overflow which overrides the ringer switch permanently, but then sound starts playing in the background while the phone’s in your pocket. Why?

If you end up in Apple’s iOS documentation and sample code, you’ll probably run into a wall. In most of Apple’s examples, you’re not told that your AVPlayer or MPMoviePlayerViewController or whatever you use simply will not play audio if the ringer is muted. Meanwhile, in Messages.app, Videos.app, and Apple’s other built-in apps, you will commonly see that audio and video, when initiated by the user by tapping a play button or similar action item, immediately play back with sound, regardless of whether the ringer is muted. Apple’s defaults for developers do not address this use case, but AVAudioSession is hardly mentioned in general audiovisual playback documentation¹.

The answer lies deep within the bowels of AVAudioSession. This AVFoundation class is a singleton for your app which controls every aspect of audio input and output.

The trickiness with AVAudioSession is that you that you don’t have to use AVAudioSession in order to play video or audio. Under most circumstances, your app will play a media file through headphones (regardless of ringer switch setting) or speakers (if the ringer is not set to Silent). When you eventually have the question “How do I play audio through the speaker when the ringer is muted?”, you’re most likely off to the internet to find an answer.

And if you find yourself in some other corner of the internet, this is the code you’ll likely be told to just shove in your UIApplicationDelegate:

This works, until you realize you then have to deal with rightfully upset customers who had their favorite podcast muted the second they opened up your news app, which happens to maybe occasionally play videos, among other side effects you weren’t ready for.

If you’re confused, you are not alone. Let’s figure this out together.

First, let’s look at that method that the internet told us to use to override the ringer switch, -[AVAudioSession setCategory:error:]. Here’s a handy table that explains what each combination of AVAudioSessionCategory (which you control) and “Ring/Silent switch” setting (controlled by the user) will result in.

AVAudioSession categories!

Silent Switch ON Silent Switch OFF
Ambient Plays sound combined with any background sounds being played by other apps. No sound from your app.
SoloAmbient (DEFAULT) Plays your sound and stops sound from other apps. No sound from your app.
Playback Plays your sound and stops sound from other apps. Can mix audio with sound from other apps, but does not by default.² Plays your sound and stops sound from other apps. Can mix audio with sound from other apps, but does not by default.²
Record No sound from your app. No sound from your app.
PlayAndRecord Plays your sound and stops sound from other apps. Can mix audio with sound from other apps, but does not by default.² Plays your sound and stops sound from other apps. Can mix audio with sound from other apps, but does not by default.²
AudioProcessing No sound from your app. No sound from your app.
MultiRoute Plays your sound and stops sound from other apps. Plays your sound and stops sound from other apps.
Most apps only need to deal with SoloAmbient as their default category and one other category, such as Playback, for playing back multimedia files. With that in mind, a simple AVAudioSession manager class can help toggle back and forth between categories. Here is an example of how to one could call -[AVAudioSession setCategory:error:] (or -[AVAudioSession setCategory:withOptions:error:]):

That’s the gist of it. But don’t worry, there are a few things you still need to worry about.

Does your app need to continue playback in the background? Perhaps you have a podcast app, or an app which wants to take advantage of Airplay or Picture-in-Picture? Well you’re in luck, because now you get to muck with UIBackgroundModes in your Info.plist!

Background modes!

Now, this isn’t really a big deal in terms of code. In your Info.plist, within the array for UIBackgroundModes (add it if it doesn’t already exist), add an entry with the string audio.

If you’re only playing audio files, you’re basically done³. If you’re playing video files, there’s more fun involved. Why would you want to continue playing videos in the background, you ask? Two words: Airplay & Picture-in-Picture. For now, I’m only going to cover Airplay, because PiP is less prone to side-effects in my opinion.

There’s one main item you have to worry about with backgrounded apps and video. Video-playing apps need to disconnect the currently-playing AVPlayerLayer from its AVPlayer by setting its player property to nil before your application goes to the background. If you don’t do this, your AVPlayer will automatically pause. That’s a good thing most of the time, but not if you want your Airplay-ing video to keep playing! Just make sure you’re only doing this for videos that are Airplaying, otherwise you’ll get some ridiculous audio coming out of your phone’s speakers after you lock the screen. Here is some slightly modified code from Apple’s Technical Q&A QA1668 – Playing media while in the background using AV Foundation on iOS:

Thanks for tuning in to this week’s episode of Controlling audio output on iOS with AVAudioSession. Tune in next week, when we’ll be talking about literally anything else.

Further reading and sources (which for my purposes are interchangeable)

Apple Technical Q&A QA1668: Playing media while in the background using AV Foundation on iOS

Apple Audio Session Programming Guide

Lynda.com’s Introducing AVAudioSession

AVAudioSession Class Reference

AVPlayer Class Reference

Meanwhile, talk to an iOS audiovisual recording & mixing engineer and you’ll probably find an AVAudioSession expert. ↩

See AVAudioSessionCategoryOptionMixWithOthers ↩

If you want to play well with other apps’ audio, you’ll want to look at the advanced features of AVAudioSession in Apple’s Audio Session Programming Guide ↩

I wrote this over on the Yahoo Engineering Tumblr. My first-ever post for them.

"Repairing a Wobbly Voigtlander Ultron 35mm f1.7 Leica Screwmount Lens"

Sat, 07 Nov 2015 11:12:23 -0500

“Repairing a Wobbly Voigtlander Ultron 35mm f1.7 Leica Screwmount Lens”

- I am reposting the meat of the instructions from Charles Barcellona’s photo.net post about repairing his Voigtlander Ultron w/ a wobbly front section. I had the same wobble, and focus was off all the time. Fixing the wobble fixed the focus.

Mainly, I just want to make sure that these instructions are never lost. Anyways, here they are:

Let me tell you folks…. I dont have the eyes I once had, but still hands to do surface mount repair in my shop (using magnifiers of course). If you have one of these lenses, you can fix it pretty easily, once you know how.

Here’s how. Read the instructions fully before doing this or you may lose the detent ball for the aperture ring. Tools needed are a piece of rubber tape or rolled up framers tape or other tacky material, and a Craftsman 45727 precision screwdriver.

Remove hood and filter if present.

Use some rubber to “grab” the nameplate ring and spin it off counter clockwise.

Undo the three screws that hold the filter holder ring on. BE CAREFUL AT THIS POINT and DO NOT REMOVE the aperture ring accidentally.

Now… over a felt or velvet placemat - carefully ease the aperture ring off the front. The ball will be at the 4 o'clock position as viewed from the front of the lens. Make sure you watch where that ball goes. You can get a spare, but don’t lose it and you won’t need to worry about it. Be careful also that you don’t lose the spring in the hole. Its a greased spring, and is not likely to get lost, unless you are careless with the next part you’re about to remove.

Next, look for the three larger holes in the ring surrounding the front element. Deep in those holes are screws (same screwdriver). The screws are held with locking compound. You’ll need to make sure the screwdriver is seated well, and is not slipping, and… ease each of those three screws out.

Lift off that ring, remembering to take some care so that the spring is not lost, and you’ll see the heads of the screws that are loose. Tighten those, using a bit of locking compount (NOT LOCKTITE!!). I’ve got some special stuff we use on circuit boards that is disolvable easily, but nailpolish, or a small dab of Testors enamel will do well.

Reassemble in reverse order, making sure when you put the inner ring back on, you get the hole for the ball, with the spring in it, at approximately 4 o'clock. The remaining locking compound on the screws will make them a snug fit when you seat them home. Thats a good thing, just don’t let the screwdriver slip. When you ease the aperture ring back on, put the ball in the hole and lower the detent side of the ring over the ball first so its captive, then lower the other side of the ring while giving a gentle twist to find and engage the aperture arm. Once that’s on, breath easy you’re home. Put on the filter retainer ring (no alignment there), the nameplate and you’re finished. Clean the front element, replace the filter and put the hood on… then go shoot.

You don’t really remove any optics, so there’s no re-alignment to be done. The optics section sits flange on flange, so it goes where it goes, and that’s that.

Love this color on hard goods.

Tue, 06 Oct 2015 15:02:28 -0400

Love this color on hard goods.

leicacraft: Nice duo via @jrphoto88 #leicacraft #leica...

Sat, 26 Sep 2015 12:54:43 -0400

leicacraft:

Nice duo via @jrphoto88 #leicacraft #leica #cameraporn #rangefinder #35mm

I never thought I’d like an all white camera, but this one looks beautiful.

ilovemyleica: (via Masters of Photography: Josef Koudelka)...

Wed, 09 Sep 2015 11:22:09 -0400

ilovemyleica:

(via Masters of Photography: Josef Koudelka)

You don’t have to have a perfectly level horizon to make beautiful photos.

Photo

Sun, 26 Jul 2015 09:58:50 -0400

Investment Diary - Buy HRTG

Thu, 04 Jun 2015 09:33:20 -0400

I like well-run insurance companies. I’ve bought AIG in the past (and continue to hold some for a family member’s portfolio), and I’m buying this in my IRA as a long-term hold. Not a trade. I’ll have a stop loss in case I’ve made a shit move, but I’m confident that is unlikely.

Weekly HRTG:

Daily chart:

Investment Diary - Cover FB

Wed, 29 Apr 2015 10:33:49 -0400

Following my short of Facebook last week following their Estimize miss, I sold at nearly the end of the day yesterday, 3 business days later. After commissions (can’t short on Robinhood yet, bummer), I made a 1.5% profit. Without commissions, I’d have been closer to 2%, not bad for a 3-day trade. I’ll have to run similar trades to gauge the effectiveness personally, but this first trade stood up to the hypothesis.

Investment Diary - Short FB

Thu, 23 Apr 2015 11:44:27 -0400

Facebook posted lower-than-expected earnings and revenue for their first quarter of 2015. I’m taking a shot with the Estimize post-earnings drift strategy. Essentially, I’m betting that over the next 3 days, Facebook stock will drift downwards. I’ll be covering the short at the end of the day next Monday. I shorted at $83.62/share.

	Silent Switch ON	Silent Switch OFF
Ambient	Plays sound combined with any background sounds being played by other apps.	No sound from your app.
SoloAmbient (DEFAULT)	Plays your sound and stops sound from other apps.	No sound from your app.
Playback	Plays your sound and stops sound from other apps. Can mix audio with sound from other apps, but does not by default.²	Plays your sound and stops sound from other apps. Can mix audio with sound from other apps, but does not by default.²
Record	No sound from your app.	No sound from your app.
PlayAndRecord	Plays your sound and stops sound from other apps. Can mix audio with sound from other apps, but does not by default.²	Plays your sound and stops sound from other apps. Can mix audio with sound from other apps, but does not by default.²
AudioProcessing	No sound from your app.	No sound from your app.
MultiRoute	Plays your sound and stops sound from other apps.	Plays your sound and stops sound from other apps.