Simon Weber

Opsgrid

2020-09-20T00:00:00-04:00

Opsgrid

September 20 2020

I’ve built up a number of side projects over the years. The ones that require a backend are hosted on individual vms from BuyVM, which means I need to monitor about a half-dozen servers.

While there’s plenty of free Pingdom-style uptime monitoring options out there, cheap options to monitor metrics like cpu, ram, and disk are more rare. The big names usually have a free tier allowing one or two hosts with minimal data retention, then a big jump to a $10+/month per host paid tier. I can’t justify spending that when I only pay $2/month for the vms themselves! There’s also some smaller independent options out there, but they tend to require custom server agents and aren’t all that much cheaper. So, I decided to build my own.

I didn’t need anything fancy, just support for the usual metrics, user-defined alerting, and, ideally, reasonable retention of past data so I could identify trends. Like my other projects, it also needed to be operationally simple.

Opsgrid is the result. It allows free server/application alerting with about 1 year of data retention per host. The trick is that Opsgrid doesn’t actually store historical metrics: instead, users connect a Google account, and it pushes data as rows to a Google Sheet per host. It also makes use of Telegraf, an open-source monitoring agent, to avoid dealing with metric collection code.

Check it out if you also run your own servers! I plan to keep it free unless it attracts a significant number of users, but even then I should be able to price it about 5-10x lower than offerings like New Relic or Datadog.

Side project income 2019

2020-01-16T00:00:00-05:00

Side project income 2019

January 15 2020

2019 was my fourth year running my own small software business. I also quit my job in the spring! I’m now self-employed but spend most of my time consulting, so I still consider the business to be “on the side”. Here’s my active projects as of the end of the year:

Autoplaylists for Google Music: iTunes Smart Playlists for Google Music.
Autoresponder for Google Hangouts: vacation/out-of-office replies for Google Chat.
Kleroteria: an email lottery and spiritual successor to The Listserve.
Repominder: release reminders for open source projects.
NYC Park Alerts: closure notifications about NYC Parks locations.
Plugserv: an ad server for my own projects.

Financials

This was my first down year:

annual revenue decreased from $3600 to $3330
annual expenses grew slightly from $240 to $280
monthly recurring revenue dropped substantially from $350 to $250

Autoplaylists and Autoresponder - which make up nearly all my revenue - both extend Google products that are being shut down, so I had actually expected more of a decline. While I began diversifying in 2018, those efforts haven’t yet impacted the bottom line.

New Projects

I added two smaller projects last year. NYC Park Alerts was inspired by my gym habits, of all things. I work out at a local rec center and showed up a few times to find it unexpectedly closed. Looking into it, I found that the city posts updates online but with no way to subscribe to them. So, I set up a scraper that lets people receive emails when locations of their choice are affected. It’s got one user (besides me) and is more of a community service, so I don’t expect to ever make money off it.

My second new project was Plugserv. It's a tiny ad server that helps me shamelessly plug my projects across my own sites. Here's a live example of what it looks like: "" Behind the scenes that copy is being retrieved from the Plugserv api, which handles rotation and tracking. It's performed better than I expected: in the six months it's been active it's served ~25k ads with a click through rate of over 1%, which beats most display ad benchmarks. Despite working well for me it hasn't picked up any other users; go check it out if you'd like to run your own ads on your own sites!

Existing Projects

2019 was also the first year I stopped running any projects. I’m usually content to let projects run despite low usage because my expenses are so low. But, I made an exception for Analytics for Google Payments: despite being free to run and having potential with business customers, it was painful to maintain and targeted a much smaller market than I expected.

MinMaxMeals, my cooking site, was more of a case of lost interest. Its recipes were heavily optimized for time, which isn’t as important to me now that I’m self-employed and work from home. It’s still a fun talking point, though, and I might return to it someday.

My remaining projects both grew slowly without my involvement. Repominder picked up a notable open source power user and Kleroteria added about 500 subscribers.

Though I didn’t do much marketing or feature work, I did spend some time on operational improvements. I consolidated my sundry virtual servers into a fleet of $2/month instances at BuyVM. Each runs a new NixOS+docker setup that’s generic enough to be cloned for new projects. I also upgraded everything to Python 3 and worked out a better way to route my custom-domain emails into Gmail for cheap (which I’ll write up eventually).

Reflection

2019 had its ups and downs, but now that I’m settled into my new lifestyle I’m feeling ready for 2020. I don’t expect it to be easy, though: Autoplaylists and Autoresponder are likely to shut down completely, and there’s a real chance I’ll find myself in the red.

So, hopefully my upcoming projects can prevent this. One of these - hosted server/application monitoring with unusually low operating costs - should launch soon, so sign up below if you’d like to get notified about it!

As always, thanks for following along and feel free to reach out with questions. If you’d like to read my previous annual summaries, they’re available here.

Plugserv

2019-08-03T00:00:00-04:00

Plugserv

August 3 2019

I’ve been creating websites for a while now. Some of these get consistent visitors, while some are nearly unknown. At some point I realized there was an opportunity here: my popular sites could advertise my other sites. So, I started manually adding small text ads to each.

This quickly became unwieldy. I’m closing in on ten active projects, and I got sick of updating all of them whenever I launched something new. I decided my next project would solve this problem.

I call it Plugserv, since it lets me shamelessly plug my websites. It’s an open source ad server: I configure it with all my ads and add a javascript snippet to each site. Plugserv then rotates my ads automatically and collects basic performance metrics. Here’s a live example of what this looks like:

Leaving Venmo after five years

2019-04-26T00:00:00-04:00

Leaving Venmo after five years

April 26 2019

I left Venmo recently, after an eventful five and a half year stay. Over that time we increased quarterly payment volume by over 100x, grew from dozens to hundreds of employees, and were both acquired and spun off. I’d grown too: with the fourth longest tenure at Venmo, I was now leading a team and part of engineering-wide discussions.

Unsurprisingly, with increased responsibilities came increased stress. Upper management turnover was especially tough, as I found myself fighting the same political battles under each regime. My enthusiasm turned to jadedness, then cynicism, then anxiety. Eventually, despite a supportive manager and a sabbatical, I became concerned for my health and decided to make a change.

I’m now officially working for myself. After taking some time to recover, I plan to focus mostly on my own software - which I suppose aren’t “side projects” anymore - and occasionally pick up consulting work. I also look forward to having time for cooking, music, and engaging with the ever-growing Recurse Center community.

Sign up below if you’d like to follow what I’m up to. Or, shoot me an email if you’re interested in part-time help, particularly around backend scaling, CI/CD, or python.

NYC Park Alerts

2019-03-30T00:00:00-04:00

NYC Park Alerts

March 30 2019

I work out at a city rec center, largely because the price - about $10/month - is hard to beat.

Schedule consistency is a downside, however. Due to events and maintenance I’ve sometimes shown up to find my location unexpectedly closed.

After some research, I found that the city does a pretty good job of posting schedule changes online. But, since there’s no way to subscribe to them, you need to check every time before visiting.

So, I built NYC Park Alerts. It lets you subscribe to locations of your choice, and will email you when updates are posted. Go sign up at parks.simon.codes if you’re also a rec center fan!

Side project income 2018

2019-01-07T00:00:00-05:00

Side project income 2018

January 7 2019

2018 marked my third year running a software business. While last year was focused on growing existing products, this year was more about diversification. I’m now running six separate projects:

Analytics for Google Payments: business metrics for Google merchants.
Autoplaylists for Google Music: iTunes Smart Playlists for Google Music.
Autoresponder for Google Chat and Hangouts: vacation/out-of-office replies for Google Chat.
Kleroteria: an email lottery and spiritual successor to The Listserve.
MinMaxMeals: cheap, healthy, and fast cooking with no regard for convention.
Repominder: release reminders for open source projects.

Financials

Autoplaylists and Autoresponder continued to be my only revenue sources. Here’s the numbers:

annual revenue increased from $2000 to $3600
expenses dropped slightly from $260 to $240
revenue run rate grew from $250 to $350, a slight slowdown in growth
monthly net subscribers grew from 10 to 15, then dropped under 5

The subscriber numbers were mostly due to increased Autoplaylists churn once Google Music’s shutdown was announced. Autoresponder growth was steady, but not enough to compensate.

To reduce my exposure to future Google changes, I worked on two new independent projects.

New Projects

Over the summer I launched Kleroteria in response to the shutdown of The Listserve. Both work the same way: people subscribe to an email list, and periodically one person is chosen to write to everyone else. My take mostly differs behind the scenes, with improved automation and free hosting (it runs on 13 free tiers, which was a fun technical exercise).

I was hoping to take over The Listserve’s 10k subscribers, but they had already deleted their email list by the time I reached out. Instead I settled for cold-emailing previous winners who’d provided their emails. This yielded about 500 subscribers from 2000 prospects. While this is a decent start, there’s no real revenue potential without much more growth. So, I hope to do some marketing in 2019, then look into options like Patreon or merchandise.

My second project was a bit of a departure: MinMaxMeals is my first content site. It launched a month ago, and has picked up a few dozen email subscribers and twitter followers so far. I plan to put consistent effort into writing and marketing over the next year. There’s a variety of revenue options if it gets traction, like affiliate links, sponsorships, or a cookbook.

Existing Projects

Autoplaylists and Autoresponder each received maintenance and some minor feature work. Notably, this included handling a small crisis while at a friend’s wedding. I also attempted a few Facebook ad campaigns to little success.

On the positive side, I made good progress on Analytics for Google Payments. A reverse engineering breakthrough led to support for new data and a public beta. It’s now got its first 10 users, largely from a cold-email campaign I ran against owners of paid Chrome extensions. That also had the side effect of sizing the market: as it turns out, there a very few paid chrome extensions. So, I may look into supporting other merchant types (like Android or YouTube) to expand it. I’m a bit hesitant given its coupling to Google, however.

Finally, poor Repominder got little attention aside from a move to free hosting. I’m not sure it’s worth investing in now that GitHub seems to have stepped up development of similar features.

Reflection

I’m pleased to have diversified in 2018, even if my new projects aren’t yet making money. I’m also proud of how much I got done – especially as my responsibilities at my day job increased.

Here’s hoping 2019 is similarly successful! Follow along by signing up below, and feel free to reach out with questions.

MinMaxMeals: cheap, healthy, and fast cooking

2018-12-03T00:00:00-05:00

MinMaxMeals: cheap, healthy, and fast cooking

December 3 2018

In a departure from my typical projects, I’ve launched a cooking site! MinMaxMeals disregards culinary convention in favor of cost, health, and speed – typical meals call for $1 of ingredients and a few minutes in a microwave.

Here’s a bit on its origins from the about page:

While today I cook this way because it’s practical, it was originally out of necessity. I had suddenly become ill, and was put on a restricted diet to figure out what foods I could tolerate. Prepping all my own meals lead to my focus on speed, cost, and nutrition, while the limited ingredients ruled out most familiar dishes. As my health returned I added more ingredients, and later - after my cooking became a conversation starter - I decided to share what I’d learned.

If this sounds intriguing, check out my recipe for oatmeal that tastes like garlic bread. Or, read about the different types of lentils and how they’re often cheaper under different names.

Running Kleroteria for free by (ab)using free tiers

2018-07-09T00:00:00-04:00

Running Kleroteria for free by (ab)using free tiers

July 9 2018

Kleroteria is an email lottery: periodically, a random subscriber is chosen to write to everyone else. I built it to replace The Listserve, which recently shut down after providing years of advice, hundreds of personal stories, and one impromptu picnic. It was a fun and personal corner of the internet that I missed immediately.

I suspect the creators of The Listserve just got tired of running it. First, it was high maintenance: the lottery and post sending was done by hand. Second, it was expensive: it was powered by MailChimp, which would have cost hundreds of dollars per month given their tens of thousands of subscribers.

I’d need to address these in my replacement. I figured I could easily run it for little more than the cost of sending emails. But could I host the entire thing for free? I wasn’t sure, and figuring it out quickly became a personal challenge.

After far too long digging through pricing docs, I came up with a serverless setup run on AWS’s indefinitely free tier. It’s illustrative of just how much Amazon gives away, but also how quickly a serverless architecture can get out of hand.

This post lays out my ridiculous thirteen-service setup to inspire future free tier shenanigans.

Infrastructure shopping

My other side projects are run unceremoniously on cheap virtual private servers. Autoresponder and Repominder, for example, run on <$1/month deals from lowendbox. A comparable free offering is Google’s f1-micro instance, though I was concerned about bottlenecking on vcpu during signup spikes.

So, I started looking at serverless options since the free tiers seemed generous. GCP looked promising, but Google’s free email offering was limited to a few thousand sends/month. I considered Azure as well, but was turned off by the lack of a free hosted database. AWS ended up providing roughly the same pieces as Google, but with the benefit of 62k free sends/month through SES.

Things were starting to take shape. I could use Lambda for the backend, DynamoDB for storage (RDS is only free for a year), and SES for email. Most of the service limits weren’t an issue - I’ll never need 25GB of storage - but Dynamo throughput was a concern. It’s measured in “capacity units” which boil down to about 25 writes/second and 50 eventually-consistent reads/second. I needed to make sure I could stay under these limits.

Dynamo math

Kleroteria stores two things: posts and subscribers. I decided to use random ids to prevent hot partitions. The full schemas ended up looking like this:

pending_posts table:
- key: randomly-generated id
- values: contents and status
subscriber table:
- key: randomly-generated id
- values: email address

Post contents are capped such that their rows are less than 4kb. Subscriber rows, on the other hand, are only ~32b on average. These row sizes are important, since capacity units are measured in 1kb chunks for writes and 4kb chunks for reads.

For example, consider sending out a post. This requires a full scan of the subscriber table. With a table of 10k rows, each under 32b, read in perfect 4kb chunks, a sustained scan at one read capacity unit would require about 40 seconds. Using 5 rcus, it’d take ~8 seconds. Similarly, 100k subscribers could be handled in ~20 seconds with 20 rcus.

Selecting a lottery winner is roughly the same thing: a full scan, then random selection. Originally, I avoided the scan by using a composite primary key. This involved a two-level index - partitioned by the first character of the id and sorted by the full id - and could select a random item in under two queries. It turned out not to be worth the trouble, though, since its distribution was biased and the AWS limit of 25 rcus allows for fast enough scans.

The other operations are more straightforward. Subscribing and unsubscribing, for example, cost 1 wcu each. Processed synchronously, though, this could cause an issue: beyond 25 signups/second I’d start getting throttled (or worse, charged). I needed a way to spread out operations during spikes.

Throttling under service limits

Kleroteria has the luxury of being entirely asynchronous, so I connected everything with SQS queues. This allows precise throttling by adjusting the queue polling rate. For example, if subscriptions were sent through a queue consumed at 1 message/second, it’d never exceed 1 wcu.

Dynamo’s capacity units aren’t the only limit in play, though. SQS and Lambda both allow only one million free requests/month. Lambda also constrains runtime and memory usage, and SES has a sending rate limit. To fit within all of these, I’m currently running:

1 scheduled lambda execution / minute
30 seconds / lambda execution
10 messages / SQS poll
10 second wait / SQS poll

This puts me at ~40k executions/month, ~120k polls/month, ~1 wcu, and ~1 email/second, which is pretty conservative. I plan to increase the message limit once it’s been live for a while.

Putting it all together, the pieces involved in subscription handling look like this:

      browser (AWS sdk + Cognito) --> SQS                     
                                       |                      
                                       v                      
      CloudWatch events -----------> Lambda --> Dynamo and SES

Note that the frontend enqueues directly to SQS. This avoids API Gateway (which isn’t free) and per-request Lambda executions. It comes with a cost, though: javascript must be enabled to sign up, and I can’t give users feedback from serverside validation. I’m hoping this doesn’t matter too much in practice.

Everything else

Astute readers will count six services so far, while I promised thirteen. The seventh is IAM, which is used for internal access control. It’s always free since it’s an essential part of AWS.

The remaining six services aren’t from AWS. One is netlify, used to host the frontend assets. I’ve found it comparable to GitHub Pages - my usual choice - though I’ve noticed surprisingly flaky uptime according to New Relic Synthetics, my external monitoring tool.

The remaining services play supporting roles and don’t require much commentary:

Sentry: error reporting (frontend + lambda)
Google Analytics: frontend analytics
BitBucket: private git repo
CloudFlare: dns (with cdn disabled, since netlify provides one)

Whew

So, there you have it: thirteen services, tens of thousands of subscribers, $0 per month. Was it worth it? Probably not, at least in the context of a side project. Reading pricing docs, planning Dynamo capacity, and setting up a local environment added days to what should have been a weekend project. That said, it was a fun challenge and the result is more robust than my usual vps setups.

Go join Kleroteria so I can justify my effort! If you’re interested in hearing more about it - or getting notified if I open source it - consider subscribing with the links below.

Analytics for Google Payments: now in public beta

2018-03-21T00:00:00-04:00

Analytics for Google Payments: now in public beta

March 21 2018

Analytics for Google Payments, a Chrome Extension to help Google merchants make sense of their business metrics, is now in public beta.

To find out more and try it out, head to the new product site at analytics.simon.codes!

Autoplaylists 2017 user survey results

2018-03-11T00:00:00-05:00

Autoplaylists 2017 user survey results

March 11 2018

In early 2018 I sent out a user survey for Autoplaylists for Google Music. My goal was to get a general idea of user satisfaction, as well as figure out what I should focus on next. Here’s a quick summary of the results:

~30% response rate from mailing list, representing ~5% of daily actives
~6:4 representation of free vs paying users
on average, users were likely to recommend it (NPS of 15)
on average, users found it reliable (4+ on 5-point scale of “never works” to “almost always works”)
common themes in feedback were:
- syncing is opaque
- complex autoplaylist definitions are difficult to manage

I don’t want to read too much into this - especially with a low response rate and over-representation of paying users - but in general I’m pleased with the results. The reliability data is particularly relieving, since I had my doubts going by my user support work.

I’m hoping to address both the common pain points this year by:

adding a syncing dashboard with past syncs and their results
adding a way to share common parts of playlists definitions

So, here’s to a successful 2018! I’m excited to run a similar survey next year now that I have results to compare against.

Analytics for the Google Payments Center: accounting data now supported

2018-01-26T00:00:00-05:00

Analytics for the Google Payments Center: accounting data now supported

January 26 2018

tl;dr install the extension here.

Last year I launched the beta of Analytics for the Google Payments, a Chrome Extension to help Google merchants make sense of their business metrics. Today I’m announcing the first major change in a while: full support for accounting data.

Basically, Analytics used to only support order data. This was useful for tracking subscription metrics like net new subscribers or churn, but didn’t help with accounting. Now, Analytics has access to every transaction, earnings report and VAT invoice, meaning it can answer questions like:

what’s my growth in profit over time?
how much sales tax did I collect in the last reporting period?
how much am I paying in Google fees?

Here’s an example graph:

A screenshot of the new earnings graph. Each bar is a monthly breakdown with the trend line showing net growth.

I’m also looking into supporting bulk exports of this new data. This would combine monthly documents like earnings reports for download all at once, rather than one at a time.

Side project income 2017

2018-01-09T00:00:00-05:00

Side project income 2017

January 9 2018

2017 was my second year of running a small software business. Unlike my first year - which you should read about first if you haven’t already - I didn’t launch any new paid products. Instead, I spent most of my time experimenting with my existing products: Autoplaylists for Google Music and Autoresponder for Google Chat.

In this post I’ll go over what I worked on and how it turned out.

Financials

Here’s an overview of the business on paper:

annual revenue increased from $600 to $2000
annual expenses decreased slightly from $300 to $260
monthly revenue run rate increased from $100 to $250
monthly net subscribers increased from 5 to 10
monthly cancelled/churned subscriptions also increased from 5 to 10

Out of these, the churn rate stands out. I’ve already started collecting feedback from cancelled Autoresponder users, but have yet to put similar effort into Autoplaylists.

Product Development

Part of my coding effort went into two new yet-to-be-monetized projects.

The first is Analytics for Google Payments, which helps merchants calculate business metrics like the ones I presented earlier. Motivated by my frustration getting that data for Autoplaylists, it’s also similar behind the scenes: both are Chrome Extensions using reverse-engineered apis. While the market is crowded - Baremetrics and ChartMogul come to mind - nobody has bothered with Google Payments because of the lack of official apis. Hopefully that continues to be the case. If Google launches an api for that data, I’ll probably just let the big players fight it out.

My second new project is Repominder. It’s a SaaS that notifies forgetful open source maintainers (like me) when a release is needed. This has a less obvious path to profit than Analytics, but may have potential if marketed towards software businesses.

Outside of my new projects my programming time went towards maintenance. Based on my analytics data and customer support interactions I chose to focus on features for Autoresponder and reliability for Autoplaylists.

Growth and Other Work

One of my major growth goals was to set up mailing lists. I’m pleased to have achieved that: I now have one for each product (and this blog!) combining for a reach of 500+ and 1+ per day growth. They’re used for announcements right now, but I may experiment with drip campaigns in the future.

Along with a mailing list, Autoplaylists got a payment plan tweak late in the year. It now includes a one-week free trial in addition to the existing one-autoplaylist freemium model. My goal was to give users a chance to try features that only work with multiple autoplaylists. While I haven’t noticed an effect on the bottom line yet, new user engagement has gone up since.

Autoplaylists also saw my first experiment with paid advertising, after an ad blocker mishap led me to realize that Google Music serves desktop text ads. This was about as targeted as possible, so I had high hopes. One month and plenty of AdWords fiddling later I had generated about 20k impressions, 40 clicks, and 4 installs. With the average cost per click at $2.31, this meant the cost of a new user was over $20. Even assuming 100% conversion this was already above my customer lifetime value, so I stopped the campaign.

In another marketing attempt I tried for earned media coverage. Unfortunately, unlike in 2016 I didn’t hear back from anyone I reached out to. I don’t blame them: with no launch announcements I lacked a solid pitch.

My last bit of notable business work was when I noticed an idea of mine had already been built. It seemed abandoned, so I contacted the owner to ask about buying it. This didn’t pan out either, as they had plans to revitalize it. Still, I feel good about the time spent researching and getting advice, since I expect to be in this situation again someday (maybe even as a seller).

Looking Forward

Overall, I’m pleased with what I got done in 2017. My biggest disappointment was the lack of a new paid product. However, I’m set up to address that next year with my untapped projects and new product ideas.

I’m also eager to pursue businesses as customers next year. Autoresponder is particularly ripe for this, since I’ve noticed a user pattern of a) small business owners and b) employees of larger companies. I haven’t reached out to them yet, but I suspect they’d be willing to pay more for added features.

So, here’s to a successful 2018! Follow along by signing up below, and feel free to reach out with questions.

Repominder

2017-12-09T00:00:00-05:00

Repominder

December 12 2017

I maintain a number of open source projects. Too often, something like this would happen:

I merge a pull request
I promptly forget about it
weeks later, the changes haven’t reached a package registry

So, I made Repominder to help OSS maintainers avoid this situation. Now when I forget about changes, I get a weekly email with diffs of affected projects. It also provides a badge showing the status of my projects (eg).

Feel free to sign up and monitor your own projects! It’s open source, and likely to remain free unless hosting becomes annoying.

tabs.executeScript as a content script alternative in Chrome Extensions

2017-07-17T00:00:00-04:00

tabs.executeScript as a content script alternative in Chrome Extensions

July 17 2017

I just updated Autoplaylists for Google Music so it uses chrome.tabs.executeScript instead of a manifest content script. It’s only been a few days, but so far it’s halved sync problems. Since I didn’t find similar recommendations elsewhere, this post explains what motivated the change and how I went about it.

First, some context on Autoplaylists. It’s a Chrome Extension that adds iTunes-style “smart playlists” to Google Music. It runs entirely in the browser and makes money from a freemium subscription model (there’s more details in my business-focused posts).

Since it uses unofficial apis, Autoplaylists depends on communication with a running Google Music tab. Specifically, it needs to:

retrieve the Google Music user id at startup
retrieve the cached library from Google’s IndexedDB at startup
retrieve Google’s xsrf cookie at startup, and on-demand (if it expires)

To do this, Autoplaylists previously used a single long-running content script which sent its tab id to the background script after loading. The background script would use that tab id for any on-demand messages.

This caused a few problems. First, a tab refresh was required after an install, upgrade, or reload, since the background script would lose the tab id and message passing channel. This was the worst issue: it’d interrupt syncing for no apparent reason, causing symptoms like these. Second, the background script had no way of knowing when tabs closed without messaging them. On-demand messages often ended up going into the void.

My new setup fixes these problems. The notable changes are:

the background script has tabs permission
at startup or when detecting a tab opening, the background script will:
- create a unique script id
- add a new message listener that detaches itself after receiving a response with the script id
- executeScript the former content script code (adapted for one-time use)
the content script code sends responses as before, but includes the script id

This moves the responsibility of running page code from Chrome into the background script. The added flexibility allows startup-time detection of tabs (after installs/upgrades/reloads), garbage collection of page code after one use, and easier handling of closed tabs for on-demand messages.

It’s not a perfect solution. Aside from the added complexity of script ids and dynamic event listeners, adding the tabs permission turned out to be costly. Unexpectedly, it’s presented to users as allowing the extension to “view your browsing activity”, which is easily misinterpreted as access to history. Furthermore, Chrome prompts users for all the extension’s permissions when just adding one. The result was a lot of scary prompts and a few negative reviews. I’m working on setting up a mailing list so I can get in front of changes like these next time.

Overall, though, the costs have been worth it. I’d recommend executeScript as a manifest content script alternative for extensions with long-running background scripts and on-demand page communication requirements. In case it’s helpful, you can find most of my relevant code in page.js, and the injected code in querypage.js. As always, feel free to reach out on GitHub or via email with questions.

Analytics for the Google Payments Center

2017-03-18T00:00:00-04:00

Analytics for the Google Payments Center

March 18 2017

tl;dr install the extension here.

If you receive money from Google you’re surely familiar with the Google Payments Center. I deal with it in the context of my subscription Chrome Extension Autoplaylists for Google Music.

Its main interface is a list of orders:

A screenshot from my account with full order numbers removed.

This is largely useless for answering business reporting questions such as:

how many new users purchased this month?
how many subscriptions churned this month?
what’s the net growth in subscriptions over time?

So, I’ve created an analytics Chrome Extension to calculate these numbers. It lets me answer these questions at a glance, like Baremetrics or ChartMogul do for other processors. Here’s the graph it generates for me:

A screenshot of the extension. Each bar is a monthly breakdown with the trend line showing net growth.

While it’s still rough around the edges, it’s already my favorite way to look at order trends. So, I’m opening it up for a free private beta. If you’d like to try it out, install it here.

Here’s a few more details:

your order information never leaves your own computer
subscription products will generate the best graph, but one-time purchases, in-app purchases, Android Apps, Youtube Fan Funding, ad payments, etc will also work
more metrics are in the works, such as monthly recurring revenue (MRR), failed charges, and life time value (LTV)
I’m interesting in addressing other pain points as well, such as bulk exporting of statements
I plan to charge after leaving beta

Side project income 2016: 0 to $100/month

2017-01-09T00:00:00-05:00

Side project income 2016: 0 to $100/month

January 9 2017, updated March 18 2017

If you’re selling an extension, check out my project Analytics for Google Payments.

In 2016, I set a goal to earn money from software I built. Being an employee or contractor wouldn’t count. I wanted the product and business experience of making something from the ground up.

I’m pleased to have achieved this: as of December, my projects net about $100/month of (mostly) passive income. This post will lay out some of what I’ve learned and hopefully help others with similar goals.

The Products

I launched two consumer subscription products last year. They are:

Autoplaylists for Google Music: iTunes-style smart playlists for Google Music, a $2/month Chrome Extension
Autoresponder for Google Chat: hosted auto-replies for Google Chat/Hangouts, a $2/month SaaS

Why did I pick these particular products? Three reasons:

they solve a problem I had, so I identify easily with users
they let me use my expertise, so I’m well-positioned to execute
they target a niche audience, so there’s little competition

The first two points make these projects work well, while the last one provides flexibility. I’ll use Autoplaylists as an example. Custom playlists like “my most-played metal not heard recently” were common in the days of local media players, and people like me missed them when migrating to hosted services. Other interested developers were turned off by the lack of an official api, but through my earlier work on gmusicapi, a reverse-engineered client library, I knew it could be done. And since only a small percent of Google Music users actually want this feature, the stakes are low and I don’t worry about competitors (including Google).

It’s worth noting that both projects are also open source. This is partly for philosophical reasons, but also because a public Github repo is a productivity boost. For example, Autoplaylists uses Github’s hosting for the product site and wiki for the support site, in addition to the obvious issue tracking and version control offerings. The added complexity around things like contributions and secret management has been straightforward to manage so far.

The Business

Running a small business seemed daunting at first, but it turned out to be pretty simple in this case. I run both projects as a sole proprietorship, mostly because I’m not worried about getting sued and have no intention of raising money. This is the simplest option: there was no extra paperwork besides the Employee Identification Number I got (which I can give out instead of my SSN). Bookkeeping just takes a few minutes of spreadsheet work per month, and my taxes remain straightforward. Even PCI compliance was a breeze since my SaaS uses Braintree’s drop-in iframe; I just had to click through a set of checkboxes to affirm I wasn’t doing anything obviously insecure. The trickiest business puzzle was figuring out how New York’s sales tax rules applied to Autoplaylists. For the details, check out my post on the tax and legal implication of launching a Chrome Extension.

Marketing, on the other hand, wasn’t so simple for me. I started the year out strong with the Autoplaylists launch. Here’s what I did, all of which I’d do again:

before launch, found passionate users for a beta and got them on a mailing list
at launch, reached out to tech journalists who’d written about Google Music. Gizmodo’s article landed tens of thousands of views for the cost of a few emails.
after launch, shared good autoplaylists to music-sharing communities

Unfortunately, I fizzled out after that. Not continuing to push Autoplaylists users onto a mailing list was a mistake, and I didn’t put much effort at all into the Autoresponder launch. While organic search and word of mouth have provided some growth, I’m planning to double down on content marketing and media outreach this year.

What’s Next?

2016 was a great start, and I’m proud to be a small business owner with happy customers. More importantly, I’ve now got the confidence for bigger ventures in 2017. Two things I’m considering are launching a product for small businesses and buying an existing product, each of which would present some interesting new challenges.

Hopefully next year I’ll be writing a similar post! If you’d like to follow along, be sure to subscribe with the links below. Also, feel free to send an email if I can be of assistance; my inbox is always open.

Publishing your first Chrome Extension, part 2: Analytics and Error Reporting

2016-07-18T00:00:00-04:00

Publishing your first Chrome Extension, part 2: Analytics and Error Reporting

July 18 2016, updated March 18 2017

If you’re selling an extension, check out my project Analytics for Google Payments.

Earlier this year, I launched my first paid Chrome extension: Autoplaylists for Google Music. The non-software work involved - like organizing a beta, wrangling paperwork, and finding users - was a bit daunting to handle solo. So, to help other first-timers publish their side projects, I’m writing up my experiences and making recommendations.

This post focuses on analytics and error reporting. For the first post on the tax and legal requirements of selling an extension, see part 1. If you’re interested in reading future posts, consider subscribing to my RSS feed.

Analytics and Error Reporting

Analytics lets you track how your extension is being used and found, while error reporting lets you find bugs users may not otherwise report. Without them, you’re sticking your head in the sand.

I recommend the free tiers of Google Analytics and Sentry. If you’ve used them before, know that there’s a bit of extra work involved to make them effective in an extension.

Google Analytics

I use Google Analytics heavily. It helps me answer questions like:

how many playlists are users creating?
how many visits, installs, and subscriptions did a writeup generate?
how does usage differ between free and paid users?
how is user growth changing over time?

To handle all this, Analytics runs on my web store page, the extension itself, and autoplaylists.simon.codes. They all report into one account as separate properties.

Hooking up the web store and homepage are trivial. Adding reporting to the Chrome extension itself is a bit more involved: you’ll need to use the Chrome Platform Analytics library. In addition, be sure to:

generate a per-user uuid, send it with requests, and keep it in sync storage
create a privacy policy and clearly link it in your extension
implement opt-out and an opt-out ui

I’ve linked my code above as examples.

Now that Analytics is running, you should customize it a bit. The web store is simplest: track installs by setting up a destination goal for “begins with /track_install”. I plot goal conversions and completions on the same chart and have it emailed to me weekly. I haven’t figured out a good way to track other web store events like reviews and comments.

Customizing reporting for your extension will of course vary based on what you’re doing. I’d recommend at least adding app views for your UI and an event for loading the extension. To give you some ideas on what else to track, here are some examples from Autoplaylists:

to segment paid vs free users: a custom dimension for paid status; custom segments for paid and unpaid.
to track sync success: events for sync success, failure, and retry; custom segments for users without syncs, without successful syncs, and with at least one successful sync.
to track onboarding success: a destination goal for playlist creation and required funnel from a zero-playlists event.

Sentry

Sentry tracks errors coming from your extension. Hooking up Raven, their client library, is simple: I don’t have much to add to their docs.

Once basic error reporting is working, there are two extra steps you should do every time you publish a new version: create a Sentry release and upload source maps. These will help you track down exactly which changes introduced an error. I’ve written a script that automates both steps (it also publishes my extension via Google’s web store api).

Keep in mind that Raven will only automatically collect exceptions, which Chrome apis don’t raise (they use lastError instead). I’ve written a decorator that will automatically send these errors to Sentry as well.

Finally, here are some parting tips:

if you use custom dimensions with Analytics, you’ll want to add that information to Raven’s user context as well.
consider wrapping Raven to make it accessible across background, UI, and context scripts.

Feel free to shoot me an email if I can help with anything: simon@simonmweber.com.

Publishing your first Chrome Extension, part 1: Taxes and Legal

2016-07-11T00:00:00-04:00

Publishing your first Chrome Extension, part 1: Taxes and Legal

July 11 2016, updated March 18 2017

If you’re selling an extension, check out my project Analytics for Google Payments.

There’s too much to cover in one post, so I’ll start with the part that was scariest for me: dealing with the tax and legal implications of selling software. I’m effectively a small business owner that sells a product, so I’ve got to follow the rules or risk getting shut down (or worse). If yours is free, consider subscribing to my RSS feed. I’ll have advice for you in the coming weeks.

Finally, I want to give thanks to Patrick McKenzie. He generously gave me some pointers on the topics below. If you find yourself wrestling with something not covered here, do consider his standing invitation to reach out.

Tax and Legal Considerations

Disclaimer time!

I am not a lawyer nor an accountant, so this post cannot be considered professional advice. Think of it as a starting point for your own research or conversations with professionals.
this is largely specific to the US, though the themes may apply elsewhere.
if you’re making real money, just pay someone to do all this for you.

With that out of the way, here are the main things I did before allowing sales:

chose a business structure
prepared to pay income tax
prepared to pay sales tax
handled Google’s merchant requirements

I’ll cover each separately.

business structures

I’m running as a sole proprietorship, which is free and simple. Basically, it means there’s no legal distinction between me and the business. The only additional paperwork was applying for an Employer Identification Number, which I got just to avoid giving out my SSN. Despite the name, it’s applicable even though I have no employees.

Reasons to not use a sole prop include things like desire to raise investor money or concern about getting sued. In these cases, a C Corp or LLC are likely more appropriate, but you should definitely talk to a lawyer before deciding – you’re in territory my side project likely won’t see.

income tax

I report income from my extension and pay taxes on that income. I include the income with my estimated tax, and next year I’ll file a Schedule C with the business’s total income and expenses.

If your extension makes little money and you typically receive a tax refund, estimated tax might be overkill.

Be sure to keep an excellent record of all your sales and business expenses. I’d recommend exporting both your earnings report and orders every month, as Google puts slightly different information in each.

sales tax

Unlike Apple’s App Store, Google’s Web Store makes you the merchant of record. This is both good and bad: it means Google’s cut is less than Apple’s, but it means you need to handle sales tax. How this works varies by state, but this is what I did for New York:

determined that my extension qualified as “pre-made software” for sales tax purposes
registered for a Certificate of Authority a month before my first sale
turned on automatic NY sales tax collection in the Payments Merchant Center

When it’s time to file sales tax, I check for sales to NY residents, look up the tax rate based on their address, and file my return online. This is tedious, but don’t let it discourage you: I’m willing to bet New York’s requirements are among the worst to deal with. For example, these sales wouldn’t qualify for sales tax in California, and many other states have reporting minimums I’m not likely to hit.

If you publish internationally, you may have to deal with sales tax on a country-by-country basis as well. For example, there’s VAT for Europe (which Google mostly handles for you) and JCT for Japan. With the amount of money I’m making, it doesn’t make sense for me to put much effort into this.

Google’s requirements

Compared to the government, appeasing Google is pretty straightforward. You’ll need:

your EIN for tax purposes
$5 for a one-time signup fee
a business address and email you’re comfortable posting publicly
your bank information for payouts
a privacy policy hosted somewhere (I use a Github wiki page)

With this you’ll create two accounts - a Payments merchant account and a Chrome developer account - and then link them together. Be sure to go through all the options in your merchant account: they include important settings like the name on users’ credit card statements.

This all may seem like a lot if you’re only making a few bucks a month. Still, I’d encourage you to go through with it: it took me only a few days in total, and the business experience has been super valuable. If you find yourself feeling overwhelmed, feel free to shoot me an email. While I’m far from an expert, I’m happy to help: simon@simonmweber.com.

Autoplaylists (iTunes smart playlists) for Google Music

2016-03-01T00:00:00-05:00

Autoplaylists (iTunes smart playlists) for Google Music

March 1, 2016

Autoplaylists - called “smart playlists” in iTunes - allow you to create playlists that always contain songs matching some set of rules. For example, you could create a playlist that contains only music you haven’t listened to recently, or your most-played songs of a certain genre.

They’re one of my favorite ways to listen to music, but have never been available in Google Music. So, I’ve created a Chrome Extension to add them. Here’s an example of one of my autoplaylists:

If this is something you’ve also found Google Music lacking, you can download it at https://autoplaylists.simon.codes. There’s a free version limited to one playlist, and a paid version supporting unlimited playlists.

The code is also open source, and can be found at https://github.com/simon-weber/Autoplaylists-for-Google-Music. Check it out and let me know what you think!

Automatic Responses for Google Chat and Hangouts

2015-10-18T00:00:00-04:00

Automatic Responses for Google Chat and Hangouts

October 18, 2015

Do you have Google accounts that forward email? If you never log into them, you might be missing Chat/Talk/Hangouts messages.

After missing an important message, I made https://gchat.simon.codes. It automatically replies to - and optionally forwards - the messages you’d otherwise miss. Check it out and let me know what you think; it’s free and open source!

Python linting at Venmo

2015-08-28T00:00:00-04:00

Python linting at Venmo

August 28, 2015
cross-posted at the Venmo blog

Quick! What’s wrong with this (contrived) Python 2 code?

import sys

class NotFoundError(Exception):
   pass

def enforce_presence(key, entries):
   """Raise NotFoundError if key is not in entries."""

   for entry in entries:
       if entry == key:
           break
   else:
       NotFoundError

If you said the unused import and missing raise keyword, you’re right! But, if you took longer than a quarter of a second to answer, sorry: you were outperformed by my linting tools.

Don’t feel bad! Linting is designed to detect these problems more quickly and consistently than a human. There are two ways to make use of it: manually or automatically. The former is flexible but not robust, while the latter risks getting in the way. We lint automatically at Venmo; here’s how we strike a balance between flexibility and enforcement.

We use a collection of linting tools. Currently we use flake8, pylint and a custom internal tool. They each address different needs: flake8 quickly catches simple errors (like the unused import), pylint slowly catches complex errors (like the missing raise), and our internal tool catches errors that are only relevant to Venmo. For easy use from the shell, we combine their output with git-lint and a short script. This setup catches a wide variety of errors and can easily accommodate new linters. Here’s what it looks like when run on the code from this post:

$ ./lint example.py
Linting file: example.py FAILURE
line 1, col 1: [F401]: 'sys' imported but unused
line 15, col 8: Warning: [W0104]: Statement seems to have no effect

Linting happens in three places during our workflow: in-editor, pre-commit, and during builds. The first step varies for each of us since we don’t all use the same editor (though vim with syntastic is a common choice). This is the step with the fastest feedback loop: if you don’t currently use linting, start with this.

The second step is implemented with a git pre-commit hook. It lints all the files about to be committed and aborts the commit if there are problems. Sometimes we opt out of this check - maybe we know about the problems and plan to address them later - by using git’s built in --no-verify flag.

Finally, any errors that survive to a pull request will be caught during build linting on Jenkins. It’s similar to the pre-commit check, but runs on all files that have been changed in the feature branch. However, unlike the pre-commit check, our build script uses GitHub Enterprise’s comparison api to find these files. This eliminates the need to download the repository’s history, allowing us to save bandwidth and disk space with a git shallow clone.

No matter when linting is run, we always operate it at the granularity of an entire file. This is necessary to catch problems such as unused imports or dead code; these aren’t localized to only modified lines. It also means that any file that’s been touched recently is free of problems, so it’s rare that we need to fix problems unrelated to our changes.

All of our configuration is checked into git, pinning our desired checks to a specific version of the codebase. Checks that we want to enable are whitelisted, allowing us to safely update our linters without worrying about accidentally enabling new, unwanted checks.

When enabling a new check, we also fix any existing violations. This avoids chilling effects: we don’t want to discourage small changes through fear of cleaning up lots of linting violations. It also incentivizes automated fixes, which saves engineering time compared to distributed manual editing.

Hopefully, sharing our linting workflow helps save you some time as well!

python-libfaketime

2015-03-29T00:00:00-04:00

python-libfaketime

March 29, 2015

I recently released python-libfaketime, which makes it easy to mock the current time during tests. It’s also two orders of magnitude faster than freezegun on my laptop. This is because instead of searching for and monkeypatching imported python stdlib modules, python-libfaketime uses libfaketime to intercept c standard library calls.

Here’s how it works:

the user sets an environment variable requesting that the dynamic linker prioritize libfaketime over the c standard library (eg LD_PRELOAD on linux)
python-libfaketime communicates the datetime to libfaketime through another environment variable

The second step is seamless to end users, but the first step can be annoying. To make this easier, python-libfaketime provides a function to modify the environment and re-exec the current process.

Give it a try if you notice slow tests while using freezegun – it made the Venmo test suite about 20% faster.

Python Logging Traps

2014-11-24T00:00:00-05:00

Python Logging Traps

November 24 2014
cross-posted at the Venmo Engineering blog

Python’s logging framework is powerful, but gives you plenty of ways to shoot yourself in the foot. To make matters worse, logging mistakes are subtle and slip through code review easily.

In my time at Venmo, I’ve seen logging mistakes cause everything from unneeded debugging to application crashes. Here are the most common culprits:

Manual Interpolation

Bad:

logger.info("My data: %s" % some_data)
# or, equivalently
logger.info("My data: {}".format(some_data))

Good:

logger.info("My data: %s", some_data)

This pushes the interpolation off to the logging framework:

interpolation overhead is skipped if info level is not enabled
encoding problems will not raise an exception (they’ll be logged instead)
tools like Sentry will be able to aggregate log messages intelligently

Pylint will detect this antipattern as W1201 and W1202.

logger.error(…e) in exception handlers

Bad:

try:
    ...
except FooException as e:
    logger.error("Uh oh! Error: %s", e)

Good:

try:
    ...
except FooException:
    logger.exception("Uh oh!")

Formatting an exception throws away the traceback, which you usually want for debugging. logger.exception is a helper that calls logger.error but also logs the traceback.

Implicitly encoding unicode data

Bad:

checkmark = '√'.decode('utf-8')
logger.info("My data: %s", checkmark)

Good:

logger.info("My data: %r", checkmark)

Using %s with a unicode string will cause it to be encoded, which will cause an EncodingError unless your default encoding is utf-8. Using %r will format the unicode in \u-escaped repr format instead.

Adding a vcs dependency to requirements.txt

2014-11-23T00:00:00-05:00

Adding a vcs dependency to requirements.txt

November 23 2014, updated August 27 2016

It’s sometimes useful to point your requirements.txt at code not yet on pypi. For example, imagine you’ve just sent a bugfix PR to one of the libraries you depend on. Instead of waiting for the PR to be merged and packaged, you can temporarily change your dependency to point at your fork.

When adding the dependency line, it’s best to provide every field. Leaving some out can have unexpected effects, like pip installing a different version of the code or recloning it on every run.

Here’s an example of a fully-specified requirement line pointing to gmusicapi 4.0.0 on Github:

git+https://github.com/simon-weber/gmusicapi.git@4.0.0#egg=gmusicapi==4.0.0

Here’s an explanation of each field:

git+https://...git: the vcs type with the repo url appended. https (rather than ssh) is usually how you want to install public code, since it doesn’t require keys to be set up on the machine you’re running on.
@4.0.0: the git ref to clone. This example points to the 4.0.0 tag. You don’t need to use a tag - pip will happily use anything that can be checked out, such as a feature branch - but I recommend it to avoid using a ref that will point somewhere else later (like master likely would).
egg=gmusicapi: the name of the package. This is the name you’d give to pip install (which isn’t always the same name as the repo).
==4.0.0: the version of the package. Without this field pip can’t tell what version to expect at the repo and will be forced to clone on every run (even if the package is up to date).

If the maintainer doesn’t change package versions between releases, you’ll want to change it on your branch so pip can tell the difference between your temporary release and the last release. For example, say you contribute a fix to version 1.2.3 of a library. To create your new version, you could:

branch from your feature branch
change the version to 1.2.4-rc.1, since it’s a release candidate of the bugfixed 1.2.3 release
use a requirements line like git+https://github.com/me/lib.git@new_branch#egg=lib==1.2.4-rc.1

VCR.py at Venmo

2014-05-01T00:00:00-04:00

VCR.py at Venmo

May 1 2014

Recently at Venmo, I spent some time improving our Python test suite. It was in pretty bad shape when I arrived: builds took 45 minutes and tests were flaky. Now, builds take 6 minutes and are much more dependable.

I was asked to write up some blog posts describing our changes, and I decided to start with mocking external interactions. The full post is at the Venmo engineering blog. Hopefully it’s helpful! Feel free to email me with questions.

Staying Busy

2014-04-17T00:00:00-04:00

Staying Busy

April 17 2014

I’ve been with Venmo for about six months now, and everything has been great! I’ve worked on all sorts of things - security fixes, features and performance improvements among them - but my main concern has been our test suite.

When I first showed up, our builds were flaky and took nearly an hour to run. Now they’re much more stable and run in five minutes. I’ve got some Venmo blog posts in the works about how we acheived this (which I’ll be sure to cross-post or link here).

This work has been fulfilling, so my personal projects have been a bit neglected. However, I’ve been open-sourcing a lot of Venmo code lately:

Thomas Boyt and I added Venmo to Gittip
A few of us wrote btnamespace, a library to make Braintree integration testing easier
I’m about to release a nose plugin to detect unmocked integration tests

I’ve got some more detailed updates in the works – stay tuned!

Google Helpouts first impressions

2013-11-07T00:00:00-05:00

Google Helpouts first impressions

November 7 2013

Google Helpouts launched on Tuesday. If you haven’t heard of it, it’s a peer-to-peer platform for getting expert help. Naturally, Google Hangouts are used for the actual sessions.

I used to provide free computer science tutoring when I was in school, so I figured this might be a nice way to keep giving back. After getting into the private beta a few weeks ago, I whipped up a free CS help listing – you can see the new version of it here. The Helpouts team looked it over and scheduled me for a one-on-one chat, which lead to my listing being accepted for the launch. I set my availability for 7:45pm - 10pm, thinking that I’d be lucky to get one or two hits.

To my complete surprise, I ended up being booked for nearly the entire night! Here’s how my sessions went:

call one: ideal

I joined the first Hangout 5 minutes early. After an awkward 10 minute wait, my first user showed up. We exchanging greetings, then paired on a Python program. Setting up a reasonable way to share code wasn’t super easy, but we eventually made good use of the Hangout’s screen sharing. The allotted 15 minutes gave us just enough time to work out a solution before he signed off. Success!

Honestly, my first Helpout couldn’t have gone much better. I was able to be immediately helpful, the Hangout worked perfectly, and I even got a nice review.

call two: no help needed

My second user was also about five minutes late. To my surprise, he didn’t need any help; he was just a developer who wanted to chat. So, we talked shop. While this wasn’t what the platform was intended for, it was still fun.

calls three through seven: no shows

Unfortunately, I soon found that my first experiences may have been flukes. My next five users either didn’t show or cancelled at the last minute.

call eight: off-topic

A user showed up for my final slot of the night. However, English wasn’t his first language, and we had some trouble communicating. I eventually discovered that he was interested in programming for the job opportunities, so I provided some resources off the top of my head: Cousera, Udacity, Codecademy, etc. I also spent some time explaining CS as a discipline and attempting to separate it from software engineering.

I disconnected with the feeling that my time wasn’t well spent. But, hey, at least someone showed up.

Impressions

Obviously, the no-shows were a huge problem. Not only did it waste my time, but it may have prevented me from helping actual users. To mitigate this, my listing now charges a dollar upfront, but promises to refund it if the user shows. So far this hasn’t gotten any takers, so it’s hard to tell if it’s working or just scaring everyone off.

I got in touch with the Helpouts team about my problem - over a Helpout - and they told me there wasn’t anything in the pipeline to fix it. However, they did seem aware of the problem. I do hope it’s addressed; it’s absolutely sapped some of my enthusiasm for the platform.

No-shows aside, my first Helpout really showed the potential of the platform: fifteen minutes of my time made a real difference! I’ll definitely keep some timeslots open on the platform, and hope to have better experiences to share in the future.

Strangers, Go, and Openshift

2013-07-31T00:00:00-04:00

Strangers, Go, and Openshift

July 31 2013

I’ve always liked the idea of Omegle: one-on-one text chats with strangers. I also hang out in irc a lot, and thought it’d be fun to combine the two. This wasn’t a new idea, but since the Recurse Center is all about learning by exploring, I built an Omegle to irc bridge with Twisted.

When I hooked it up in #rochack, someone had the idea to have the stranger talk to our resident markov chain bot. The stranger had no way to know they were talking to a bot, and hilarity ensued.

The thing was, only folks in our channel could watch. To share this, I built Strangerbotting. It’s written in Go, and runs on Openshift, Red Hat’s newish open source PaaS. There are a couple of pieces to the app:

gomegle: a client library for Omegle
gomarkov: simple, fast, in-memory markov chains for text generation
strangerbotting-backend: manages one bot-to-stranger conversation and streams it out over websockets. Runs on Openshift.
strangerbotting-frontend: a simple webclient to display conversations. Also runs on Openshift.

There’s nothing too magical going on in the code, so I won’t bother talking about it. However, I have some new opinions on Go and Openshift that I wanted to write down.

Golang

Before getting started with Go, I’d seen a lot of folks complain about the more pedantic parts of the language. To me, this indicates some combination of:

disagreement with Google’s engineering culture
unfamiliarity with Go’s design goals

For example, unused imports in Go raise a compile error. This reflects both Google’s conservative engineering tendencies (they wouldn’t want such code checked in), and the goal of fast compilation (see here for details).

Maybe it’s last summer’s Google indoctrination speaking, but I’m fine with these parts of the language. That said, I still find Go’s error handling - which eschews exceptions for multi-value returns - a bit wonky. For those unfamiliar with the language, here’s the relevant part of the Go faq:

Why does Go not have exceptions?

We believe that coupling exceptions to a control structure, as in the try-catch-finally idiom, results in convoluted code. It also tends to encourage programmers to label too many ordinary errors, such as failing to open a file, as exceptional.

Go takes a different approach. For plain error handling, Go's multi-value returns make it easy to report an error without overloading the return value. A canonical error type, coupled with Go's other features, makes error handling pleasant but quite different from that in other languages.

Go also has a couple of built-in functions to signal and recover from truly exceptional conditions. The recovery mechanism is executed only as part of a function's state being torn down after an error, which is sufficient to handle catastrophe but requires no extra control structures and, when used well, can result in clean error-handling code.

This means that the majority of function calls get wrapped in a conditional check on a returned error, which can seem verbose and tedious. While this certainly required some getting used to, it doesn’t bother me anymore. After all, I end up using a similar number of try/except blocks when writing robust Python code.

What does bother me is the loss of debugging information. For example, here’s an error I received from Go’s http client: Post http://front2.omegle.com/events: EOF. The unclear message and lack of a stacktrace left me guessing: is my code broken? Is Go’s? Maybe it was the server? I ended up reluctantly grepping the stdlib source, since I didn’t even have a line number to go off of.

Thankfully, there’s enough to like in Go that I’ll put up with some debugging annoyances. First off, goroutines and selection over channels make for really easy concurrent code. As somebody who was new to both Twisted- and CSP-style concurrency a month ago, I now greatly prefer the latter.

The stdlib is also pretty decent, even if it can feel somewhat inconsistent. For example, there’s no deque implementation, but there is a one for suffix arrays. I’m willing to write this off as a combination of Python spoiling me and Go being relatively new.

However, despite its youth, Go’s community is fantastic. The #go-nuts channel and official blog are informative and active, and play.golang.org makes sharing code a breeze.

Overall, I came away pleased with Go, and I’ll definitely consider it for backend work in the future.

Openshift

Openshift, however, left me with mixed feelings. On one hand, I love the ability to ssh into a PaaS box and have a persistent filesystem. Rather than futz around with custom buildpacks, I just downloaded and compiled Nginx to host the Strangerbotting frontend. Their free tier is generous, too; I wouldn’t have to pay for a three node host/reverse-proxy to redundant backend setup.

However, there’s not much else to like. In particular, their bizarre naming choices are a constant source of confusion. Apps are the top-level construct. They are made up of gears (VMs), which run cartridges (buildpacks).

To run the three node architecture I referred to earlier, you don’t make an application with three gears, though; that’d be too easy. Instead, you need to make three applications, each with one gear. You see, you only get one “web gear” per application, which the other gears then support (by eg hosting a database or build system).

Since their docs don’t make any of this clear, expect to dig around the forums for answers. Be warned, they’ve got a very “googling for .NET answers on SO” kind of vibe to them.

I ran into all sorts of other minor difficulties. For one, there’s no official Go cartridge. There is a Red Hat community catridge, but that’s been broken for months now, apparently. After spending a half a day trying to fix it and cursing at their slow cartridge development system, I gave up. I’m currently deploying a binary over git, which I’m not too happy about.

These kinds of problems left me with the impression that the service isn’t quite there yet. So, while I won’t be paying for Openshift any time soon, I will be rooting for them. An open source PaaS is a noble goal, and more I always welcome competition for my dollars.

Python protobuf on Google App Engine

2013-06-18T00:00:00-04:00

Python protobuf on Google App Engine

June 18 2013

Today, if you try to use Google Protocol Buffers on App Engine, you’ll run into an error like this one:

from google.protobuf import descriptor 
ImportError: No module named protobuf

The problem is that Google’s own App Engine apis also use the google package namespace, and they don’t include the protobuf package.

Thankfully, there’s a simple way to fix this. First, vendorize the library as you normally would. I just ripped the site-packages folder from a virtualenv into the application root:

app.py
app.yaml
vendor
├── google
│   └── protobuf...
└── httplib2...

Then, just add your google directory to the google namespace, and add your vendor directory to the system path. I used this bit of code:

import os
import sys

import google  # provided by GAE

# add vendorized protobuf to google namespace package
vendor_dir = os.path.join(os.path.dirname(__file__), 'vendor')
google.__path__.append(os.path.join(vendor_dir, 'google'))

# add vendor path
sys.path.insert(0, vendor_dir)

That’s it! There’s no need for weird hacks.

Chrome extension hacks: Google Music to turntable

2013-06-05T00:00:00-04:00

Chrome extension hacks: Google Music to turntable

June 5 2013, updated March 18 2017

If you’re selling an extension, check out my project Analytics for Google Payments.

I just shipped my first Recurse Center project: a Chrome extension that scratches a personal itch. I use turntable.fm every now and then, which lets me dj music with friends. There, I can search for a song, and if someone has previously uploaded it, queue it up for everyone to hear. If turntable doesn’t have a song, I can upload a file from my computer.

However, I don’t normally have music files on my laptop; my library is kept in Google Music. So, if turntable was missing a song, I used to:

open Google Music and sign in
find and download the song
switch to turntable and upload it
delete the file

My extension lets me do this right in turntable. Now, all I do is click “Upload from Google Music”, find the song and hit upload. The extension does the same thing behind the scenes.

If this sounds useful, you can get it here: Turntable Uploader for Google Music™. But, this blog post isn’t about advertising: I wanted to document all the weird hacks that went into making this work. If you’re already a Chrome extension guru, you may already know this stuff; it’s intended as a post I would have wanted before writing a line of code.

First, a brief intro to Chrome extensions. If you’ve written one before, you can probably skip this part.

Intro to Chrome extensions

Chrome extensions are just normal javascript and html, but with optional extra permissions. For example, cross domain requests and cookie access are kosher – you just need the user to approve them.

Your extension can be made up of two different kinds of scripts. The first is the background script (or event script; the same thing but not always running). This does the heavy lifting, since it gets access to special chrome.* apis. These enable stuff like the cookie access I mentioned earlier.

You can also run a bunch of content scripts. Unlike the background script, these are specific to a certain page. They get triggered when a tab matches a url pattern you specify (eg http://turntable.com/*). They don’t get chrome.* access. They can access the DOM, but run in an isolated world – basically, they can modify the DOM, but can’t mess with other code running on the page. For example, my content script can’t remove an event handler that a turntable script has set up.

At a first glance, content scripts sound pretty limited. However, you can get around all of the restrictions above with a bit of hackery. To access chrome.* apis, there’s a two-way messaging interface to the background script: you just offload the work there. Anything json-encodable is fair game for transport. Even the isolated world isn’t bulletproof: with DOM access, you can inject a script tag to get at the global namespace.

My extension uses all three pieces mentioned above: background, content, and injected scripts. All together, they communicate like this:

our background script
     ^
     |
     |
   (chrome message passing)
     |
     v
our content script
     ^
     |
     |
   (the dom)
     |
     v
our injected code
     |
     |
   (global namespace)
     |
     v
other code on the page

Now that we know what we have to work with, I’ll go over how I addressed each of the big pieces of my solution.

Getting the Google Music library and downloading songs

I’ve spent far too much time with the Google Music protocol from my work on gmusicapi, so I already knew which endpoints to hit.

Auth presented a hurdle, though, since it requires either plaintext credentials (yuck) or OAuth (annoying). I got around this with my extra Chrome host and cookies permissions: I just have the user open Google Music in another tab, then piggyback on that session. My requests will automatically send Google cookies, and I just have to grab an xsrf cookie for use in the url.

Uploading to turntable

This was the toughest nut to crack. I considered reverse engineering the endpoints that turntable’s client page used for uploading, but this had a number of disadvantages:

the user wouldn’t see the upload in the turntable interface
I don’t have the javascript chops to implement a complicated upload protocol
future protocol changes would break stuff

A better approach: tricking turntable’s own clientside code into believing the user had initiated a normal upload. This makes turntable do all the work, and is business as usual for the user.

After some quality time in the DevTools debugger, a friend of mine (thanks, Charlie!) figured out a way to do this. turntable uses a third party library called plupload. Unfortunately, a high level window.plupload.upload(File) function isn’t accessible; it’s hidden inside turntable closures. However, a similar function is stored directly as a handler on the main file input, meaning that we can spoof an upload with something like $('..input[file]..').onchange.call(our_File).

Since injected code gets around the isolated world restriction, this is totally possible: we just need to also inject an html5 File containing the desired mp3.

Getting a File from a Blob

Html5 Blobs are easy to create, and just represent binary data. Html5 Files, though, can only be created when a user interacts with a file input (here’s the File api docs, if you want the details).

Luckily, Files aren’t much different from Blobs: they just add a filename and date of modification. Duck typing to the rescue!

//b is our Blob
b.name = 'myfile.mp3';
b.lastModifiedDate = new Date();
// tada!

Putting it all together

Those are the main hacks. Combining everything, this is about what happens when an upload is requested:

content script messages the background script with a file id to upload
background script performs a cross domain request to Google and retrieves a Blob
background script encodes the Blob as a base64 dataurl
Blob dataurl (now json-compatible) is messaged back to the content script
content script injects the entire dataurl, along with code to do the Blob to File spoofing, trigger the plupload code, and clean up when done

From there, I just added a bunch of messaging to get the ui working and a third party library to display the Google Music library. You can grab my ugly code on my GitHub.

I plan to write continue writing posts like these during my time at the Recurse Center, so if you dug this, Twitter or RSS are the best ways to get more.

Many thanks to my fellow Recursers who read over this post: Leo Franchi and Erik Taubeneck.

gmusicapi retrospective

2013-05-14T00:00:00-04:00

gmusicapi retrospective

May 14 2013

gmusicapi has come a long way since I pushed my first commit back in January 2012.

The project has grown more than usual during this semester, since – for the first time – I worked on it for school credit. Here’s what I spent my time on:

a number of big features: scan-and-match support, album art operations, Music Manager OAuth support, python 2.6 compatibility. . .
a new dynamic testing system for my end-to-end tests
a huge rewrite to allow for more flexible protocol declarations
continuous integration that also catches protocol changes
documentation improvements and a new irc support channel
version 1.0!

Github’s graphs provide a quick overview of what I’ve been up to. Here are a few other metrics:

documentation visits x5, bounce rate halved
PyPi download rate x2
use in a couple of real-world products

I figure this is a good chance to reflect on some of the decisions I’ve made, both for my sake and for those who might come this way in the future.

Don’t count out end-to-end testing

I have a few unit tests, but the majority of my tests are end-to-end black-box tests that exercise my clients against actual Google servers. Naturally, they violate core unit test principles: they’re stateful, cannot be run offline, and run slowly.

One thing I’ve learned while coming to this solution: there’s a lot of dogma surrounding test strategies.

It’s not that I’m against unit testing. I’d love to have more unit tests! Unfortunately, writing them isn’t free, and in my case, end-to-end tests offer more bang for my buck. I get more coverage with less code and get to test my expectations about Google’s servers. gmusicapi is also pretty small, so the loss of granularity when bug-hunting isn’t a huge deal.

Of course, there’s some extra work required to make your end-to-end tests robust. Backend flakiness can be addressed with retry and backoff. Test dependence and state management can be handled with TestNG-like frameworks (I use Proboscis). Protocol changes can be caught by validating responses against a schema.

Currently, I have a few Google accounts I use exclusively for testing, and I keep their credentials in TravisCI encrypted variables. There’s one big downside to this: encrypted variables can’t be used with other folks’ pull requests. In the future, I hope to figure out a way to safely manage a public test account (I think it’s possible with 2-factor auth).

Invest in support

I have trouble saying no to requests for help. Interestingly, gmusicapi also seems to attract many users who are new to Python. Because of these factors, I used to spend a fair amount of time answering support emails. I didn’t mind this, but after receiving a few emails on the same topics, I figured it’d be better to flesh out the gmusicapi documentation.

Basic questions were addressed with installation and getting starting sections. These were no-brainers, and I should have added them earlier.

I also added a section for a less obvious topic: people want the reverse-engineered protocols I’m using. So far, I’ve thrown together a hack that introspects my protocol declaration code, but I’ve got plenty of room to improve this.

The #gmusicapi irc channel on Freenode is another recent support addition. I wish I’d set it up earlier! Recently, devs building on gmusicapi have started hanging out and answering questions, which has been a huge help.

I still go the extra mile when I do direct support. Helping users out is its own reward, but I’m also flattered to have received thank-you emails after irc chats, money, and – once – even a gift in the mail!

Looking forward

I’m very pleased with what I’ve learned and accomplished so far. My next step is to get the project into a self-sustaining state. I figure this means knocking out todos, beefing up internal documentation, and refactoring all of the wonky bits that have accumulated over time.

Thankfully, I’ll have plenty of time this summer at the Recurse Center!

Development on a Chromebook: an opinionated guide

2013-04-20T00:00:00-04:00

Development on a Chromebook: an opinionated guide

April 20 2013, last updated January 6 2017

I started using a Samsung 550 Chromebook as my on-the-go machine two semesters ago. It worked nicely for taking notes, but I remained a skeptic: how could I ever write code from a glorified web browser?

Fast forward 6 months: today, I love hacking on my Chromebook, and I have no problems working offline. It took some effort to get everything set up, so I’ve put together my recommendations to get other folks up to speed.

First, a disclaimer: the device was given to me as part of the Google Student Ambassador Program, and Google pays me for brand advocacy at my school. That said, I’m not getting paid to write this, and this is my advice, not Google’s. I’m a hacker, not a shill (and I’ll even save hn the work of linking pg’s submarine essay).

Anyway: let’s get started. It’s easiest to work over ssh, so I’ll cover this first. Later, I’ll get to working offline. I won’t talk about cloud development webapps (maybe check out Nitrous.IO?). Also, I don’t have any advice if you prefer heavy IDEs; my usual tools are a terminal and web browser.

For when you have a good connection

ssh

You want Secure Shell as your ssh client. It’s basically openssh wrapped for NaCl, with hterm powering the ui.

It supports everything you’d expect, like key authentication, per-connection profiles, port forwarding, custom color schemes, and even your ~/.ssh/config file (well, probably; mine is simple).

The chromium-hterm mailing list is where updates are posted. If you need an upcoming feature now, there’s also a hidden dev channel (you need to be logged into an account that’s on the mailing list for this link to work).

A quick tip: set Secure Shell to ‘Open as Window’, otherwise Chrome will intercept keyboard shortcuts (which e.g. makes Control-W close your terminal). You’ll probably also want to set TERM to xterm. The FAQ (linked below) has the details and is worth reading through:

Secure Shell FAQ
Solarized colors (run the commands in a DevTools javascript console)

vpn

Legend has it that CrOS supports OpenVPN and L2TP/IPsec. However, they’re notoriously difficult to get working in some configurations; I never got mine working.

edit: I haven’t tried this for about a year, and it seems like the team is making progress.

remote desktop

If you need a graphical environment, you can use Chrome Remote Desktop. This provides vnc-like functionality across Windows, Mac and Linux, and can be set up for repeated or one-off access. Note that you cannot currently remote into a Chromebook.

I rarely need this.

Crosh Window

Crosh, the built-in CrOS shell, can (and should) usually be avoided. If you find yourself using it, Crosh Window takes away some of the pain; it fixes the Control-W problem mentioned earlier and gets you an up-to-date version of hterm.

For when you’re offline or on a terrible connection

GalliumOS

edit: GalliumOS is a linux distribution that’s designed for Chromebooks. For example, it supported my Pixel’s keyboard layout and touchscreen out of the box. If you’re comfortable maintaining your own linux install - or regularly work with VMs - I’d suggest it over crouton.

My setup is a dual-boot installed with chrx, which you can find instructions for here. If you run into problems, the subreddit and irc channel are quite active.

crouton

For offline work, you’ll want root access to a local Linux install. crouton is by far the best way to go about this: it runs Ubuntu in a chroot. This has security implications (check the README), but you avoid the performance hit of virtualization, and keep all the CrOS functionality.

You’ll need to have your Chromebook in developer mode (i.e. rooted) to use it, which is easy: I just flipped a hardware switch. The specifics for going about this vary by model, so just search for instructions. Once in dev mode, you’ll want to hit Control-D on each boot to skip the “you’re in developer mode” warning (there’s a 30-second wait otherwise). Have faith: this isn’t nearly as annoying as it sounds.

The crouton README has all the information you need to get started. Note that you can run a normal graphical environment (e.g. Xfce) alongside CrOS. I prefer using Secure Shell to ssh into localhost so I can keep my terminal customizations and stay in CrOS. If this sounds appealing, here’s what I did:

used the crouton cli-extra target (eg crouton -t cli-extra ...).
installed openssh in my chroot
start sshd, then use Secure Shell to connect to my-user@localhost

To make life a bit easier, I stuck /etc/init.d/ssh start into my chroot’s /etc/rc.local (which crouton runs upon mounting). Now, when I want to work locally, I just Control-Alt-Forward to get my local shell, $ sudo enter-chroot, Control-Alt-Back to CrOS and then run Secure Shell. You could probably get your chroot to mount and run sshd on boot if you use it all the time.

mosh

As an alternative to ssh on flaky connections, you can use mosh. It’ll need to be installed on your server to use it.

edit: I used to recommend running it inside crouton, but there’s now a proper mosh Chrome packaged app.

Parting words

edit: zRAM is now enabled by default, so you don’t need to worry about turning it on yourself.

If you’re a fellow Chromebook hacker and think I missed something, definitely let me know. I’ll do my best to keep this guide updated as better tools arrive.

edit: here’s a link to the hn discussion.

Tough cookies: a debugging story

2013-02-14T00:00:00-05:00

Tough cookies: a debugging story

February 14 2012

I used to think of (web) cookies as simple key/value pairs. That was before I spent an hour tracking down a bug in gmusicapi.

The symptoms were simple. The user would successfully log in. Then, on their first request to a web service endpoint (eg music.google.com/loadalltracks), Google would reject their request and redirect them to a login page.

The auth code was immediately suspect. However, there were a few complications: - I could not recreate the issue - that code hadn’t changed much since last known-good version

The user who reported the bug stopped by #gmusicapi on Freenode (thanks, Leonardo!) and humored my requests for debug information. Here’s what we found out:

the login was successful; calls to Music Manager endpoints succeeded
the user was outside of the US
normal login to music.google.com through a browser was fine
the bug did not depend on multiple login status, locale assumptions or current login status
the user had all the proper session cookies

Throughout all this, I still could not recreate the issue. In a lucky guess, I used a test account from outside the US: bingo! This only affected non-US logins!

Now that I had a way to recreate the bug, I fired up git bisect to track down the commit that introduced it. Inside a function to take a requests.Request and send it off to Google, here’s the relevant code before:

if send_xt:
    request.params['u'] = 0
    request.params['xt'] = self.get_web_cookie('xt')

# web_cookies is a CookieJar
request.cookies = self.web_cookies

prep_request = request.prepare()
s = requests.Session()
res = s.send(prep_request)

return res

and after:

if request.cookies is None:
    request.cookies = {}

#Attach auth.
if send_xt:
    request.params['u'] = 0
    request.params['xt'] = self.get_web_cookie('xt')

if send_clientlogin:
    request.cookies['SID'] = self.client.get_sid_token()

if send_sso:
    #dict <- CookieJar
    web_cookies = {c.name: c.value for c in self.web_cookies}
    request.cookies.update(web_cookies)

prepped = request.prepare()
s = requests.Session()

res = s.send(prepped, **session_options)
return res

The commit was some refactoring from two separate request-sending functions (eg send_web_request and send_musicmanager_request) to one. Do you see the bug? Here’s a hint, changing:

web_cookies = {c.name: c.value for c in self.web_cookies}
request.cookies.update(web_cookies)

to:

request.cookies = self.web_cookies

solves the problem (hackily, since then clientlogin and sso auth can’t be sent together, but that never happens anyway - and this will all be rewritten soon).

If you didn’t figure it out, here’s the issue: auth was stored in Python Cookie objects, but I had attached them to the request as a name/value dictionary. In doing so, there was information loss:

in the case of cookies with identical names, only one would be sent
Cookie-specific fields like secure and domain were dropped

In their Python form, Cookies are not simple name/value pairs! Attaching Cookies directly to the request kept all the relevant information and solved the problem.

I’d like to thank Lukasa and SigmaVirus24 on #python-requests for pointing me to the relevant Requests internals, and for generally putting up with my mad raving. Lukasa also had what I thought was some sharp insight into the situation:

simon_weber: I suppose it would be nice to be able to set both simple and Cookie cookies using Requests

Lukasa: I think more than that, we want to discourage it. Cookies are complicated and easy to get wrong (as this entire discussion shows)

Lukasa: And so we’d rather that people use the known-good code in Requests

He’s exactly right, of course, and I already have an entry on my todo list for this (I’ve just got a lot on my plate at the moment).

At least this story has a happy ending: despite the confusion, gmusicapi soldiers on with happy international users. As for me, having learned from this bug hunt, hopefully I’ll never be suckered into disrespecting the surprising complexity of the cookie - nor their internet cousins of the same name.

What's in a genre?

2012-09-07T00:00:00-04:00

What’s in a genre?

September 07 2012

I’m taking a course on data mining this semester. Our first assignment: mine some data. The dataset and techniques don’t matter; the point is to extract meaning in any way possible. I’m greenhorn data miner; hopefully I’ll be able to look back at this post and laugh at my own naivete.

For my dataset I chose my own Google Music library. It’s unique, big enough (7600+ songs), and well organized. Plus, it’s a cinch to access with my Google Music api.

My analysis was simple: I investigated the occurrences of words in genres. I figured the most frequent words would be genres themselves (eg ‘metal’ in ‘power metal’), but there was also the chance of exposing common adjectives (eg ‘post’ in ‘post-rock’ and ‘post-metal’).

A few lines of Python later, and I had my results. The first thing I noticed: I listen to a lot of metal. A third of my songs are some kind of metal. If you put all the genre words into a hat, you’d pick ‘metal’ almost a quarter of the time. Next up: ‘rock’ and ‘jazz’. Rounding out the top six are two adjectives - ‘alternative’ and ‘progressive’ - as well as ‘accompaniment’ (as in Jamey Aebersold).

Metal bands also claim the longest genres in my library. Novembre is the champion, boasting this mouthful: ‘Progressive atmospheric doom metal’.

Now that we’ve figured out I’m a jazz-playing metalhead, let’s take a look at the least common words. ‘Country’ appears only once (and at the risk of sounding one-sided, it labels the fantastic Slaughter of the Bluegrass). There’s a bunch of mispellings, too, like ‘reggaer’ and ‘sountrack’.

Here’s a quick chart of all the words I found:

This assignment turned out to be a surprising amount of fun. For any other music lovers who want to take a dive into their libraries, I’ve got the source on GitHub.

Returning to the east

2012-08-29T00:00:00-04:00

Returning to the east

August 29 2012

I spent the summer as a Google intern at their Mountain View headquarters. Having recently returned to the east, the California sun has now melted into Pennsylvania rain; it’s an anticlimactic end to an otherwise fantastic summer.

Google exceeded my expectations. I arrived skeptical, expecting to see their world-famous perks masking a typical BigCo™ culture - where engineers are codemonkeys, bureaucracy reigns, and nobody cares about users. I’m happy to report that this is far from the truth. While Google as a whole is definitely nothing like a startup - don’t let anyone tell you otherwise - the culture felt healthy. Googlers constantly stick up for their users, openly asking the hard questions of management.

Google’s also got an incredible amount of talent, which was reflected in those I worked with. My team was responsible for making Google’s call centers more efficient (yes, Google does pick up the phone, but only for paying customers). I was tasked with a webapp to provide real-time visualization of our call centers’ statuses. This allows managers to intelligently shuffle agents around to different call queues. The entire project was my responsibility: project management, frontend, backend, monitoring, deployment. . .the works. I got to play with all kinds of internal secret-sauce, and despite ballooning requirements, my team and I were really pleased with what I shipped.

When I wasn’t banging out code, the Google perks didn’t disappoint. Tech talks and events: held daily (one of which I presented). The food: delicious. The booze: classy. Googlers are encouraged (expected?) to work hard and play hard. This seems to align with what I experienced at visits to other bay area companies: GitHub, Quora, LinkedIn, Stripe, Facebook, etc.

Silicon Valley boasts perfect weather, but the best part was the people. I hung out with a posse of interns from various tech companies. I danced in Google’s SF Pride contingent. I watched classic demoscene while discussing phreaker war stories at Hacker Dojo. I ate donut burgers with Zuck, partied with PJ Hyett, and boogied with Alan Eustace.

No, seriously: this is Alan and me on the dance floor.

All these people had a common feature: real passion for their work. Being around this energy has inspired me, so - with my Google legal restrictions lifted - I’m ready to jump into my side projects again. I’ll be maintaining my Google Music api, but probably won’t implement any huge new features. Instead, my time will likely be spent dogfooding it in a project to sync 3rd party media players to Google Music. I’ve already finished a proof of concept; now it’s time to buckle down and ship something. That is, if this unacceptable (read: not absolutely perfect) east coast weather doesn’t kill me first.

Cold-email a VP, publish in Phrack

2012-04-14T00:00:00-04:00

Cold-email a VP, publish in Phrack

April 14 2012

In the popular sense of the word, I’m no super-hacker. While coding, my sunglasses and fingerless gloves stay on the desk.

My collection of sexy trenchcoats is also lacking. (credit)

Despite these disadvantages, back in September I discovered a vulnerability in the web interface of a Netgear wireless router. Now published in Phrack, my exploit code allowed for nasty things like stealing admin credentials and hiding network devices. I wanted to let Netgear know, so I wrote my first disclosure: a friendly email briefly describing myself, the flaw, and my intentions of publishing.

Unfortunately, that was the easy part. A Netgear security email was nowhere to be found. In fact, I couldn’t even find a way to submit a support ticket (this has since changed).

If I could just get my message to a human, I figured it would end up in the right place. After all, who wants to be responsible for blowing off a security flaw? On Netgear’s contact page, I found a press relations email. No response. Investor relations channel? Nope (I must not be rich enough). Support emails found by Googling? Nothing.

This obviously wasn’t going to work. It was time to pivot.

Ten more minutes of searching got me everything I needed: the direct emails of five random support staff, plus the most executive position I could muster: the VP of engineering. Brazenly dumping them all in the recipient box, I tried again.

Within 10 minutes, one of the support staff got back to me, eagerly CC’ing his boss’s boss. I apologized, and my ensuing communication with Netgear was pleasant and to the point.

My takeaway: it’s easier to beg for forgiveness than to ask for permission. Well, that, and don’t let a lack of fingerless gloves keep you from submitting to Phrack.