I’m here at the IETF discussing HTTP/2.0 (was SPDY).

One of the most interesting developments to me is that the rate of protocol evolution is about to step onto a rocket ship. In the past, we’ve been dependent on standards, browsers, and servers to all simultaneously move to new protocols. It takes about 15 years, in a good case! But mobile apps are cracking this open. With native code, instead of relying on the browser’s network stack, you just include your client library (imagine a facebook client library which speaks a custom protocol optimized just for facebook). Since these apps are not doing general purpose web surfing, they’re only connecting to their own servers. If you own the client, and you own the server, who needs a standard?

This is going to lead to massive innovation in protocols. Companies can invent new paradigms and algorithms, and then either keep these to themselves or kick them back to the IETF for later. From what I hear, SPDY sounds like a promising starting point for these apps, and it will lead to splintering. But you don’t have to wait for Google, Microsoft, and Firefox to agree on protocols anymore! As Will Chan puts it, it’s like HTML for HTTP – everyone can change it in practically real time.

The one last challenge is getting your protocol to work across the internet, where proxies, intermediaries, carriers, and other middlemen are always trying to trip you up. But fortunately, we have an answer for that too – tunnel it all through SSL, which you should be doing as a best practice anyway.

The world renowned belshe.com tech blog has taken a beating in the last year as the lead author has been busy working the startup life at Twist.

Fortunately, the nice folks at Fierce Developer were able to whip us into shape, and publish a guest post for me. Let me know what you think!

Fierce Developer: Mobile Networking is Where HTML5 Falls Short

When we were developing SPDY, we wanted to know if SPDY was always faster or just sometimes faster than HTTP. The following chart is what convinced us that SPDY is indeed almost always faster. I didn’t publish this at that time because I didn’t like looking at web performance as a single number. “Is it faster” includes many variables, such as the network simulated (bandwidth & latency), the packet loss rate, the content chosen, and the measurement (total PLT, first PLT, second PLT, or time-to-first-render). You’d really want a whole stack of these charts, rather than just a single combination.

What I like about this chart is how easy it is to compare two protocols. If the two protocols are identical in performance, all points would be on the midline (red). If one protocol is slower, then the points will fall closer to that axis.

Notes about this test:

• Used a static copy of the Alexa Top-300 websites (excluded porn sites), full content, unchanged
• Simulated 2Mbps download, 386Kbps upload, 100ms RTT, 0% packet loss
• No SSL
• The average speedup was ~40% on this test
• This test predates CWND changes in the kernel
• Server was linux, client was chrome on Windows 7

The US debt is a big problem (currently ~$16T, or about ~$50K per person). I’m willing to do my share to help pay it off. To do this, I need the government to commit to a sustainable plan so that I’m not just throwing good money after bad.

Thus I make this promise:

If Washington can make spending match tax receipts, I will donate an extra $1,000,000 to the USA to help pay down the debt.

In the last couple of weeks many people have asked me to comment on guypo’s benchmark blog post, “Not as SPDY as You Thought”.  Guy shared the post with me before he posted it.  Overall, I disagree with his title, but I don’t disagree with his results much, so I haven’t felt pressed to comment.  He tested something that nobody else has tested, and after reviewing his methodology, it’s mostly fine. Some suggestions have been made for improvement, which he was very open to, and we’ll likely see additional test results coming soon.  But his results are not contrary to Google’s or my own results; they’re just a different test.

The reason his results aren’t contradictory is because Guy’s test doesn’t test SPDY page loads.  Guy tested partial SPDY page loads, not full SPDY page loads.  More specifically, he tested this case:  if you upgrade your primary domain, but few of your other domains, your CDN, etc, how does SPDY perform?  This is a perfectly valid case to test – especially when sites may take an incremental approach to upgrading.  And I’m not surprised at all that if you only upgrade half your page to SPDY, that the results are not as good as if you upgrade all of your page to SPDY.

In the report, Guy breaks out domains into “1st party domains” and “3rd party domains”.  He argues that since you don’t have control over the 3rd party content servers, it may not get SPDY-ized, and therefore his test is valid.  Ok – thats a good point.  But how do we define “3rd party”?  I consider “1st party” to be any content which you, as the site owner have control to change directly.  So if you load your main content from www.google.com and your images from images.google.com, those are both 1st party domains.  Unfortunately, Guy’s classifier did not classify them this way.

To understand what I mean, lets take a look at the domains used on a few sample pages and how his test loaded resources from them.  I simply picked three from his test results.   Every page tested is different, but the patterns below are common to many of the top websites.

I hope you can now see why I don’t discredit Guy’s test results.  On these pages, 25-50% of the 1st party controlled resources which could have been loaded over SPDY weren’t loaded over SPDY at all. If you only partially use SPDY, you only get partial results. This is okay to me.

Nobody should think I’m discrediting Guy’s work here.  He’s done a great job with great vigor, and it takes an incredible amount of time to do these tests.  He’s planning to do more tests, and I’m very thankful that he is doing this and that Akamai is letting him do so.

In the next wave of tests, I expect we’ll see that SPDY benefits are increased.  Keep in mind that your average site isn’t going to see the 2x speed boost.   The overall benefit of SPDY is conditional on many factors, and websites today have not yet been tuned for SPDY.  Most sites will see benefits in the 5-20% range (like Google did).   A few will see 50% better.  A few will see worse.  Everyone will benefit from new optimization possibilities, less complex websites, and a more network and mobile friendly protocol. More testing like Guy’s is the key to a better HTTP/2.0.

When you build a mobile app that uses the network, it is instantly clear that your app needs to be robust against all sorts of network failures:

Dealing with these is hard, but not impossible. Applications retry at various levels trading off battery life and user perceived performance all the time. After enough work, you can make the app functional.

But if you try to write an app using HTML5, how do you do this?

You can’t.

The web simply isn’t designed for partial network failures on a web page. Web pages are comprised of hundreds of subresources from multiple sites these days. What happens when CSS #5 out of 32 resources fails to load? What happens when you can’t connect to i.amazon.com even though you already loaded the main content from www.amazon.com? Does your application even know? Generally not. You can trap for some sorts of errors; but the browser will NOT automatically retry on any of these failures for you. Most likely you’ll be left with a web page which renders incorrectly, hangs forever, or throws javascript errors on the whole page because a critical set of code just never got loaded.

Of course, these problems can happen on your desktop, too. But they generally don’t happen as often. And when they do occur, every user easily becomes his own network administrator thanks to the web browser’s handy dandy “reload” button. How many times have you drummed your fingers for a few seconds before reloading a page? Probably a lot! But on mobile, network errors occur *all* the time. Do mobile apps have ‘reload’ buttons? Generally not – users are becoming quite accustomed to apps which can handle their own errors gracefully.

Sadly, I think this is one more nail in the coffin against HTML5 on mobile. Browsers need to be completely overhauled to properly deal with network errors and retries before HTML5 can be a serious contender to native applications.

For a long time, my voicemail wasn’t set up. People got mad at me.

So I setup my voicemail, and now it has a greeting. But I don’t listen to messages. People get mad at me again.

The trouble with voicemail is its just too inefficient. I have to dial-in, wait for it to connect, listen to the rambling, and then act on it. The whole process is quite maddening – 20-30 seconds each! I know that at this very moment I have voicemails from my sister and brother, and I even have spare time. But the mere thought of the painful process makes me not do it. I’d rather write a whole blog post about how I hate listening to voicemail.

Hmm… I wonder how many messages I’ve got queued up?

Microsoft published their SPDY proposal today to the IETF. They call it “HTTP + Mobility”. Here are some quick comments on their proposal.

a) It’s SPDY!
The Microsoft proposal is SPDY at its core. They’ve fully retained the major elements of SPDY, including multiplexing, prioritization, and compression, and they’ve even lifted the exact syntax of most of the framing layer – maintaining SYN_STREAM, SYN_RESET, SYN_REPLY, HEADERS, etc.

It’s a huge relief for me to see Microsoft propose SPDY with a few minor tweaks.

b) WebSockets Syntax
When SPDY started a couple of years ago, WebSockets didn’t exist. Microsoft is proposing taking existing SPDY, and changing the syntax to be more like WebSockets. This won’t have any feature impact on the protocol, but does make the protocol overall more like other web technologies.

Personally, I don’t think syntax matters much, and I also see value in symmetry across web protocols. I do think the WebSocket syntax is more complicated than SPDY today, but its not that big of a deal. Overall, this part of the Microsoft proposal may make sense. I’m happy that Microsoft has presented it.

c) Removal of Flow Control
The Microsoft proposal is quick to dismiss SPDY’s per-stream flow control as though it is already handled at the TCP layer. However, this is incorrect. TCP handles flow control for the TCP stream. Because SPDY introduces multiple concurrent flows, a new layer of flow control is necessary. Imagine you were sending 10 streams to a server, and one of those streams stalled out (for whatever reason). Without flow control, you either have to terminate all the streams, buffer unbounded amounts of memory, or stall all the streams. None of these are good outcomes, and TCP’s flow control is not the same as SPDY’s flow control.

This may be an example of where SPDY’s implementation experience trumps any amount of protocol theory. For those who remember, earlier drafts of SPDY didn’t have flow control. We were aware of it long ago, but until we fully implemented SPDY, we didn’t know how badly it was needed nor how to do it in a performant and simple manner. I can’t emphasize enough with protocols how important it is to actually implement your proposals. If you don’t implement them, you don’t really know if it works.

d) Optional Compression
HTTP is full of “optional” features. Experience shows that if we make features optional, we lose them altogether due to implementations that don’t implement them, bugs in implementations, and bugs in the design. Examples of optional features in existing HTTP/1.1 include: pipelining, chunked uploads, absolute URIs, and there are many more.

Microsoft did not include any benchmarks for their proposal, so I don’t really know how well it performs. What I do know, however, is that the header compression which Microsoft is advocating be optional was absolutely critical to mobile performance for SPDY. If the Microsoft proposal were truly optimized for mobile, I suspect it would be taking more aggressive steps toward compression rather than pulling it out.

Lastly, I’m puzzled as to why anyone would propose removing the header compression. We could argue about which compression algorithm is best, but it has been pretty non-controversial that we need to start compressing headers with HTTP. (See also: SPDY spec, Mozilla example, UofDelaware research)

e) Removal of SETTINGS frames
SPDY has the promise of “infinite flows” – that a client can make as many requests as it wants. But this is a jedi mind trick. Servers, for a variety of reasons, still want to limit a client to a reasonable number of flows. And different servers have very different ideas about what “reasonable” is. The SETTINGS frame is how servers communicate to the client that they want to do this.

I’m guessing this is an oversight in the Microsoft proposal.

f) Making Server Push Optional
Microsoft proposes to make server push optional. There is a fair discussion to be had about removing Server Push for a number of reasons, but to make it optional seems like the worst of all worlds. Server Push is not trivial, and is definitely one of the most radical portions of the protocol. To make it optional without removing it leaves implementors with the burden of all the complexity with potentially none of the benefits.

The authors offer opinions as to the merits of Server Push, but offer no evidence or data to back up those claims.

h) Removal of IP Pooling
The Microsoft writeup eliminates connection pooling, but it is unclear why. Connection pooling is an important element of SPDY both for performance and for efficiency on the network. I’m not sure why Microsoft would recommend removing this, especially without benchmarks, data, or implementation details. The benchmarks clearly show it has measurable benefit, and without this feature, mobile performance for the Microsoft proposal will surely be slower than for SPDY proper.

Conclusion
I’m happy with the writeup from Microsoft. I view their proposal as agreement that the core of SPDY in acceptable for HTTP/2.0, which should help move the standardization effort along more quickly. They’ve also raised a couple of very reasonable questions. It’s clear that Microsoft hasn’t done much testing or experimentation with their proposal yet. I’m certain that with data, we’ll come to resolution on all fronts quite quickly.

I filed my second objection to the Honda Civic Hybrid Settlement this year, and they were nice enough to send me a CD of all the other letters that were submitted to the court.

Scorecard

• There were 3 letters in favor of the settlement
• There were 45 letters opposing the settlement

The Supporting Letters Were From

• The lawyers, Jonathan Cuneo and Nicholas Chimicles (who also filed for fee reimbursement of $2.332M and $2.995M, respectively).
• The mediator of the original settlement, Howard Wiener.
• Rust Consulting, paid for by American Honda Motor.

Summary

• The lawyers who are getting paid millions love it.
• Not a single member of the settlement class wrote in favor of the settlement.
• Honda just wants to pay this ransom and move on.

I’m under no illusion that my letter will have any effect.

It is a measurable fact that Cuneo and Chimicles are poor negotiators. Although damages are in the thousands of dollars per individual, they could only negotiate a $100 settlement for us. This is in spite of the fact that another member of the class, Heather Peters, was able to successfully negotiate $9,867 for herself – nearly 100 times more than Cuneo and Chimicles.

Sadly, Cuneo and Chimicles will not be fired for their lackluster performance. Instead, for this one case, they will each be paid a multi-million dollar sum greater than the average American will earn in his entire lifetime (median salary * 45 years).

Recent SPDY news comes from some big brands: Twitter, Mozilla, Amazon, Apache, Google.

• Mar, 2012
  Twitter now supports SPDY
   
• Feb, 2012
  A draft of the SPDY specification is submitted to the http-bis working group for consideration toward the HTTP/2.0 effort.
   
• Feb, 2012
  Firefox 11 now supports SPDY
   
• Jan, 2012
  Google publishes SPDY Tech Talk, Announces that SPDY with SSL is now faster than HTTP without SSL for Google sites.
   
• Dec, 2011
  Apache mod-spdy enters beta
   
• Nov, 2011
  Amazon launches Kindle Silk browser with SPDY support
   

Looking forward to seeing what comes next!

Mobile Apps use HTTP. But they usually don’t use it to transfer HyperText – rather they are using it to transfer JSON, XML, or other data formats. Just like their web counterparts, secure transmission is desirable.

But, if you ever trace a fresh SSL connection, you know that its a nasty process:

• DNS
• TCP handshake
• SSL handshake
• Server sends certificate
• DNS to CA
• TCP to CA
• OCSP to CA
• Finish SSL handshake
• Finally do what you wanted to do….

SSL is designed so that you can pick up some random certificate and check it dynamically. This is a good thing for the web, where the user coasts from site to site, happily discovering new content which needs new validation.

But this process is pretty costly, especially on mobile networks. For my own service, I just did a quick trace over 3G:

• DNS (1334ms)
• TCP handshake (240ms)
• SSL handshake (376ms)
• Follow certificate chain (1011ms) — server should have bundled this.
• DNS to CA (300ms)
• TCP to CA (407ms)
• OCSP to CA #1 (598ms) — StartSSL CA uses connection close on each!
• TCP to CA #2 (317ms)
• OCSP to CA #2 (444ms)
• Finish SSL handshake (1270ms)

With the web, this verification process makes some sense – you ask the CA to be your trust point and verify that he trusts the certificate provided.

But why do this with a mobile app? Your mobile app has a lot of trust with it – they downloaded it from you, its signed by Apple, and if the code has been compromised, well, heck, your app isn’t really running anyway.

What we really want for mobile apps is to bake the server’s certificate into the app. If the server’s certificate needs to change, you can auto-update your app. In the example above, doing so would have shaved about 3000ms off application startup time.

The downside of this is that if your certificate changes, your app won’t verify. Then what to do? Simple – force an auto update.

There is another advantage to this approach. If you can verify your own certs, you don’t need a CA provided certificate anyway. These silly 1-2 year expirations are no longer necessary. Sign your own cert, and verify it yourself. Since our CAs have been getting hacked left and right in 2011, this is probably even more secure.

PS: SSL is hard. In this one trace, I can spot at *least* 3 low-hanging-fruit optimizations. I haven’t mentioned them, because they are pervasive everywhere on the net. There are errors here at every level – the client is missing opportunities, the server is missing opportunities, and the CA is missing opportunities! It’s no wonder that SSL is slow. The chance that your combination of client + server + CA will have some dumb performance bug is ~99%.

I haven’t read about this anywhere, but in Jan 2012, Chrome 16 was the World’s Most Popular Browser. A minor stat, but a testament to modern update policies. You won’t read about this in any press release, but this is the #1 reason why Chrome’s security is better than any other browser.

I had a frustrating day. I’m writing some simple ajax/xhr tests, and I can’t get the browser to issue my requests:

Origin http://localhost is not allowed by Access-Control-Allow-Origin

Searching on Google, it’s clear that lots of other people are having the same problem, but nobody has an answer – the browser nazis are being overzealous, protecting us from localhost – really?

Fortunately, chrome has a workaround. Use this commandline:

chrome.exe –disable-web-security

And you can get your job done.

Just spent 3 days waiting for turnaround on a contract where the other party’s counsel did nothing but twiddle the bits on the confidentiality clause. What a waste of time, productivity, and legal bills.

Don’t get me wrong – the other lawyer is not the only one at fault here. My lawyer is equally to blame.

What the hell? Why is it that my lawyer, and the other lawyer both thought they needed to reinvent a confidentiality clause? Can’t we just create one, have it ratified by the courts and live with it? Why do two lawyers get to rack up wages at $300/hr to modify 2 sentences that they’ve modified a thousand times before?

Lawyers who don’t work on fixing this problem are the problem. They all need to get together, settle this, and stop raping America.

Heather Peters has become semi-famous this week because of her small-claims suit against Honda. Honda reached a class-action settlement because the gas mileage of the Civic is simply not up to par. This matches my own experience. Aside from the first few months owning the car where I was both actively hypermiling and also driving exclusively highway miles, the gas mileage in my car has been disappointing. My hybrid gets ~38mpg right now, a far cry worse than the claimed 50mpg that I should be getting.

The class action lawsuit was a joke. The lawyers walked away with millions, while the consumers were compensated with $100 and a video. And each Civic Hybrid consumer has thousands of dollars in damages. Worse, the settlement bars anyone who even complained about the settlement from receiving the settlement. I complained, of course, and my punishment is that I didn’t get the $100.

Since there seems to be renewed interest in this whole affair, here is the full text of the letter I submitted to the court regarding the class action suit back in December 2009.

UNITED STATES DISTRICT COURT
CENTRAL DISTRICT OF CALIFORNIA
EASTERN DISTRICT – RIVERSIDE

JOHN TRUE, et al.,
Plaintiffs, Case No.: 5:07-cv-00287-VAP-OP

v.

AMERICAN HONDA MOTOR CO., INC

Defendants.

OBJECTIONS TO CLASS ACTION SETTLEMENT

I am a member of the settlement class for Case No. 5:07-cv-00287-VAP-OP (True vs. Honda Motor Company Inc.). This letter officially declares my objection to the settlement on the grounds that this settlement is not in the best interest of the members of the settlement class.

Background
This lawsuit was brought against American Honda Motors (AMH) because the fuel efficiency of their cars does not match the advertised claims. Evidence has been presented that the cars were advertised to achieve 50MPG while only actually achieving 32MPG for many drivers. Drivers of these cars often purchase them specifically because of the alleged low cost of operation. When the gas mileage fails to achieve advertised claims, the consumer is left to pay the difference. I believe the difference is an accurate representation of the damages caused by AMH.

The Honda Civic warranties apply to various parts of the car for 50,000 to 150,000 miles, and consumers are sold the car with the expectation that 100,000 miles is reasonable
. Gasoline in California has averaged ~$3.20 per gallon over the last 3 years
. Using these figures, if the car only achieved 32MPG instead of 50MPG, the per-consumer damages are ~$3,600.

Objections
I have 4 objections to this settlement.

Objection #1: Failure to correct damages for the settlement class
Given the magnitude of damages to consumers, the settlement offer simply is not in the best interest of the settlement class. The settlement reimburses the members of the class for less than 2% of the damages incurred.

In the settlement, members are given 3 options:
a) Sell your hybrid, and get a $1000 coupon on another Honda (not a hybrid).
b) Give your hybrid to a family member, get a $500 coupon on another Honda (not a hybrid).
c) Get a $100 check.
All 3 options include a video for how to get better gas mileage.

Even if the settlement member wanted a new car (which many do not), the $1,000 rebate still would still be less than one-third of the damages incurred due to unexpectedly low cost-of-operation.

But worse, for most consumers, the $100 settlement is so small that it does not offset the damages at all. At the same time, this settlement greatly reduces the consumer’s claims against Honda, and may even void the consumers current warranty. Why would any consumer sell this privilege for a mere $100? (see the second objection #2).

Objection #2: Overreaching settlement releases
As part of this settlement, all members of the settlement class release all claims against AMH regarding anything related to the fuel economy of the vehicle. Specifically, the settlement covers anything

“involving, based on, relating to, arising out of or in any way connected with, directly or indirectly, the advertising of the fuel economy or m.p.g. of the HCH…”

With regard to existing warranties, it states

“Nothing in this Agreement shall be interpreted to modify or diminish the manufacturer’s limited warranty with respect to a Class Vehicle; provided, however, that any such claim of breach of any warranty or any extended warranty based on the advertising or representations made by AHM with respect to fuel economy, mileage or m.p.g are in fact released.”

This clause is completely overreaching! For instance, Honda warrants the batteries in the vehicle for 50-150K miles. When I was sold the vehicle, the salesperson told me that the batteries would not need replacement for 150K miles. However, as the batteries get old (much like the batteries in your cellphone), they hold their charge for diminishing amounts of time. The car will still operate just fine even with diminished battery life, but the MPG rating will drop significantly! I fear that if I accept this settlement, and then I go to Honda to get help with replacement of my previously-under-warranty batteries, I will no longer have a claim. Honda will be able to say that my car is working fine, and that the battery problem related to MPG was Released as part of this settlement.

As such, this contract specifically DOES alter my warranty, despite claims to the contrary.

Objection #3: The retaliatory nature of the settlement against its members.
As part of this settlement, any member of the settlement class that objects to the settlement is specifically barred from collecting any benefits if the settlement is accepted by the court. This clause is clearly not in the best interest of the settlement class, and is actually retaliatory in nature.

It is clear to me that the damages I have personally incurred due to AMH’s wrongful advertising are far in excess of the $100 settlement. Purchasing a new Honda has no appeal to me. I do not want to purchase a new car from Honda at any time in the future, and even if I did, I would like to get a hybrid. As such, options (a) and (b) simply do not make any sense.

But the negotiators of this settlement have defined only 3 options, and I am required to either accept them or be ignored from any settlement result. This clearly only serves the interest of the legal staff and does not serve the interest of the plaintiffs.

Objection #4: Negotiation of excessive attorney compensation at the expense of the settlement members
I recognize that the attorneys have worked hard on this case for the last year. While they are pushing for a quick settlement which will enable them to receive a hefty payout of nearly 3 million dollars, they have failed to negotiate a settlement which actually addresses the original problem in the claim. If they believe that settlement members would benefit by accepting a plan which voids their warranty and only recoups 2% of the damages, then the attorney fees should also be diminished – I propose they should be set to 2% of the requested fees, or $92,187.

My Status in the Settlement Class
The following responses are the required responses as per the proposed settlement in progress.

(i) Name address and phone number
Michael Belshe
[DETAILS OMITTED FROM BLOG]

(iii) Grounds for objection
Stated above.
(iv) Proof
Stated above.
(v) No persons will be called to testify
(vi) I do intend to appear at the Final Approval Hearing, schedule permitting.
(vii) I have appeared in no other cases as a settlement objector or counsel in the past 5 years.

Recommendations
I have 5 recommendations for the settlement:
(a) The negotiators should compute a reasonable figure for damages to each consumer. (I computed $3,600).
(b) The negotiators should negotiate a settlement for at least 50% of those damages.
(c) Objecting to a proposal must not cause members to be excluded from a settlement.
(d) The settlement must re-address the warranty voiding effects mentioned above.
(e) The attorney fees should be in-line with what settlement members receive in the settlement. If the settlement members recoup 50% of their damages, then the attorneys should only collect 50% of their $2.95M fee as well.

Respectfully,

Michael A Belshe