Coffee on the Keyboard

Mistakes: Importing Data with MySQL

2009-11-03T01:53:00Z

I spent the better part of today trying to import—in various attempts—between 800 MB and 3.1 GB of data into a local MySQL server. The whole time I was Doing It Wrong™. Now I feel like this:

I was using pipe viewer, which obscured the error for most of the day. (If there’s a way to stop pv from eating messages to stderr, please let me know!) But eventually I figured out it was hitting the max_allowed_packet limit and my import was dying. I was using the MySQL docs as my reference point, and they imply that the following should work:

$ mysql --max_allowed_packet=32M db < data.sql

I know now that this does not work. And I feel like a moron. If you read carefully, which I did not, you’ll see that they’re talking about server startup, not client startup, even though the example uses “mysql” instead of “mysqld”.

When I stopped using pipe viewer, I got this series of errors:

$ mysql --max_allowed_packet=16M db < data.sql
ERROR 1153 (08501) at line 175458: Got a packet \
    bigger than 'max_allowed_packet' bytes
$ mysql --max_allowed_packet=128M db < data.sql
ERROR 1153 (08501) at line 175458: Got a packet \
    bigger than 'max_allowed_packet' bytes
$ mysql --max_allowed_packet=256M db < data.sql
ERROR 1153 (08501) at line 175458: Got a packet \
    bigger than 'max_allowed_packet' bytes
$ mysql --max_allowed_packet=1G db < data.sql
ERROR 1153 (08501) at line 175458: Got a packet \
    bigger than 'max_allowed_packet' bytes

Now here’s the thing, data.sql was just shy of 80 MB. So obviously something was wrong.

A quick edit to /etc/my.cnf:

[mysqld]
...
max_allowed_packet=32M
...

Then all I had to do was restart mysqld, and the next time I tried:

mysql db < data.sql

It finished without a problem.

7 hours and a massive facepalm later… I hope this keeps someone from wasting the same amount of time I did on such a dumb mistake.

Getting Involved with Mozilla Web Dev

2009-10-12T21:09:53Z

We’ve got a new page on the Mozilla Web Dev blog about getting involved.

On it, you’ll find information about some of our biggest projects, including Addons and Support. There are links to the project pages on the Mozilla Wiki, names of people to contact including IRC handles, and links to the source, tests and bug lists.

This page is a starting point. At the bottom you’ll find a link to a list of (hopefully) all the websites we maintain.

Mozilla Web Development is Open Source

2009-09-29T04:28:17Z

At Mozilla, our web development projects are open source, publicly available, and interested in community contributions.

This is one of those things that makes absolute sense when you hear it, but you may have never heard, or thought, of it before.

It’s not terribly well-publicized—that’s something Mike Morgan and I are going to start working on this week—but there are ways you can get involved:

There’s info about most, if not all, our current projects on the Mozilla Wiki.
An advanced search in Bugzilla can tell you a lot about what’s going on.
You can find web dev people on IRC in #webdev, #sumodev, and a few others. (I’m jsocol on irc.mozilla.org.)

Why would you want to contribute to Mozilla web development projects?

You care about the web, and making it better.
Our websites are used by millions of people.
Because of that, we work at scales most developers don’t usually get to see.

Hopefully I’ll have something to add by the end of the week, but in the meantime, come on over and say “hi” on IRC!

(Don’t) Close Firefox with Last Tab

2009-09-25T07:04:55Z

Prior to Firefox 3.5, the default behavior of closing the last tab was to leave a blank tab. That changed in Firefox 3.5, and the default behavior is now to close the whole window with the last tab. If you’re on Windows or tend to close tabs with Ctrl-W, like me, this can be pretty annoying if you forget about it, what with reopening the browser and all.

Fortunately, it’s easy to change.

Venture into about:config. Go to the address bar and enter “about:config” and press enter. You’ll see a long list and at the top, a text box called “Filter:”

Buyer Beware here. You probably saw a warning when you tried to go to about:config. That’s because you can significantly alter the behavior of Firefox here, and you need to be either very careful about what you change, or very confident in your ability to fix it.

Enter “tabs” into the filter and you’ll see a list like the picture above. There may be more items depending on the addons you have installed. Now look for “browser.tabs.closeWindowWithLastTab” and double click it. It should turn bold, and in the right columns it will say “user set” “boolean” “false”.

That’s it!

Of course, if you’re not comfortable mucking about in about:config, or you also want to restore the close button to that last tab, there’s an addon* for that.

* Of course, I haven’t used, and can’t vouch for, that addon, but it’s there.

Upgrade Flash in Firefox

2009-09-23T03:23:05Z

If you upgraded Firefox lately, you might have seen a message on the start page urging you to upgrade Flash. As of yesterday, something like 12 million people had clicked through to upgrade. That’s a great number.

However, when you got to the Adobe site, and clicked the “Agree and Install” button, instead just saving an installer file, you might have seen this:

Right there? That’s enough to deter me from this whole process, so I can definitely understand if it scared you off. But this is an important update, a security update, so you need to do it.

Fortunately, it’s very easy to get around this annoying development in Flash. (Their hearts are in the right place, automatic updates, but their implementation leaves something to be desired. Of course, the Adobe Updater makes you quit Firefox to update Photoshop, so this clearly isn’t their strong suit.)

For starters, head to Adobe’s Get Flash page. If you see that annoying bar, click on the [x] in the right corner:

Now, more towards the middle of the page, look for a link that says “click here to download.”

That will start downloading the normal installer we’ve come to know and love. Yeah, you’ll still have to restart Firefox, but you were going to need to do that, anyway.

There you have it, avoiding Adobe’s strange new extra software and getting Flash up-to-date with the latest (really, really important) security fixes.

NextUp

2009-09-02T04:30:20Z

I never properly introduced my latest(?) little side project: NextUp.info.

NextUp is a very straightforward app that helps you create an agenda, then stick to it. You enter a list of topics (or, say, speakers) and how long they’re allowed to run. When you create the agenda and start the meeting, the topics count down, much like PTI.

The back story is pretty simple: I was on a project with at least four distinct teams, so we had weekly “status update” meetings. These really should have run for no more than 20 minutes, but we were lucky to get out in under two hours. So, NextUp came to mind.

The name was, essentially, the only relevant domain available. Don’t lie, you’ve probably picked a title based on available domains. It’s becoming a real problem.

It wasn’t very pretty or usable until recently, when my friend Ben (no, not that one) stepped up and helped me out with a UI and design. IE users will not get the pretty text shadows, and I’m pretty sure it’s just broken in IE6. Sorry, upgrade, or tell your IT department to do it.

Excitement about HTML5: Forms

2009-08-22T17:38:50Z

I shouldn’t imply that I don’t like HTML5. I don’t like certain parts of it—the redundant new elements that add no functionality and are of little use except to A List Apart. But other parts of the spec are very exciting. Forms, in particular, are getting a much-needed facelift.

Forms will be able to require the user-agent to do pre-submission validation. Obviously we can’t rely on this for security reasons, but it will save us from writing JavaScript validation scripts and give users a better experience with our forms.

And the validation is fairly robust. <input/> elements gain an attribute called pattern, which accepts a simple regular expression, like "[A-E][0-9]{7}".

But you probably won’t need the pattern attribute very often, since there is a whole slew of new types that the UA will be expected to provide controls for, and validate. Crucial types, like email and url, and difficult types like datetime-local and color.

jQuery UI is great, but it doesn’t compare to this kind of power.

I only realized the sorry state of our current form controls very recently, after getting an iPhone. Built-in form fields on the iPhone can trigger different input methods. A field, like ZIP code, that accepts only numbers, opens the keyboard to the numbers first. A field that wants a URL can open a keyboard with a .com button to save time. But web pages can’t do this. They can only say that an input accepts arbitrary text.

For mobile or touch-enabled UAs, knowing that you can open a keypad instead of a full keyboard is a great step forward. For everyone, having date and color pickers built into the UA means saving both developers’ and users’ time. Less time to build and less time to download, consistent experience in the UA across web sites.

Forms also get a new event, input, which is a little like the current change event, except you don’t need to wait for the element to lose focus for the event to fire. That’s just useful.

Controls get new methods, stepUp() and stepDown(), to enable, for example, very fast forward/backward selection on a date input.

The last fun addition I’ll mention (go read the links, they’re all to the same page) is the autocomplete attribute. It doesn’t do what you hope it does. What it does is specify to the UA that it should not remember and suggest values for this input. It lets the UX designer decide whether to use Firefox or IE’s built-in autocomplete on a per-field basis. Useful. (Not as useful as, say, an autocompleteUrl attribute that could load some JSON or XML, but still useful.)

Building forms is going to be so, so much better once these are widely support.

What Time Is It There?

2009-08-11T04:57:08Z

We just pushed an update to What Time Is It There?, so I thought I’d take the time to mention it here, since I’ve already done so on Twitter. You can now link to a time.

What Time Is It There?, or WTIIT, is a quick, visual reference card for US time zones. It’s a collaborative effort of Ben Lew and I. And when I say “collaboration,” I mean it was 100% Ben’s idea, and I refactored some JavaScript for him.

Ben’s original version showed you the current time in six time zones. My first update completely rewrote the JavaScript and added the ability to specify a time, say “at 9:15pm eastern” or “in 2 hours 30 minutes”. Let’s call that version 1.0, release to the wild.

Rather quickly I noticed a bug causing problems with when “PM” was capitalized and there was no space before it (eg: “at 5PM” instead of “at 4pm” or “at 1 PM”). So I sent him a patched version that also changed the behavior when no “am” or “pm” was given. Version 1.0.1 assumed that hours between 12 and 6 were in the afternoon unless you told it otherwise, while 7-11 are assumed to be morning.

Last night I sent Ben a new patch, version 1.0.2, that adds what I think is an important new feature: you can now link to a time.

When you enter a time and hit enter or click outside the box, the URL is updated with that time. If you copy the URL, you can save or send it and when you return to that URL, the page will start off with whatever time you saved.

WTIIT 1.0.2 includes two additional changes. “a” and “p” now work if you don’t like typing the “m”, and—finally!—the pesky 12 noon/midnight bug is squashed. Previous behavior with 12 o’clock was, shall we say, wonky at best, and often plain wrong. (It could literally display the opposite of what you typed.) Now, it parses noon and midnight correctly.

Just thought I’d share.

Reservations about HTML5

2009-08-08T23:21:22Z

A day or so ago, Adrian Bateman, from Microsoft’s IE team, posted his team’s thoughts on the current draft of the HTML5 spec.

Reading it is brutal. Bateman takes issue with basically everything added since HTML4. He goes through and individually criticizes many of the new tags, sometimes with extremely detailed, multi-paragraph critiques. I guess this is what happens when you’re not sufficiently involved at the beginning.

Of course, there is still plenty of time to complain, since the HTML5 spec won’t reach its final stage until 2022.

Bateman and the IE team, even while sounding like they don’t even want HTML5, do bring up a few things that have been bothering me about the spec.

Let me be very clear: I think new tags like <audio> and <video> are wonderful. Breaking the Adobe monopoly is great. There are still some issues (refusing to specify a codec meaning you can’t build support into the browser, for instance) but those types of tags are going to help push the web to a better place.

The parts that bother me are the new, highly touted “structural” tags, like <header>, <footer>, <section>, and worst, <article> and <aside>.

My issue is that all of these tags are perfect semantic additions, if all you want to do is put magazine articles on the web. These are tags that represent a current prevailing paradigm in text-focused web design that has been strongly influenced by print design and layout. They all have a use on A List Apart, and on most blogs, but besides <header>—or possibly <nav>—can you see any of them in Gmail or Google Calendar?

Ryan Doherty already demonstrated a semantic version of Gmail, and didn’t need a single HTML5 tag to do it. (But please, please, don’t assume I’m putting words in Ryan’s mouth about HTML5. He’s also used the <audio> tag to great effect.)

Issues with the ambiguity of certain new tags aside, these tags privilege, even codify, a certain paradigm in design. For lack of a better word, the “wall of text” style. To that end, I worry that (a) they do nothing to help the cause of ~~emerging~~ hell already mainstream web application development—the Rich Internet Application— and (b) they may actively discourage designers from trying new, paradigm breaking ideas.

I imagine a conversation between a designer and the developer tasked with implementing the design. Maybe this isn’t a “great” developer, but honestly, how many of those are there? The conversation ends: “Well, there just isn’t a tag for that.” Back to the drawing board.

What I like about the <div> and <span> are their flexibility. By themselves, they’re nearly meaningless, just a block-level or inline element, respectively. What makes them work is the way we’ve developed the use of the id and class attributes.

What HTML5’s new tags lack is extensibility, just like HTML4. They freeze us at a moment in time (about 5 years ago) and in design. To that end, I reiterate John Allsopp’s call for new attributes, like role from XHTML, rather than new tags. Do we really need five new block-level elements? Or should we allow some sort of extensible mechanism to create, in effect, an infinite number of new block-level elements?

Since I suggested it, here’s how I imagine it working: let’s say HTML5 adopts the role attribute, and it’s applicable to everything. Now, instead of XHTML’s short list of values for role, let’s define a list of values, but allow that list to be extended. So, our common values—like header, footer, navigation, menu, section, article, frame, search, banner, posssibly data and interactive, I’m sure we could come up with far more even now—have some defined meaning, but they are not the only possible values.

The tricky part is: how do we go about defining a new role? The simple, naïve solution is allowing people to use anything, and then use CSS selectors and JavaScript to make it work. While this opens up the extensibility to the greatest number of people, it also makes it fairly meaningless, and doesn’t really expand on the id/class system we have now.

The other end of the spectrum is requiring all roles to have a definition that is somehow meaningful to the user-agent. The XML way to do this would be roughly to allow the role attribute to exist in any namespace and still be interpreted, providing a definition for the UA somewhere on the web. Something like this:

<html ... u:xmlns="http://my.dtd/url">
...
<div u:role="mycustomrole" role="interactive">

This, of course, limits the number of people who can create, and host, definitions for roles. It is also incompatible with the HTML-style variant of HTML5 (as opposed to XHTML5, the XML-style variant). So let’s throw this out now.

The other method I can see off the top of my head is a hybrid, and puts much of the onus on the UA to be smart, and on the community to share.

There will need to be a lot of discussion among browser vendors on what “meaningful to the UA” means, and what kind of definitions are necessary from that, but let’s assume that they’ve all come to some sort of consensus—or, since I’ve always thought the consensus model was flawed, that some common format has emerged. There is a file format that can be placed on the web.

This is a hybrid scheme: if all I want is to label some elements with a new role, I can put anything in there. But if I want something meaningful to the UA, with a definition file, I can add it as a <link> element in the header.

<link rel="roles" type="application/role+xml" href="http://mysite.com/myroles"/>

MIME-types and formats are left as exercises to the reader.

Now, here’s where the sharing comes in. Let’s say my friend or follower wants to use a similar role, and thinks mine is close enough. He can then put the same <link/> tag. Hopefully, a smart UA, would have cached the definition file, do a conditional GET request, get a 304 Not Modified from my server. If my definition file gets too popular for my server to handle, I could perhaps move it over to Google Code and have my server issue a 301 Moved Permanently, which would hopefully convince UAs to stop pinging me.

There is some question about conflicting role definitions. What makes a “unique key,” is it a combination of a role and a URL? Can I define multiple roles per URL, or only one? If there are multiple roles per URL, can I, as a developer, mix and match? Which definition would take precedence? These are not trivial questions, and would have to go up for debate in WHATWG.

This is just one idea for an extensible method of building on HTML that doesn’t lock us into a single design paradigm. What I don’t like about the current HTML5 spec is that a lot of is is neither extensible nor forward-thinking. I don’t know how many of you plan to keep making the same old blogs and webzines for the next 13 years, but I, for one, would like to move forward.

EzGz: WordPress GZip Plugin

2009-07-27T15:33:44Z

By default, WordPress (at least in the current generation, 2.8) does not enable any kind of compression on its output. Some plugins, like WP Super Cache, add gzip compression, but that’s an awfully big plugin if all you want is compressed output.

So I wrote EzGz this afternoon. It is, by far, the simplest plugin I’ve ever written. The actual source is probably two orders of magnitude smaller than the comments (plugin details and the license).

All EzGz does is enable the built-in PHP gzip/deflate handler. This does require output buffering, so it’s not right for everyone. Some themes or other plugins may already do this.

I’ve tried using the zlib.output_compression ini setting, but on my server, with zlib working otherwise correctly, the setting had no effect. So I stuck with the output buffering.

You can download it, or check it out of svn, then turn it on and see if it works.