CS50 Blog

This is a job for cron!

Sun, 24 Oct 2010 12:05:00 -0700

So it turns out that last screenshot of cs50.tv's guestmap in 10/13's post should have had a lot more markers on it! Recall that we use Google Spreadsheets to store visitors' names and hometowns. It turns out it's not sufficient to write (or talk about) a script that gecodes those hometowns, saving the resulting GPS coordinates back in the spreadsheet itself. You actually have to remember to run that script!

Truth be told, I thought I'd set up a "cronjob"on our server whose purpose in life was to execute that geocoding script nightly. Unfortunately, it seems I never did. (Or, in my defense, I did but set it up under a username that I later deleted. Though that probably still makes it my fault.) But no matter, what a perfect opportunity to talk about cron!

cron is a utility generally found on Linux systems (and others) that allows you to schedule the execution of commands. For instance, if you've written a command-line program called foo, you can schedule foo to be executed every 5 minutes without having to log into the server yourself every 5 minutes and run it manually. Alternatively, you could schedule foo to be executed once nightly, or once weekly, or, really, anytime you'd like, however often you'd like. cron is often used by the system itself to, say, rotate logs (i.e., rename them so a file doesn't get too big). In CS50, we use cron (via rsnapshot) in order to back up students' source code every 5 minutes to save them the inevitable anguish of accidentally deleting some file. (The disk space it's cost us to maintain so many backups has been well worth the tears saved!)

Anyhow, using cron is wonderfully simple. You essentially run a command like

crontab -e

that allows you to edit (per the -e) a text file (otherwise known as a "cron table"), each of whose lines represents a scheduled command. In order to enable daily geocoding of our Google Spreadsheet, I added the one-liner below to the webserver's own crontab, where /path/to/geocode is the full path to the command-line geocoding script I wrote:

@daily /path/to/geocode

It turns out that my one-liner is actually short-hand notation for

0 0 * * *

which means, right to left, that this command (/path/to/geocode) should be executed every day of the week, every month, every day of the month, when the clock strikes 0 hours and 0 seconds, where * denotes "every." In normal-people terms, it means "run this command every night at midnight." To be honest, I almost always forget which number means which, but Wikipedia has a wonderful cheatsheet. Allow me to excerpt:

.---------------- minute (0 - 59) 
 |   .------------- hour (0 - 23)
 |   |   .---------- day of month (1 - 31)
 |   |   |   .------- month (1 - 12) OR jan,feb,mar,apr ... 
 |   |   |   |  .----- day of week (0 - 6) (Sunday=0 or 7)  OR sun,mon,tue,wed,thu,fri,sat 
 |   |   |   |  |
 *   *   *   *  *  command to be executed

I should confess that I've actually told a little white lie in the interests of simplicity. Our cronjob is actually

@daily /bin/nice /path/to/geocode

which means that the command should be "nice'd" before it's run. Like cron itself, nice is a utility found on most Linux systems (and others) that allows you to raise and lower commands' priorities (i.e., how much CPU time they get). By default, nice lowers a command's priority, so our cronjob really tells the system to "run this command every night at midnight but run this command at a lower priority so it doesn't slow other, more important things (like file downloads) down."

Odds are, if you have a shell account on a Linux system (that you can access via SSH), you have the ability to schedule cronjobs, at least if the system administrator allows you. If you instead have an account with a third-party Web host that only provides you with a Web-based panel, you may be able to schedule cronjobs via that panel's GUI in a slightly more user-friendly way.

Anyhow, thanks to cron (and my actually using it), cs50.tv's guestmap is indeed now updated nightly, and it currently looks like this!

Permalink | Leave a comment »

Mapping CS50

Sun, 03 Oct 2010 19:30:00 -0700

For a while, cs50.tv (and its sister sites) have had a "guestmap," whereby visitors are invited to record their names and hometowns on a Google Map, if only for the fun of it. To see the map, head to http://cs50.tv/ and click Guestmap in the site's top-left corner. (Feel free to add yourself!) For the impatient, here's a screenshot of what it looks like at the moment:

When we decided (spontaneously one night) to implement this guestmap, we weren't sure if anyone would notice (or care!) if we did, so we didn't want to spend too much time on its implementation. In particular, we didn't want to bother setting up an entire database (e.g., MySQL) just for this one application. Simpler would have been to store folks' submissions in a local file (e.g., CSV, SQLite, or XML). But the plan was to start replicating cs50.tv's content across multiple servers, in which case we'd then have to keep copies of that file synchronized across multiple servers. Doable, but somewhat annoying. If we used a local file as our database, we'd also have to worry about concurrent writes (as when two visitors happen to submit their names and hometowns simultaneously, the result of which might be to lose one of the visitor's data). Also doable, and not very hard (thanks to functions like flock). But we'd still have to implement the form via which we ask for visitors' names and hometowns. (And at this point in the story, it was probably already midnight or so.)

If only there were someone to whom we could outsource both the form and the database...

Turns out there was! Odds are you're familiar with Google Spreadsheets, which is essentially Google's web-based answer to Microsoft Excel. It's actually wonderfully useful, in large part because it makes it so easy to share (and edit simultaneously) spreadsheets with other people. But with Google Spreadsheets can you also create web-based forms whose submissions end up in a Google Spreadsheet: each row represents a submission, and each column represents a visitor's answer to a particular question. And you can then share that spreadsheet with yourself (or others) in a variety of formats, among them CSV.

Nice! Even without writing any code, I was almost finished implementing the guestmap! (Okay, admittedly, it took a few minutes to create the form and share it with myself as CSV, but way fewer minutes than it'd have taken to implement all that myself.)

So, at this point in the story, I had two URLs: one for the form and one for the CSV file. My plan was (1) to embed the former in an iframe on cs50.tv so that folks could add themselves to the guestmap without leaving cs50.tv itself and (2) query the latter anytime someone wanted to look at the map. All that remained was to write the code that queries the CSV and generates the map.

Thanks to the Google Maps API, this part, too, was amazingly easy. To embed a Google Map in one of your own pages, you first need to include the API in your page's head, as with:

<script type="text/javascript" src="http://maps.google.com/maps/api/js?sensor=false"></script>

You then need to provide a container for it in your page's body, as with:

<div id="map" style="height: 100%; width: 100%;"></div>

Lastly, you need to tell Google to transform that container into a map, as with:

new google.maps.Map(document.getElementById("map"), {
 center: new google.maps.LatLng(42.37411257777324, -71.11905097961426),
 disableDefaultUI: true,
 mapTypeControl: false,
 mapTypeId: google.maps.MapTypeId.HYBRID,
 navigationControl: true,
 zoom: 2
});

Of course, that map won't have any markers on it until you plant them yourself. Planting a marker, though, requires knowing the GPS coordinates of each visitor's hometown. Probably not something most people know. So, how can we geocode (i.e., convert) each visitor's city, state, and country (that he or she inputted via the Guestmap's web-based form) into a latitude and longitude?

Well, the Google Maps API does provide a JavaScript geocoding service, but using that service would mean that every visitor's hometown would need to be geocoded every time someone pulls up the guestmap by that someone's own browser. Certainly not the most efficient approach. Plus, Google appears to limit the number of geocoding requests a browser can make in a short period of time, the result of which is that some visitors wouldn't end up marked on the map. (We, um, learned that the hard way.) Better to do the geocoding server-side and remember each visitor's coordinates so that we needn't query for them again.

As luck would have it, Google also provides a web service with which you can do exactly that. Via Google's Geocoding API can you ask for the GPS coordinates of some location (e.g., 33 Oxford Street, Cambridge, Massachusetts) with code like the below:

$o = json_decode(file_get_contents("http://maps.googleapis.com/maps/api/geocode/json?address=33+Oxford+Street,+Cambridge,+Massachusetts+02138&sensor=false"));

I called json_encode in the above because what's returned is a JSON object like the below:

{
  "status": "OK",
  "results": [ {
    "types": [ "street_address" ],
    "formatted_address": "33 Oxford St, Cambridge, MA 02138, USA",
    "address_components": [ {
      "long_name": "33",
      "short_name": "33",
      "types": [ "street_number" ]
    }, {
      "long_name": "Oxford St",
      "short_name": "Oxford St",
      "types": [ "route" ]
    }, {
      "long_name": "Cambridge",
      "short_name": "Cambridge",
      "types": [ "locality", "political" ]
    }, {
      "long_name": "Cambridge",
      "short_name": "Cambridge",
      "types": [ "administrative_area_level_3", "political" ]
    }, {
      "long_name": "Middlesex",
      "short_name": "Middlesex",
      "types": [ "administrative_area_level_2", "political" ]
    }, {
      "long_name": "Massachusetts",
      "short_name": "MA",
      "types": [ "administrative_area_level_1", "political" ]
    }, {
      "long_name": "United States",
      "short_name": "US",
      "types": [ "country", "political" ]
    }, {
      "long_name": "02138",
      "short_name": "02138",
      "types": [ "postal_code" ]
    } ],
    "geometry": {
      "location": {
        "lat": 42.3787770,
        "lng": -71.1168170
      },
      "location_type": "ROOFTOP",
      "viewport": {
        "southwest": {
          "lat": 42.3756294,
          "lng": -71.1199646
        },
        "northeast": {
          "lat": 42.3819246,
          "lng": -71.1136694
        }
      }
    }
  } ]
}

And so we can get at the GPS coordinates we want with code like the below:

$lat = $o->results[0]->geometry->location->lat;
$lng = $o->results[0]->geometry->location->lng;

And we can then store those coordinates somewhere. (We happen to use the Google Data Protocol in order to write them back to our own Google Spreadsheet in a column that the web-based form itself doesn't touch.)

With all those pieces now assembled, we render the map itself with code like the below, wherein CSV is the URL of our Google Spreadsheet:

<!DOCTYPE html "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" style="height: 100%;">
  <head>

    <link rel="stylesheet" type="text/css" href="http://yui.yahooapis.com/2.8.1/build/reset-fonts/reset-fonts.css" />

    <meta http-equiv="content-type" content="text/html; charset=utf-8"/>

    <script type="text/javascript" src="http://yui.yahooapis.com/2.8.1/build/yahoo-dom-event/yahoo-dom-event.js"></script>
    <script type="text/javascript" src="http://maps.google.com/maps/api/js?sensor=false"></script>
    <script type="text/javascript">
    // <![CDATA[

        YAHOO.util.Event.onDOMReady(function() {

            var gmap = new google.maps.Map(document.getElementById("map"), {
             center: new google.maps.LatLng(42.37411257777324, -71.11905097961426),
             disableDefaultUI: true,
             mapTypeControl: false,
             mapTypeId: google.maps.MapTypeId.HYBRID,
             navigationControl: true,
             zoom: 2
            });

            // resize as needed
            YAHOO.util.Event.addListener(window, "resize", function() {
                if (typeof(gmap) !== "undefined")
                {
                    google.maps.event.trigger(gmap, "resize");
                }
            });

            // prepare infowindow
            var infowindow = new google.maps.InfoWindow();

            <?

                // get subscribers
                $rows = array();
                $handle = fopen(CSV, "r");
                fgetcsv($handle);
                while ($row = fgetcsv($handle))
                    $rows[] = $row;
                fclose($handle);

            ?>

            <? foreach ($rows as $row): ?>

                <? if (!@$row[9] || $row[12]) continue; ?>

                var marker = new google.maps.Marker({
                 map: gmap,
                 position: new google.maps.LatLng(<?= $row[10] ?>, <?= $row[11] ?>)
                });
                google.maps.event.addListener(marker, "click", function(e) {
                    infowindow.setContent("<div style='width: 240px;'><b>" + <?= json_encode(CapitalizeLastName($row[1])) ?> + "</b>" + "<br />" + <?= json_encode($row[9]) ?> + "</div>");
                    infowindow.open(gmap, this);
                });

            <? endforeach ?>

        });

    // ]]>
    </script>

    <title>Map</title>

  </head>

  <body class="yui-skin-sam" style="height: 100%;">
    <div id="map" style="height: 100%; width: 100%;"></div>
  </body>
</html>

Permalink | Leave a comment »

On downtime and backplanes

Wed, 11 Aug 2010 02:44:00 -0700

Note to self: get (even) better about actually posting.

The irony is that at least once per day I think to myself, "Oh that would make for an interesting post." Clearly my problem is with execution! Unfortunately, sleep usually trumps execution. Suffice it to say I don't have a diary.

Those who follow cs50.tv (and its sister sites cs75.tv, cs259.tv, and computerscience1.tv) may have noticed some downtime over the past several days. The server on which cs50.tv lives (nay, lived) is actually a pretty decent machine: a Dell PowerEdge 2950 III with two quad-core Xeon CPUs, 32GB of RAM, and 6TB of disk space. Most everything in the machine that can be redundant is redundant: it has two PSUs so that if one of them (or the electrical circuit to which it's connected) dies, the other can take over the load; 6 HDDs in a RAID6 configuration, which means that as many as 2 HDDs can die without downtime or loss of data, and 2 NICs so that if one of them (or the switch to which it's connected) dies, we can move the other to our second switch. Because they have moving parts, PSUs and HDDs are among the few components in modern computers that tend to fail these days, so these redundancies help mitigate that risk.

Unfortunately, a server tends to have just one backplane, a circuit board into which important stuff connects (namely, in our case, HDDs). And that's what keeled over this past Wednesday, for no particular reason. The first symptoms were that the VMs that lived on this box became suddenly unresponsive (i.e., cs50.tv "went down"); on at least one of their consoles was mention of "I/O errors," which generally mean RAM or disk problems. We thus proceeded to look at the server's own console, which boasted, let's call it, a red screen of death. (The server was running VMware ESX 4.0, whose screen apparently turns red when really bad things happen.) We thus rebooted the whole server and booted into the RAID controller's configuration utility (a tool with which you configure RAID6 and such), which informed us that disks 1 and 2 were "Failed" and, get this, disk 0 was "Missing." I'll admit I wondered for a moment who had stolen my hard drive.

Unfortunately, reseating the hard drives achieved nothing, nor did rebooting several times. It was time to call Dell. We were fortunate to reach an excellent technician (thanks, Fadi!) who concurred that the problem was most likely the server's backplane, the "Failed" disks, and/or the RAID controller (in descending order of probability).

Within 24 hours, Dell was on site. (Thanks, Tom!) Sure enough, we were in need of a new backplane. Once installed, all 6 of our HDDs started blinking green. (Green is good.) Drives 1 and 2 now appeared as "Foreign," which meant they were functional but unrecognized, presumably the result of the failed backplane. Thanks to RAID6, no matter: we imported the drives back into the array, and we were back in business.

Well, almost. Unfortunately, on its way out the door, the backplane had taken some of our data with it: an assortment of files and directories were either corrupted or lost. And so, despite all our redundancies, we had to restore from backups anyway. (Thank you, Mr. Murphy.)

If there's a silver lining in this particular episode, it's that we used it as an opportunity to upgrade (almost) all of CS50's servers to ESXi 4.1 in one fell swoop (which involved wiping some drives anyway), and we also transitioned cs50.tv from lighttpd to nginx, the latter of which, we're hoping, will perform a bit better under load.

This is CS50, behind the scenes.

Permalink | Leave a comment »

Downloading Chrome and then some...

Mon, 03 May 2010 04:00:00 -0700

Our thanks to Motiur Rahman for submitting this bug of the day, a screenshot he took while downloading Chrome on Ubuntu:

Spot it yet?

djm

Permalink | Leave a comment »

Creating Nodes with Style

Mon, 03 May 2010 03:38:00 -0700

Time passes... An embarrassing number of days months has passed since the last post, and it turns out we're not very good at approving comments. (Apologies to those whose comments were in limbo for so long!) Unfortunately, it seems most every time we have an idea for a post, something comes up (including, sometimes, sleep) and out the window goes the idea. But not this time! For those who dabble in JavaScript, read on...

CS50 has a suite of apps that (with a few exceptions) share a common navigation header. Note the common drop-down in the top-right corner of HarvardEnergy, HarvardMaps, HarvardNews, HarvardTweets, and Shuttleboy. (Hm. Which of those names is not like the others?) Up until recently, that common header was the result of a whole lot of copy-paste, which was getting increasingly annoying to maintain, and so I decided to factor it out (like I probably should have from the get-go). Within the body element of each app's index.php now is, essentially, a line like:

require("/local/path/to/common.php");

For purely aesthetic reasons, though, I wanted the links to the left of the drop-down to be crimson (#600) and the links to the right of the drop-down to be black (except for the "50"). All of those stylizations are easy to do with inline CSS, as with the below:

<a style="color: #600;">API</a>

However, I also wanted the links to be underlined only when hovered over. Half of that effect is easily achieved with similarly inline CSS:

<a style="color: #600; text-decoration: none;">API</a>

Unfortunately, a pseudo-selector like :hover can't be implemented in this same fashion. (See Stack Overflow for a nice discussion.) Of course, it's easy to include in an internal (or external) stylesheet:

<style type="text/css">
<!--

a:hover
{
    text-decoration: underline;
}

-->
</style>

But, for a page to be considered valid XHTML, any style elements must be children of a page's head, not body, element. The implication is that I'd need to put that CSS inside a second file (e.g., style.php) and then add, within the head element of each app's index.php file, a line like:

require("/local/path/to/style.php");

Or I could let HTTP grab that CSS by adding, within the head element of each app's index.php file, a line like

<link href="/remote/path/to/style.css" rel="stylesheet" type="text/css" />

whereby style.css contains only the CSS, not the XHTML tags. I was annoyed by both solutions, though, since my goal was to simplify, ideally by centralizing all common code in just one file.

It occurred to me that a lot of JavaScript libraries support "lazy-loading" of CSS rules, whereby you can incorporate additional CSS rules into a page via JavaScript (cf. YUI 2's Get Utility), but I didn't want to have to include yet another JavaScript library in every app just to underline some links. Fortunately, you can re-create that same functionality yourself quite easily! Inside of common.php, I essentially included the JavaScript below:

<script type="text/javascript">
// <![CDATA[

// wrap code in an anonymous function so that 
// variable (e) doesn't conflict with variables defined elsewhere
(function() {

    // create style element
    var e = document.createElement("style");

    // add attribute to element
    e.type = "text/css";

    // define CSS rules in a text node
    var t = document.createTextNode("div#apps a { text-decoration: none; } div#apps a:hover { text-decoration: underline; }");

    // make text node a child of element
    e.appendChild(t);

    // make element a child of page's head element
    document.getElementsByTagName("head")[0].appendChild(e);

})();

// ]]>
</script>

As you may have gathered from its comments, that code instructs a browser to insert this node inside a page's head:

<style type="text/css">
<!--

div#apps a
{ 
    text-decoration: none;
} 

div#apps a:hover 
{ 
    text-decoration: underline;
}

-->
</style>

(I added some whitespace to the above for clarity.) Incidentally, the XHTML in common.php is wrapped with

<div id="apps">

and

</div>

to ensure that my CSS rules are applied only to the elements within. Ironically, it took me longer to write this post than it did to actually code that solution! But a neat application of JavaScript DOM functions that offers some insight into how libraries like YUI achieve their own magic!

djm

Permalink | Leave a comment »

America's Most Reliable Wireless Network

Wed, 17 Feb 2010 04:54:00 -0800

Okay, the bugs in this screenshot (also courtesy of a student) are even more subtle at first glance than the last. But we think our caption for this one is even wittier.

Perhaps needless to say, someone forgot to interpolate his or her variables.

djm

Permalink | Leave a comment »

Sign up for a Bank of America account today. Er, tomorrow.

Wed, 17 Feb 2010 04:51:00 -0800

So, after making fun of Domino's in class one night for their printf fail, a student passed along this screenshot from their online bank account:

Sorry, no account number or PIN in the screenshot!

See the bug? It's pretty subtle at first. But it boils down to a difference in opinion with BoA as to what "today" means.

We'll leave it to the reader (that's you) to noodle over why this bug happened around the time it did.

djm

Permalink | Leave a comment »

Try Tasty Chicken

Wed, 17 Feb 2010 04:42:00 -0800

So this is a screenshot that I took just before checking out from dominos.com one night. (I used to like their thin-crust pizza, but, frankly, I'm not a fan of their new recipe.) At the time, I was teaching Computer Science E-75: Building Dynamic Websites at Harvard Extension School, and it seemed a fun lecture example of a bug that someone at Domino's really should have caught. (It persisted for months until they replaced this last-second upsell attempt with an ad for "Chocolate Lava Crunch Cakes," whatever those are.)

See it yet?

Indeed, the price of those Kickers should probably be $6.30, not $6.3. Of course, as floating-point values, the two are arguably the same (imprecision of floats aside), but most people prefer their money formatted to two decimal places.

How to fix? Without seeing the code, it's hard to say for sure, but odds are the bug relates to improper use of a function like printf, which allows programmers to format numbers as they see fit. For instance, if f is a float, it can be formatted like money with printf in C as follows:

printf("%.2f", f);

The .2 tells printf to format the float to two decimal places. Of course, the site's probably not implemented in C (in which much of CS50 is taught). In fact, a quick glance at the site's URLs suggests it's implemented in Java (c.f. http://www.dominos.com/home/index.jsp) using JSP, which has its own version of printf as well as alternatives, including java.text.NumberFormat, java.text.DecimalFormat, and java.util.Currency. Of course, because of the imprecision of floats, they really shouldn't be used for money at all, unless Domino's is willing to lose (lots of) pennies over time. Better to store prices as cents or use some fixed-point data type (e.g., MySQL offers a DECIMAL data type that might help in this case).

Whatever the language or database in use, though, someone screwed up. And it shouldn't have been hard to fix! I suppose replacing Kickers with Chocolate Lava Crunch Cakes was, as they say, a workaround.

djm

Permalink | Leave a comment »

This is CS50.

Wed, 17 Feb 2010 04:11:00 -0800

So this first post has been on the to-do list for quite some time, as we felt the inaugural post should really be something special. Unfortunately, we keep coming up with some really good ideas for second and third posts but, having not yet posted the first, we haven't quite gotten around to posting those ideas either.

So we're not very good at this blogging thing it seems.

But that ends tonight. We're finally crossing the first post off the to-do list. Rather than aspire to be special, though, why don't we aim a bit lower and just explain the intent of this blog.

I happen to teach a course (CS50) at Harvard College as well as, like Dan, a few at Harvard Extension School, and, most every day of the week, it seems, some interesting technical question or topic comes up that's fun (for us, at least) to discuss. Given that both of us learned most everything we know from others, we thought it'd be fun (and only fair) to have some of those discussions a bit more publicly in hopes that others might enjoy learning what we've learned from others.

Officially, this blog is "on the intellectual enterprises of computer science and the art of programming." In the interests of full disclosure, we stole that from CS50's own catalogue description. But it would appear to give us license to discuss most anything of technical interest, so that's why we chose it. Admittedly, it's probably misleadingly lofty, since I wouldn't be surprised if our next post has a lolcat. But, overall, we do aspire for this blog to be educational (though occasionally silly) in nature. So rather than just comment on technical things, we'll try to explain them along the way.

In the interests of self-defense, I should say that both Dan and I used to cringe anytime we heard the word "blog," since we didn't quite understand why we'd want to read on a website what other people, oh, had for breakfast. But these things seem here to stay, and a blog seems an apt forum to share technical thoughts publicly. So thus was born the CS50 Blog.

Actually, originally, this blog was to be about funny bugs we've encountered in real life, so the next few posts may be a bit screenshot-heavy as we clear out our backlog. Rest assured they have nothing to do with the intellectual enterprises of computer science.

Anyhow, Dan tells me he had yogurt + granola + nuts + berries for breakfast. I had lunch.

This is the CS50 Blog.

djm

Permalink | Leave a comment »