Myles' shared items in Google Reader

Correlation

(author unknown) — Thu, 05 Mar 2009 21:00:00 PST

Sunday Sweets: More 80s

Jen — Sun, 01 Mar 2009 18:16:40 PST

Sunday Sweets is my weekly NON-Wreck feature, the better to show how wrecky the Wrecks are. As always, if you know who made one of the uncredited cakes, please let me know!

It's been so much fun seeing what you guys turned up in your search for 80s Sweets. To narrow my selection down, I'm going to focus on things that haven't had a huge come-back recently, like Transformers and Indiana Jones. (Okay, so "huge" is relative. Boo Crystal Skull!) I'm also going to hold off on Super Mario for now, since he's got enough material out there to fill several posts on his own.

So let's kick things off down at Fraggle Rock, shall we?

Sarah A. found this insanely detailed Fraggle diorama by Kimberly Chapman. The upper left-hand corner even has Sprocket waiting beside the hole in the wall! I like that Kimberly made it so detailed without being too busy: any more and the cake would have looked chaotic, but this seems to strike the right balance.

Ah, but where are the Doozers, you ask? Not to fear; Amy R. found them!

These Doozers are sooo cute, but I'm afraid it took me a moment of squinting to figure out that's supposed to be a giant radish. (Sorry - it's been a while.) I'm curious what the Bleeding Heart Bakery used to make the lettering, too: is it too much to hope they're sugar rods?

Speaking of adorable characters, check out Rainbow Brite's sprite Twink:

Michelle M. found this cuddly bit of cakery by Dora of Cakes Garden. I had this doll when I was little, and I used to love Rainbow Brite's crinkly holographic outfit, too - remember that, girls?

Sorry, I'm veering off into extreme girliness again. Uh, here you go, guys:

Eh? A little Space Invaders action? Get a load of that pixelated perfection!
Erin C. found it over on Hello Naomi's Flickr stream.

Here's some more arcade awesomeness, and our fondant-free Sweet of the week:

I know it's "just" a sheet cake, but that piping is perfectly done and nicely detailed. Karon Z. found it here, but the baker isn't listed.

Moving along in the evolution of gaming, who among us didn't have one of these?

Ah, how I miss those painful square controllers jabbing into my palms. [wistful smile]

This was the groom's cake at Rachel G.'s wedding, by the way.

Ok, that's enough for this week. Tune in next Sunday for all 80s cartoons. It's gonna be, like, totally righteous, dude.

(Speaking of which, anyone have a classic GI Joe cake to send me? Or how about Thunder Cats? And I'm still waiting for something featuring the Snorks - c'mon, people, if you can't find one, make one! Is that too much to ask?)

Sierpinski Valentine

(author unknown) — Thu, 12 Feb 2009 21:00:00 PST

The Best of Times

Wide Lawns — Tue, 20 Jan 2009 15:57:36 PST

For the past year or so I've heard so many people telling me that we live in the worst times in all of history. Many people I know, some of whom are related to me, really believe that we are living in the "End Times" prophesied in Revelations.

"Look how terrible things are," they say, "Wars, disasters, the economy, disease, racism and poverty."

Everywhere I go people are whining gloom and doom. The world is coming to an end. We're all going to die and not just die, but die painfully.

"I wish I could just go back in time," someone said to me recently.

This person wanted to go back in time to escape the horrors of the present.

"Not me," I said.

"Why? It was so much better in the past."

"Were you there?" I asked.

"No, of course not," they said, "But it was great in the olden days. Everything is awful now."

This person was wrong on both counts. It was neither better in the past, nor is everything awful now. You think things are bad now, go back in time and you'll see some bad.

We live in the best of times. The present time, right now, is the greatest and best time the human race has ever known and we are all lucky to be alive right here and right now. Stop for a moment to be thankful.

The past was horrible and the people who think the past was so much better are ignoring history and idealizing it, creating their own fairy-tale that never really existed.

Anyone who thinks that the present is the worst time in human history clearly skipped class the day they learned about the Civil War, or the Holocaust. Speaking in terms of only the United States the Civil War era wins hands down for the worst time in America. You couldn't pay me to go back there. My vote for worst time in the entire world is probably World War II. On one hand you've got calculated, organized genocide all over Europe. On the other hand you've got atomic bombs on Japan, but to be fair the Japanese weren't exactly angels back then either. Ask the Chinese about that. During the 30s and 40s we had unprecedented mass slaughter and destruction on a worldwide scale. I have no desire to go back to that.

"But what about all the wars now?" some ask.

This isn't a new thing. There have always been wars. War is terrible, but I think we have a lot less of it than we did in the past and that we now have better negotiating skills to try to avoid it. Countries also have more incentive than ever to resolve conflicts sensibly. I admit that there are wars now and there will probably be more, but I'm optimistic that there are fewer of them and that war will continue to decline in the future.

How about all the disasters? There have always been disasters, natural and otherwise. Because of information technology we now know about disasters sooner. We can watch disasters happening live (which we seem to love) and we can see the aftermath of a disaster immediately. In the past this wasn't possible so it just seemed like there were fewer of them. A hundred years ago if there was a tsunami in Asia we wouldn't know about it in America, nor would anyone here have cared. Hundreds of thousands of people would still be dead but we wouldn't have had pictures and video and interviews recounting the horrors. Now we know when something happens around the world and we actually care. Many might argue that seeing disasters live desensitizes us and that we exploit tragedy. Generally, I disagree. I think seeing it creates empathy and urgency. When we see something we can't deny it. Now, because of technology and advances in transportation we can go help people all over the world. Medical technology can save people who would have been left for dead years ago. We have helicopters to evacuate people, planes to fly in relief, doctors who can save and TVs to show us how bad things are so that we can empathize and then hopefully help. So yes, we still have all kinds of disasters and we always will, but now we're better equipped to deal and to help.

And what about all the technology that's been developed to help predict natural disasters? Even fifty years ago we'd have no idea when a hurricane was coming (or a blizzard if you live in the north). Now we have plenty of time to prepare. We can't predict everything, but I think we're working on that and with the technology we do have, lives are really saved.

But the economy!!!!!

The economy isn't at it's greatest right now. I know. However, I feel like this is a correction and that our economy was over-inflated and out of control and that maybe some good can come out of the financial problems we're dealing with right now. First of all, too many people got greedy and were living and operating businesses far above their means. We became too materialistic. I mean really, do we need so many malls and mega-stores or gigantic shopping centers at every highway exit, flanked by seventy-five chain restaurants all offering Tyrannosaurus sized portions of fatty, sugary food? No we do not. I'm sad when people lose jobs, but we don't need all of this consumerism and mass consumption polluting our world. It has killed the creativity, drive and spark in so many people. I'm glad to see a lot of it go. We all got spoiled. Too many of us fell for the illusion of power and grandeur created by the pursuit and acquisition of a bunch of meaningless junk. Scaling back will do us all some good. We'll get through it. Maybe we'll learn to stop valuing meaningless clutter. Perhaps we'll begin to help one another and we'll come to stop wanting and wanting and wanting and taking and taking and taking.

At times this past year I felt like everyone I knew was ill. I felt like disease was everywhere. I can't count the number of times I heard conversations about cancer and other devastating, horrible diseases. Yes, we have diseases. People get sick and die and suffer and it seems so unfair and when they're gone we miss them so much, but think, in the past, even in the recent past, it was so much worse. Had I lived even a hundred years ago I would likely have been dead already, from an illness that's not such a big deal now. Before my genetic condition even appeared to kill me I could have died as a child from malnutrition, polio, small pox and even measles. I may have succumbed to tuberculosis or likely died in childbirth. Think of the advances we've made in medicine. Maybe some of the people we've lost, we would have lost sooner or they would have died in greater suffering with less dignity. In the past, even the most privileged members of society died from things like typhoid and cholera which are now rare and which we have come to only associate with the most destitute, primitive and most poorly developed places in the world. Because of medical advances so many of us have been spared suffering. People with infertility can give birth. So few of us have lost a child, or several children from common viruses.

I'm thankful for psychology, which is such a new science that has saved and improved so many lives. I wouldn't want to live in a world where the mentally ill were tortured, abused, misunderstood or said to be demon possessed. Now we have medications, therapies and just plain understanding for the mentally ill and think of the multitudes of people who can live functioning lives, contributing to society because of drugs and treatment instead of being locked, raving in attics to die in pain.

I like that as a woman that I have options in life and that I can make choices about what I want to do. I'm thrilled that women are allowed to understand their own bodies and how they work and talk openly about them without shame. I'm glad my father and husband never saw me as property and that it isn't legal for my husband to beat me. How great is it that I've been raised in a society that produced a man like my husband who would never dream of not contributing to the housework?

My mother, who is very young, can vividly remember the race riots burning even in small towns across America in the 60s. She remembers segregation, colored water fountains, lynch mobs and a world where black people and white people did not mix. I'm thankful that this is incomprehensible to me. For race issues alone, I am grateful to be alive now. I can't fathom that centuries ago one race of people thought it was acceptable to enslave another and to create, perpetuate and believe in a lasting mythology that said a group of people from another place, who looked and believed differently, were not human beings. And then they fought a war to preserve it! That is horror. Anti-miscegenation laws were horror. The way that even forty and fifty years ago it was socially acceptable to taunt, tease and terrorize, to caricaturize and infantalize men and women of different races, cultures, orientations and abilities is a horror.

We still have war, disasters natural and man-made. We still will have racism, disease and poverty, but in every area we have improved and we're continuing to improve. The time we live in now is truly the best time and we are all lucky to have been born when we were.

There are many serious and often overwhelming problems in our country and in the world. There are issues that frustrate me and worry me. I'm afraid of a lot, but I'm optimistic. Change takes a long time. It takes generations to alter mindsets and it happens bit by bit. Often we will stride forward and then slip back a little but eventually we get there. It's hard not to be impatient because we want it all now. We want it so badly. I know I do. There's a lot I'd like to be different even in my optimism. I also know that we can't always put a time limit on change and that Moses never made it to the promised land. Dr. Martin Luther King, Jr. couldn't live to see the inauguration of a black president. It's a sacrifice we have to make. Our ancestors sacrificed for us and each generation must take responsibility for its descendents. Don't give up on fighting for change just because you think you won't be around to see it or that it will never happen. It will.

There is one thing that I believe above all others and it is this. That everything is going to be ok and that really, it is more ok than we realize right now.

Looking forward at all the problems we face, the solutions may seem insurmountable or impossible. They aren't. When you look ahead, like a little kid on a road trip asking repeatedly if we're there yet, it seems like we've got forever left to go. Looking back though, you can see how far we've come.

We have come so far.

Numerical Sex Positions

(author unknown) — Thu, 09 Oct 2008 21:00:00 PDT

Immortal McHorror burger is 12 years old, looks just like new

(author unknown) — Thu, 02 Oct 2008 07:00:48 PDT

Shared by michellecopter
oh my god.

Karen Hanrahan has been using the same McDonald's hamburger as a prop in her "Healthy Choices for Children" class since 1996 -- 12 years! -- and it's hardly aged a day in all that time. McDonald's should add "immortality" to its list of Unique Selling Propositions for its burgers (unless Karen has an ornate oil painting of the burger in her living room in which it slowly ages, grows mouldy, and decomposes).

The burger on the right, off the paper is a 2008 burger. I had to buy it to get the groovy paper and bag. The meat is a tad darker, the bun a little less golden but in 12 years it will look exactly like that too. Do you find this horrifying? McDonalds fills an empty space in your belly. It does nothing to nourish the cell, it is not a nutritious food. It is not a treat. I marvel at how McDonalds has infiltrated our entire world. A hamburger here tastes exactly the same in China or some around the world place.

1996 McDonalds Hamburger

Babies so cute you could eat them

Marc Abrahams — Sat, 29 Dec 2007 21:01:19 PST

Investigator Wendy Cooper of Canberra writes:

This photo was taken on the south coast of New South Wales, Australia on a recent visit. The sign was outside an establishment that you north Americans would call a “diner”

FLASHBACK: Will America Assassinate General Musharraf?

(author unknown) — Fri, 28 Dec 2007 07:27:01 PST

General Musharraf wants to remain president-in-uniform till 2012. America wants to keep Pakistan occupied by its armed forces for as long as possible. It seems that with these complimentary objectives, Musharraf and Washington are getting along well. The reality, however, is totally different.

Snorting a Brain Chemical Could Replace Sleep

Alexis Madrigal — Thu, 27 Dec 2007 18:00:00 PST

Scientists are reporting that a nasal spray of a key brain hormone cures sleepiness in sleep-deprived monkeys. With no apparent side effects, the hormone might be a promising sleep-replacement drug.

Don't be clever

Justin Etheredge — Sun, 16 Dec 2007 12:48:57 PST

I'm sure that most of us in our programming careers have seen something like this...

  public static int GetNextSize(int i)

    //multiply it by four and make sure it is positive

    return i > 0 ? i << 2 : ~(i << 2) + 1;

Wasn't it nice that they at least commented the line? Ha. Well, maybe you have not seen something exactly like that, but I'm sure that you have seen something equally as asinine. And I'm also sure that when you saw this the first thing that went through your head was "why would anyone ever write something like this?" And surprisingly, the answer is that they did it because they are a programmer. Well, a programmer who is not in control of their urges. :-)

Most programmers love to be clever. In fact, we probably got into programming because we loved being clever. We loved writing small, streamlined code that ran faster than anyone thought it could and at the same time confounded all those who read it. The problem is that the programming world is quickly changing and one of my favorite quotes of all time no longer applies:

"Real programmers don't write comments - it was hard to write, it should be hard to understand."

Now, obviously that never applied and I hope you realized that, but a fundamental shift started happening years ago that has really solidified now. The shift is that the complexity in software has moved from the smaller parts of the applications to the design of the application as a whole, and so the top priority for almost all code should be in how easy it is to read and write, because the systems that they are involved in have become so much more complex.

As applications get larger and more complex, we rarely have to worry about how fast we can read a file in, but we think long and hard about how to make the format of that file easy to parse and flexible. In fact, I remember when XML first came out and there were a good number of people that were predicting its downfall because it was so verbose, but that was actually its strongest selling point. There have been several "Binary" XML specs, but not surprisingly none of them have caught on because they take away the one thing that made XML so attractive, its readability and parseability. The thing that most of those people didn't understand was that bandwidth would get cheaper, computers would get cheaper and faster, but people's time will always get more expensive. The more simple and easier to work with (while still being flexible) a standard is, the more likely that it will be adopted. This has held true for HTML, XML, and sometime in the future it may even hold true for REST web services.

But as programmers these same rules apply, no longer do we get any credit for writing a routine that parses a file 5 percent faster, especially not if it makes the code harder to read, longer to write, longer to debug, or harder to test. No longer does a programmer get respect for using pointer arithmetic and bit-shifting in order to speed up math operations. These kind of optimizations (for the most part) are just pointless exercises that are often written for nothing more than the programmer's ego. (And before you guys start ranting about graphics programming or physics modeling, yes, I do realize that there are places in the world for code like this) But that does remind me of one of my favorite quotes from "Code Complete" by Steve McConnell:

"Programming is not like being in the CIA, you don't get credit for being sneaky."

(It took me a little bit to find that, my copy of "Code Complete" has more highlights than Paris Hilton's hair)

So, if we don't get credit for writing clever code anymore, then what do we get credit for? Well, we get credit for making our code simple and easy to understand. We get credit for loose coupling, for testability, for shallow hierarchies, for good exception handling, for fluid interfaces, for not giving our co-workers headaches, essentially for all around good design. We get credit for being good software engineers, not being winners of the International Obfuscated C Code Contest (although you will get a certain kind of credit for that, but you get my point).

Now hopefully anyone in their right mind would have written the method at the beginning of this post like this...

  public static int GetNextSize(int i)

    return Math.Abs(i * 4);

Without caring one bit about performance (of which I honestly don't know or care what the implications are at this point) I can say with 100% confidence that this code is better. And why? Because I can immediately look at it and tell what it is doing. Even without the comment it is more readable than the code at the top that has the comment, and that is what we want, self-documenting code. Because as the Pragmatic Programmers pointed out, you don't want to have to comment on what the code is actually doing, you only want to comment on the assumptions that were made in order to write the code. Information that cannot be derived just by reading the code itself. The only problem is that the more complex our code gets, the more and more we need comments just to explain what the code is doing.

And don't think that I am saying that there is no place for optimization, that is not true, there is just less room for optimization in the source itself, but there is lots of room for optimizations in terms of overall system performance and in terms of developer productivity. It is more important to focus on the big picture and solve performance problems that are system wide, or refactor code so that changes can be made much faster, than it is to solve a performance problem in a single line of code...unless of course that line of code is being called 8 million times by 50 parts of the system.

So next time you go to write a super clever line of code, think to yourself "Will the benefits of this super cleverness be outweighed by the future issues in maintaining and understanding the code?" And if there is any hesitation at all, then you better not be clever, because 3 months from now you will come across that code and say "What the hell was I thinking?" Then you'll end up rewriting it anyway.

And one final note, if you are the kind of person who just has to write super clever looking code that no one can make sense out of, then they have a language for you, it is called Perl. :-) (let the flames begin)

Leo Hsu and Regina Obe: CrossTab Queries in PostgreSQL using tablefunc contrib

(author unknown) — Thu, 27 Dec 2007 00:57:00 PST

The generic way of doing cross tabs (sometimes called PIVOT queries) in an ANSI-SQL database such as PostgreSQL is to use CASE statements which we have documented in the article What is a crosstab query and how do you create one using a relational database?.

In this particular issue, we will introduce creating crosstab queries using PostgreSQL tablefunc contrib.

Installing Tablefunc

Tablefunc is a contrib that comes packaged with all PostgreSQL installations - we believe from versions 7.4.1 up (possibly earlier). We will be assuming the one that comes with 8.2 for this exercise. Note in prior versions, tablefunc was not documented in the standard postgresql docs, but the new 8.3 seems to have it documented at http://www.postgresql.org/docs/8.3/static/tablefunc.html.

Often when you create crosstab queries, you do it in conjunction with GROUP BY and so forth. While the astute reader may conclude this from the docs, none of the examples in the docs specifically demonstrate that and the more useful example of crosstab(source_sql,category_sql) is left till the end of the documentation.

To install tablefunc simply open up the share\contrib\tablefunc.sql in pgadmin and run the sql file. Keep in mind that the functions are installed by default in the public schema. If you want to install in a different schema - change the first line that reads
SET search_path = public;

Alternatively you can use psql to install tablefunc using something like the following command:
path\to\postgresql\bin\psql -h localhost -U someuser -d somedb -f "path\to\postgresql\share\contrib\tablefunc.sql"

We will be covering the following functions

crosstab(source_sql, category_sql)
crosstab(source_sql)
Tricking crosstab to give you more than one row header column
Building your own custom crosstab function similar to the crosstab3, crosstab4 etc. examples
Adding a total column to crosstab query

There are a couple of key points to keep in mind which apply to both crosstab functions.

Source SQL must always return 3 columns, first being what to use for row header, second the bucket slot, and third is the value to put in the bucket.
crosstab except for the example crosstab3 ..crosstabN versions return unknown record types. This means that in order to use them in a FROM clause, you need to either alias them by specifying the result type or create a custom crosstab that outputs a known type as demonstrated by the crosstabN flavors. Otherwise you get the common a column definition list is required for functions returning "record" error.
A corrollary to the previous statement, it is best to cast those 3 columns to specific data types so you can be guaranteed the datatype that is returned so it doesn't fail your row type casting.
Each row should be unique for row header, bucket otherwise you get unpredictable results

Setting up our test data

For our test data, we will be using our familiar inventory, inventory flow example. Code to generate structure and test data is shown below.

CREATE TABLE inventory
(
  item_id serial NOT NULL,
  item_name varchar(100) NOT NULL,
  CONSTRAINT pk_inventory PRIMARY KEY (item_id),
  CONSTRAINT inventory_item_name_idx UNIQUE (item_name)
)
WITH (OIDS=FALSE);

CREATE TABLE inventory_flow
(
  inventory_flow_id serial NOT NULL,
  item_id integer NOT NULL,
  project varchar(100),
  num_used integer,
  num_ordered integer,
  action_date timestamp without time zone 
  	NOT NULL DEFAULT CURRENT_TIMESTAMP,
  CONSTRAINT pk_inventory_flow PRIMARY KEY (inventory_flow_id),
  CONSTRAINT fk_item_id FOREIGN KEY (item_id)
      REFERENCES inventory (item_id) 
      ON UPDATE CASCADE ON DELETE RESTRICT
)
WITH (OIDS=FALSE);

CREATE INDEX inventory_flow_action_date_idx
  ON inventory_flow
  USING btree
  (action_date)
  WITH (FILLFACTOR=95);

INSERT INTO inventory(item_name) VALUES('CSCL (g)');
INSERT INTO inventory(item_name) VALUES('DNA Ligase (ul)');
INSERT INTO inventory(item_name) VALUES('Phenol (ul)');
INSERT INTO inventory(item_name) VALUES('Pippette Tip 10ul');


INSERT INTO inventory_flow(item_id, project, num_ordered, action_date)
	SELECT i.item_id, 'Initial Order', 10000, '2007-01-01'
		FROM inventory i;
		
--Similulate usage
INSERT INTO inventory_flow(item_id, project, num_used, action_date)
	SELECT i.item_id, 'MS', n*2, 
		'2007-03-01'::timestamp + (n || ' day')::interval + ((n + 1) || ' hour')::interval
		FROM inventory As i CROSS JOIN generate_series(1, 250) As n
		WHERE mod(n + 42, i.item_id) = 0;
		
INSERT INTO inventory_flow(item_id, project, num_used, action_date)
	SELECT i.item_id, 'Alzheimer''s', n*1, 
		'2007-02-26'::timestamp + (n || ' day')::interval + ((n + 1) || ' hour')::interval
		FROM inventory as i CROSS JOIN generate_series(50, 100) As n
		WHERE mod(n + 50, i.item_id) = 0;
		
INSERT INTO inventory_flow(item_id, project, num_used, action_date)
	SELECT i.item_id, 'Mad Cow', n*i.item_id, 
		'2007-02-26'::timestamp + (n || ' day')::interval + ((n + 1) || ' hour')::interval
		FROM inventory as i CROSS JOIN generate_series(50, 200) As n
		WHERE mod(n + 7, i.item_id) = 0 AND i.item_name IN('Pippette Tip 10ul', 'CSCL (g)');

vacuum analyze;

Using crosstab(source_sql, category_sql)

For this example we want to show the monthly usage of each inventory item for the year 2007 regardless of project. The crosstab we wish to achieve would have columns as follows: item_name, jan, feb, mar, apr, may, jun, jul, aug, sep, oct, nov, dec

--Standard group by aggregate query before we pivot to cross tab
--This we use for our source sql
 SELECT i.item_name::text As row_name, to_char(if.action_date, 'mon')::text As bucket, 
 		SUM(if.num_used)::integer As bucketvalue
	FROM inventory As i INNER JOIN inventory_flow As if 
		ON i.item_id = if.item_id
	WHERE (if.num_used  0 AND if.num_used IS NOT NULL)
	  AND action_date BETWEEN date '2007-01-01' and date '2007-12-31 23:59'
	GROUP BY i.item_name, to_char(if.action_date, 'mon'), date_part('month', if.action_date)
	ORDER BY i.item_name, date_part('month', if.action_date);

--Helper query to generate lowercase month names - this we will use for our category sql	
SELECT to_char(date '2007-01-01' + (n || ' month')::interval, 'mon') As short_mname 
		FROM generate_series(0,11) n;

--Resulting crosstab query --Note: For this we don't need the order by month since the order of the columns is determined by the category_sql row order


SELECT mthreport.*
	FROM 
	crosstab('SELECT i.item_name::text As row_name, to_char(if.action_date, ''mon'')::text As bucket, 
		SUM(if.num_used)::integer As bucketvalue
	FROM inventory As i INNER JOIN inventory_flow As if 
		ON i.item_id = if.item_id
	  AND action_date BETWEEN date ''2007-01-01'' and date ''2007-12-31 23:59''
	GROUP BY i.item_name, to_char(if.action_date, ''mon''), date_part(''month'', if.action_date)
	ORDER BY i.item_name', 
	'SELECT to_char(date ''2007-01-01'' + (n || '' month'')::interval, ''mon'') As short_mname 
		FROM generate_series(0,11) n')
		As mthreport(item_name text, jan integer, feb integer, mar integer, 
			apr integer, may integer, jun integer, jul integer, 
			aug integer, sep integer, oct integer, nov integer, 
			dec integer)

The output of the above crosstab looks as follows:

Using crosstab(source_sql)

crosstab(source_sql) is much trickier to understand and use than the crosstab(source_sql, category_sql) variant, but in certain situations and certain cases is faster and just as effective. The reason why is that crosstab(source_sql) is not guaranteed to put same named buckets in the same columns especially for sparsely populated data. For example - lets say you have data for CSCL for Jan Mar Apr and data for Phenol for Apr. Then Phenols Apr bucket will be in the same column as CSCL Jan's bucket. This in most cases is not terribly useful and is confusing.

To skirt around this inconvenience one can write an SQL statement that guarantees you have a row for each permutation of Item, Month by doing a cross join. Below is the above written so item month usage fall in the appropriate buckets.

	
	--Code to generate the row tally - before crosstab
	SELECT i.item_name::text As row_name, i.start_date::date As bucket, 
			SUM(if.num_used)::integer As bucketvalue
		FROM (SELECT inventory.*,  
			  date '2007-01-01' + (n || ' month')::interval As start_date,
			  date '2007-01-01' + ((n + 1) || ' month')::interval +  - '1 minute'::interval As end_date
			FROM inventory CROSS JOIN generate_series(0,11) n) As i 
				LEFT JOIN inventory_flow As if 
		ON (i.item_id = if.item_id AND if.action_date BETWEEN i.start_date AND i.end_date)
	GROUP BY i.item_name, i.start_date
	ORDER BY i.item_name, i.start_date;

	
	--Now we feed the above into our crosstab query to achieve the same result as 
	--our crosstab(source, category) example 
	SELECT mthreport.*
	FROM crosstab('SELECT i.item_name::text As row_name, i.start_date::date As bucket, 
			SUM(if.num_used)::integer As bucketvalue
		FROM (SELECT inventory.*,  
			  date ''2007-01-01'' + (n || '' month'')::interval As start_date,
			  date ''2007-01-01'' + ((n + 1) || '' month'')::interval +  - ''1 minute''::interval As end_date
			FROM inventory CROSS JOIN generate_series(0,11) n) As i 
				LEFT JOIN inventory_flow As if 
		ON (i.item_id = if.item_id AND if.action_date BETWEEN i.start_date AND i.end_date)
	GROUP BY i.item_name, i.start_date
	ORDER BY i.item_name, i.start_date;') 
		As mthreport(item_name text, jan integer, feb integer, 
			mar integer, apr integer, 
			may integer, jun integer, jul integer, aug integer, 
			sep integer, oct integer, nov integer, dec integer)

In actuality the above query if you have an index on action_date is probably more efficient for larger datasets than the crosstab(source, category) example since it utilizes a date range condition for each month match.

There are a couple of situations that come to mind where the standard behavior of crosstab of not putting like items in same column is useful. One example is when its not necessary to distiguish bucket names, but order of cell buckets is important such as when doing column rank reports. For example if you wanted to know for each item, which projects has it been used most in and you want the column order of projects to be based on highest usage. You would have simple labels like item_name, project_rank_1, project_rank_2, project_rank_3 and the actual project names would be displayed in project_rank_1, project_rank_2, project_rank_3 columns.

	
SELECT projreport.*
	FROM crosstab('SELECT i.item_name::text As row_name, 
		if.project::text As bucket, 
		if.project::text As bucketvalue
	FROM inventory  i 
			LEFT JOIN inventory_flow As if 
	ON (i.item_id = if.item_id)
	WHERE if.num_used > 0
GROUP BY i.item_name, if.project
ORDER BY i.item_name, SUM(if.num_used) DESC, if.project') 
	As projreport(item_name text, project_rank_1 text, project_rank_2 text, 
			project_rank_3 text)

Output of the above looks like:

Tricking crosstab to give you more than one row header column

Recall we said that crosstab requires exactly 3 columns output in the sql source statement. No more and No less. So what do you do when you want your month crosstab by Item, Project, and months columns. One approach is to stuff more than one Item in the item slot by either using a delimeter or using an Array. We shall show the array approach below.


SELECT mthreport.row_name[1] As project, mthreport.row_name[2] As item_name,
	jan, feb, mar, apr, may, jun, jul, aug, sep, oct, nov, dec
	FROM 
	crosstab('SELECT ARRAY[if.project::text, i.item_name::text] As row_name,
		to_char(if.action_date, ''mon'')::text As bucket, SUM(if.num_used)::integer As bucketvalue
	FROM inventory As i INNER JOIN inventory_flow As if 
		ON i.item_id = if.item_id
	  AND action_date BETWEEN date ''2007-01-01'' and date ''2007-12-31 23:59''
	  WHERE if.num_used  0
	GROUP BY if.project, i.item_name, to_char(if.action_date, ''mon''), 
		date_part(''month'', if.action_date)
	ORDER BY if.project, i.item_name', 
	'SELECT to_char(date ''2007-01-01'' + (n || '' month'')::interval, ''mon'') As short_mname 
		FROM generate_series(0,11) n')
		As mthreport(row_name text[], jan integer, feb integer, mar integer, 
			apr integer, may integer, jun integer, jul integer, 
			aug integer, sep integer, oct integer, nov integer, 
			dec integer)

Result of the above looks as follows:

Building your own custom crosstab function

If month tabulations are something you do often, you will quickly become tired of writing out all the months. One way to get around this inconvenience - is to define a type and crosstab alias that returns the well-defined type something like below:


CREATE TYPE tablefunc_crosstab_monthint AS
   (row_name text[],jan integer, feb integer, mar integer, 
	apr integer, may integer, jun integer, jul integer, 
	aug integer, sep integer, oct integer, nov integer, 
	dec integer);
	  
CREATE OR REPLACE FUNCTION crosstabmonthint(text, text)
  RETURNS SETOF tablefunc_crosstab_monthint AS
'$libdir/tablefunc', 'crosstab_hash'
  LANGUAGE 'c' STABLE STRICT;

Then you can write the above query as


SELECT mthreport.row_name[1] As project, mthreport.row_name[2] As item_name,
	jan, feb, mar, apr, may, jun, jul, aug, sep, oct, nov, dec
	FROM 
	crosstabmonthint('SELECT ARRAY[if.project::text, i.item_name::text] As row_name, to_char(if.action_date, ''mon'')::text As bucket, 
SUM(if.num_used)::integer As bucketvalue
	FROM inventory As i INNER JOIN inventory_flow As if 
		ON i.item_id = if.item_id
	  AND action_date BETWEEN date ''2007-01-01'' and date ''2007-12-31 23:59''
	  WHERE if.num_used  0
	GROUP BY if.project, i.item_name, to_char(if.action_date, ''mon''), date_part(''month'', if.action_date)
	ORDER BY if.project, i.item_name', 
	'SELECT to_char(date ''2007-01-01'' + (n || '' month'')::interval, ''mon'') As short_mname 
		FROM generate_series(0,11) n')
		As mthreport;

Adding a Total column to the crosstab query

Adding a total column to a crosstab query using crosstab function is a bit tricky. Recall we said the source sql should have exactly 3 columns (row header, bucket, bucketvalue). Well that wasn't entirely accurate. The crosstab(source_sql, category_sql) variant of the function allows for a source that has columns row_header, extraneous columns, bucket, bucketvalue. Don't get extraneous columns confused with row headers. They are not the same and if you try to use it as we did for creating multi row columns, you will be leaving out data. For simplicity here is a fast rule to remember.
Extraneous column values must be exactly the same for source rows that have the same row header and they get inserted right before the bucket columns.

We shall use this fact to produce a total column.


--This we use for our source sql
 SELECT i.item_name::text As row_name, 
 	(SELECT SUM(sif.num_used) 
		FROM inventory_flow sif 
			WHERE action_date BETWEEN date '2007-01-01' and date '2007-12-31 23:59'
				AND sif.item_id = i.item_id)::integer As total, 
				to_char(if.action_date, 'mon')::text As bucket, 
 		SUM(if.num_used)::integer As bucketvalue
	FROM inventory As i INNER JOIN inventory_flow As if 
		ON i.item_id = if.item_id
	WHERE (if.num_used  0 AND if.num_used IS NOT NULL)
	  AND action_date BETWEEN date '2007-01-01' and date '2007-12-31 23:59'
	GROUP BY i.item_name, total, to_char(if.action_date, 'mon'), date_part('month', if.action_date)
	ORDER BY i.item_name, date_part('month', if.action_date);

--This we use for our category sql	
SELECT to_char(date '2007-01-01' + (n || ' month')::interval, 'mon') As short_mname 
		FROM generate_series(0,11) n;
		
--Now our cross tabulation query
SELECT mthreport.*
	FROM crosstab('SELECT i.item_name::text As row_name, 
	(SELECT SUM(sif.num_used) 
		FROM inventory_flow sif 
			WHERE action_date BETWEEN date ''2007-01-01'' and date ''2007-12-31 23:59''
				AND sif.item_id = i.item_id)::integer As total, 
		to_char(if.action_date, ''mon'')::text As bucket, 
		SUM(if.num_used)::integer As bucketvalue
	FROM inventory As i INNER JOIN inventory_flow As if 
		ON i.item_id = if.item_id
	WHERE (if.num_used  0 AND if.num_used IS NOT NULL)
	  AND action_date BETWEEN date ''2007-01-01'' and date ''2007-12-31 23:59''
	GROUP BY i.item_name, total, to_char(if.action_date, ''mon''), date_part(''month'', if.action_date)
	ORDER BY i.item_name, date_part(''month'', if.action_date)', 
	'SELECT to_char(date ''2007-01-01'' + (n || '' month'')::interval, ''mon'') As short_mname 
		FROM generate_series(0,11) n'	
	) 
		As mthreport(item_name text, total integer, jan integer, feb integer, 
			mar integer, apr integer, 
			may integer, jun integer, jul integer, aug integer, 
			sep integer, oct integer, nov integer, dec integer)

Resulting output of our cross tabulation with total column looks like this:

If per chance you wanted to have a total row as well you could do it with a union query in your source sql. Unfotunately PostgreSQL does not support windowing functions that would make the row total not require a union. We'll leave that one as an exercise to figure out.

Another not so obvious observation. You can define a type that say returns 20 bucket columns, but your actual crosstab need not return up to 20 buckets. It can return less and whatever buckets that are not specified will be left blank. With that in mind, you can create a generic type that returns generic names and then in your application code - set the heading based on the category source. Also if you have fewer buckets in your type definition than what is returned, the right most buckets are just left off. This allows you to do things like list the top 5 colors of a garment etc.

Pakistani Taliban unites under Baitullah Mehsud

(author unknown) — Sat, 15 Dec 2007 07:28:51 PST

Battlemap of Swat operations; blue arrow arrows are the Pakistani Army's advance to date. Click map to view.

As President Pervez Musharraf lifts the state of emergency, the Taliban in the tribal areas and the Northwest Frontier Province have united under a single banner, and a single leader. On Friday, a shura, or council, of 40 senior Taliban leaders established the Tehrik-i-Taliban Pakistan -- the Movement of the Taliban in Pakistan -- and appointed powerful South Waziristan Taliban commander Baitullah Mehsud its leader.

The shura was made up of Taliban representatives from the seven tribal agencies of North and South Waziristan, Khyber, Orakazi, Bajaur, Mohmand, and Kurram, as well as the settled districts of Swat, Bannu, Tank, Lakki Marwat, Dera Ismail Khan, Kohistan, Buner, and the Malakand division.

The Tehrik-i-Taliban Pakistan demanded the Pakistani military halt operations in Taliban territory and release of their members. The Taliban also stated it would continue the fight against Coalition forces in Afghanistan.

"The meeting participants have demanded an immediate end of the military operation being carried out in Swat, and given a 10-day ultimatum to the government to pullout troops from the area," the Nation reported. The Tehrik-i-Taliban Pakistan also demanded "the closure of the military checkposts in North and South Waziristan and release of all Taliban activists including former Lal Masjid Khateeb Maulana Abul Aziz."

“Our main aim is to target the US allies in Afghanistan but the government of Pakistan’s ill-strategy has made us to launch a defensive Jihad in Pakistan,” spokesman Maulvi Omar stated. “The government of Pakistan would be paid in the same coin now,” Mehsud said.

The consolidation of the disparate "local Taliban" movement is a logical step in the Taliban's insurgency campaign in northwestern Pakistan. The Pakistani Taliban, while allied with al Qaeda and the Afghan Taliban, has operated as local groups. The creation of a unified Taliban movement in Pakistan will allow them to better coordinate both military and political operations inside Pakistan, as well as with the Afghan Taliban and al Qaeda.

The meteoric rise of Mehsud stems from his ability to organize and command large numbers of fighters, fend off the Pakistani military in South Waziristan, take the fight to neighboring agencies and districts, and organize a nationwide suicide bombing campaign. The government cut a deal with Mehsud in 2006 to end the fighting in South Waziristan. Last year it was estimated Mehsud commanded an army of 30,000 fighters. He has been directly implicated in a series of suicide attacks on military and government officials throughout the course of 2007.

Mehsud leaps over some able and influential Taliban leaders in North and South Waziristan, including Sadiq Noor, Mullah Nazir, and Noor Islam. It is unclear if Faqir Mohammed of Bajaur and the outlawed Tehrik-e-Nifaz-e-Shariat-e-Mohammadi (TNSM - the Movement for the Implementation of Mohammad's Sharia Law) were represented at the meeting, but it is likely. A representative of Maulana Fazlullah's Swat branch of the TNSM was in attendance. Abu Kasha, a key link between al Qaeda's Shura Majlis and the Taliban, likely holds a senior position in the Tehrik-i-Taliban Pakistan.

Iraq by the numbers: Graphing the decrease in violence

(author unknown) — Mon, 17 Dec 2007 07:32:19 PST

Multinational Forces-Iraq has released the data of the effects of the "surge" on the security situation. The reduction in deaths, attack trends, sectarian violence, and improvised explosive device, suicide, and car bomb attacks is dramatic. The number of weapons caches found per year has more than doubled. The graphs below have been provided by Multinational Forces Iraq. Click each graph to view in detail.

The weekly attack trends are now down to or below 2004 levels, with fewer than 600 attacks overall reported per week. These numbers include ineffectual attacks. The number peaked during the summer of 2007, with almost 1,600 attacks in one week in June. Click map to view.

The number of deaths per month nationwide is down to January 2006 levels, at about 600 per month. The numbers peaked in December 2006, with about 3,000 deaths per month. Click to view.

High-profile attacks: Suicide car and vest bombs, and car bombs, are down from a peak of near 130 per month to about 30 per month, the lowest level since May 2006. Click to view.

The number of IED attacks are now at 2004 levels, with roughly 20 per day. This is down from the peak of 60 per day in May of this year, when the surge was peaking. Click to view.

The sectarian violence in Baghdad has decreased dramatically over the past year. Areas that were hotspots now are quiet. Click to view.

The number of weapons caches discovered increased by almost two and a half times from 2006 to 2007. Click to view.

"Civilian," "Military" Nukes: What's the Difference?

Noah Shachtman — Mon, 10 Dec 2007 15:17:14 PST

A week after its release, the intelligence report on Iran's nukes continues to stir controversy. According to the document, Tehran allegedly suspended its military nuclear program -- but kept the civilian one going. Which begs a whole bunch of questions. Like, what's the difference between the two? And how much weapons-relevant work can be done in a “civilian” research program versus a “military” program?

The short answer: a lot. The most direct connection between nuclear energy (civilian programs) and nuclear weapons (military programs) is in the critical materials required. Virtually all commercial reactors generate energy by “burning” plutonium or enriched uranium in light water reactors. These reactors all produce plutonium, as well, though not always faster than they burn it. Plutonium and enriched uranium are two of the easiest materials to use for constructing the core of a nuclear weapon.

Any state with enrichment facilities (to produce enriched uranium, as Iran is trying to do) or reprocessing facilities (which extract pure plutonium from spent uranium fuel rods) would therefore already be skilled in what most people consider the most difficult part of building a nuclear weapon: obtaining the necessary fissile material. And even in states that import nuclear fuel, anyone trained to handle plutonium and/or enriched uranium for a nuclear energy program will still have useful skills for a nuclear weapons program.

Aside from enrichment and reprocessing, civilian nuclear energy programs use some other facilities that can be directly useful for weapons program. “Hot cells,” for instance, are heavily shielded rooms with remote handling equipment, used for manipulating “hot,” or highly radioactive, materials – they would be useful in any bomb construction.

However, the most important overlap between civilian and military nuclear programs lies in the general types of expertise required to make either type of program successful. Nuclear power and nuclear weapons require similar expertise in computer programming and modeling; metallurgy; materials, chemical, electrical, and systems engineering; along with basic scientific expertise in physics, chemistry, and math. Powerful computer models are used to predict the behavior of fuel geometries (in power plants) and weapons designs, for example – even the behavior of rubber that encapsulates nuclear weapons components is modeled to maximize its life and strength.

It does take time to bend these types of skills and knowledge to the creation of a working nuclear weapon, though. How much time will depend on many variables specific to each situation: the level of scientific and engineering expertise present, the amount of materials and knowledge that can be acquired from abroad, the isotopic composition of the available materials, etc. Partially because of variables like these, crafting a nuclear warhead – actually building the thing – takes a lot of trial and error from an engineering standpoint, even though the basic problems and science required to solve them are pretty well-known. For instance, designing a trigger mechanism for a bomb is more difficult than it sounds, as is shaping a perfectly spherical core of plutonium metal for an implosion weapon (not to mention designing the explosive array to implode the metal).

In theory, time is actually the only barrier between a civilian nuclear energy program and a military nuclear weapons program. The technical skills required for both are useful in many industries, and they can easily be picked up at many universities. Further, the level of separation between civilian and military nuclear programs has historically varied; France’s military program had “substantial interconnections” with its civilian energy program, but China’s nuclear weapons stemmed from “a dedicated military program with no major technological contribution from civilian nuclear energy.”

In the broadest sense, then, the ultimate distinction between civilian and military nuclear programs is one of intent. Are nuclear weapons the desired end product? If anything, this makes the National Intelligence Estimate on Iran more worrisome than many are contending: the NIE explicitly says “we do not know whether [Iran] currently intends to develop nuclear weapons.” Iran could simply be biding its time, working on the hardest part of nuclear technology (enrichment) in order to be able to move quickly towards a bomb later. (It is even possible, though unlikely, that they solved their weaponization problems in 2003 and just need time to perfect their enrichment technology.) Historically, though, most countries that pursued nuclear weapons established dedicated military facilities to do so, rather than relying on their civilian nuclear industries. Let’s hope that Iran is following this pattern.

-- Eric Hundman

ALSO:

* Gates Gets it Right on Iran's Nukes
* Intel Insider: Iran Report Ain't Political
* State Department Skeptic: Be Careful About Iran
* White House Changes Iran Intel Story
* Ahmadinejad: The Movie
* NIE: A Timeline
* International Inspectors 2, Dick Cheney 0
* Diplomatic "Disaster" Led to Iran Intel Spill?
* Iran's Chance to Come Clean?
* Spooks = '76 Buccaneers?
* Reporters Help Bust Iran's Nuclear Program?
* Intel Report: Iran Halted Nuke Arms in 2003
* Iran Nuke in "18 Months"? Unlikely.
* Iran's "Industrial" Nukes: Yawn
* Iran's Nukes: Time to Freak?
* Glimmers of Hope in Iran Report
* Iran's Nuclear Scientist Game

What Doesn't Kill You ...

(author unknown) — Wed, 28 Nov 2007 03:00:00 PST

Insults that pay the bills... By Tara Servatius.I can still remember the first column I ever wrote. Some women's studies major glued together a couple of half-cocked theories she'd picked up in class with a paste of rage at the whole male gender and published it on the editorial page of the UNC-Chapel Hill's Daily Tar Heel. There wasn't an original thought in her entire column, but I didn't know that because I was 20 and not well-read enough to recognize it. It was the kind of...

Talk at Yale: Part 1 of 3

Joel Spolsky — Mon, 03 Dec 2007 08:42:05 PST

This is part one of the text of a talk delivered to the Yale Computer Science department on November 28. The rest of the talk will be published tomorrow and Wednesday.

I graduated with a B.S. in Computer Science in 1991. Sixteen years ago. What I’m going to try to do today is relate my undergraduate years in the CS department to my career, which consists of developing software, writing about software, and starting a software company. And of course that’s a little bit absurd; there’s a famous part at the beginning of MIT’s Introduction to Computer Science where Hal Abelson gets up and explains that Computer Science isn’t about computers and it isn’t a science, so it’s a little bit presumptuous of me to imply that CS is supposed to be training for a career in software development, any more than, say, Media Studies or Cultural Anthropology would be.

I’ll press ahead anyway. One of the most useful classes I took was a course that I dropped after the first lecture. Another one was a class given by Roger Schank that was so disdained by the CS faculty that it was not allowed to count towards a degree in computer science. But I’ll get to that in a minute.

The third was this little gut called CS 322, which you know of as CS 323. Back in my day, CS 322 took so much work that it was a 1½ credit class. And Yale’s rule is, that extra half credit could only be combined with other half credits from the same department. Apparently there were two other 1½ credit courses, but they could only be taken together. So through that clever trickery, the half credit was therefore completely useless, but it did justify those weekly problem sets that took 40 hours to complete. After years of students’ complaining, the course was adjusted to be a 1 credit class, it was renumbered CS 323, and still had weekly 40 hour problem sets. Other than that, it’s pretty much the same thing. I loved it, because I love programming. The best thing about CS323 is it teaches a lot of people that they just ain’t never gonna be programmers. This is a good thing. People that don’t have the benefit of Stan teaching them that they can’t be programmers have miserable careers cutting and pasting a lot of Java. By the way, if you took CS 323 and got an A, we have great summer internships at Fog Creek. See me afterwards.

As far as I can tell, the core curriculum hasn’t changed at all. 201, 223, 240, 323, 365, 421, 422, 424, 429 appear to be almost the same courses we took 16 years ago. The number of CS majors is actually up since I went to Yale, although a temporary peak during the dotcom days makes it look like it’s down. And there are a lot more interesting electives now than there were in my time. So: progress.

For a moment there, I actually thought I’d get a PhD. Both my parents are professors. So many of their friends were academics that I grew up assuming that all adults eventually got PhDs. In any case, I was thinking pretty seriously of going on to graduate school in Computer Science. Until I tried to take a class in Dynamic Logic right here in this very department. It was taught by Lenore Zuck, who is now at UIC.

I didn’t last very long, nor did I understand much of anything that was going on. From what I gather, Dynamic Logic is just like formal logic: Socrates is a man, all men are mortal, therefore Socrates is mortal. The difference is that in Dynamic Logic truth values can change over time. Socrates was a man, now he’s a cat, etc. In theory this should be an interesting way to prove things about computer programs, in which state, i.e., truth values, change over time.

In the first lecture Dr. Zuck presented a few axioms and some transformation rules and set about trying to prove a very simple thing. She had a computer program “f := not f,” f is a Boolean, that simply flipped a bit, and the goal was to prove that if you ran this program an even number of times, f would finish with the same value as it started out with.

The proof went on and on. It was in this very room, if I remember correctly, it looks like the carpet hasn’t been changed since then, and all of these blackboards were completely covered in the steps of the proof. Dr. Zuck used proof by induction, proof by reductio ad absurdum, proof by exhaustion—the class was late in the day and we were already running forty minutes over—and, in desperation, proof by graduate student, whereby, she says, “I can’t really remember how to prove this step,” and a graduate student in the front row says, “yes, yes, professor, that’s right.”

And when all was said and done, she got to the end of the proof, and somehow was getting exactly the opposite result of the one that made sense, until that same graduate student pointed out where, 63 steps earlier, some bit had been accidentally flipped due to a little bit of dirt on the board, and all was well.

For our homework, she told us to prove the converse: that if you run the program “f := not f” n times, and the bit is in the same state as it started, that n must be even.

I worked on that problem for hours and hours. I had her original proof in front of me, going in one direction, which, upon closer examination, turned out to have all kinds of missing steps that were “trivial,” but not to me. I read every word about Dynamic Logic that I could find in Becton, and I struggled with the problem late into the night. I was getting absolutely nowhere, and increasingly despairing of theoretical computer science. It occurred to me that when you have a proof that goes on for pages and pages, it’s far more likely to contain errors in the proof as our own intuition about the trivial statements that it’s trying to prove, and I decided that this Dynamic Logic stuff was really not a fruitful way of proving things about actual, interesting computer programs, because you’re more likely to make a mistake in the proof than you are to make a mistake in your own intuition about what the program “f := not f” is going to do. So I dropped the course, thank God for shopping period, but not only that, I decided on the spot that graduate school in Computer Science was just not for me, which made this the single most useful course I ever took.

Now this brings me to one of the important themes that I’ve learned in my career. Time and time again, you’ll see programmers redefining problems so that they can be solved algorithmically. By redefining the problem, it often happens that they’re left with something that can be solved, but which is actually a trivial problem. They don’t solve the real problem, because that’s intractable. I’ll give you an example.

You will frequently hear the claim that software engineering is facing a quality crisis of some sort. I don’t happen to agree with that claim—the computer software most people use most of the time is of ridiculously high quality compared to everything else in their lives—but that’s beside the point. This claim about the “quality crisis” leads to a lot of proposals and research about making higher quality software. And at this point, the world divides into the geeks and the suits.

The geeks want to solve the problem automatically, using software. They propose things like unit tests, test driven development, automated testing, dynamic logic and other ways to “prove” that a program is bug-free.

The suits aren’t really aware of the problem. They couldn’t care less if the software is buggy, as long as people are buying it.

Currently, in the battle between the geeks and the suits, the suits are winning, because they control the budget, and honestly, I don’t know if that’s such a bad thing. The suits recognize that there are diminishing returns to fixing bugs. Once the software hits a certain level of quality that allows it to solve someone’s problem, that person will pay for it and derive benefit out of it.

The suits also have a broader definition of “quality.” Their definition is about as mercenary as you can imagine: the quality of software is defined by how much it increases my bonus this year. Accidentally, this definition of quality incorporates a lot more than just making the software bug-free. For example, it places a lot of value on adding more features to solve more problems for more people, which the geeks tend to deride by calling it “bloatware.” It places value on aesthetics: a cool-looking program sells more copies than an ugly program. It places value on how happy a program makes its users feel. Fundamentally, it lets the users define their own concept of quality, and decide on their own if a given program meets their needs.

Now, the geeks are interested in the narrowly technical aspects of quality. They focus on things they can see in the code, rather than waiting for the users to judge. They’re programmers, so they try to automate everything in their life, and of course they try to automate the QA process. This is how you get unit testing, which is not a bad thing, don’t get me wrong, and it’s how you get all these attempts to mechanically “prove” that a program is “correct.” The trouble is that anything that can’t be automated has to be thrown out of the definition of quality. Even though we know that users prefer software that looks cooler, there’s no automated way to measure how cool looking a program is, so that gets left out of the automated QA process.

In fact what you’ll see is that the hard-core geeks tend to give up on all kinds of useful measures of quality, and basically they get left with the only one they can prove mechanically, which is, does the program behave according to specification. And so we get a very narrow, geeky definition of quality: how closely does the program correspond to the spec. Does it produce the defined outputs given the defined inputs.

The problem, here, is very fundamental. In order to mechanically prove that a program corresponds to some spec, the spec itself needs to be extremely detailed. In fact the spec has to define everything about the program, otherwise, nothing can be proven automatically and mechanically. Now, if the spec does define everything about how the program is going to behave, then, lo and behold, it contains all the information necessary to generate the program! And now certain geeks go off to a very dark place where they start thinking about automatically compiling specs into programs, and they start to think that they’ve just invented a way to program computers without programming.

Now, this is the software engineering equivalent of a perpetual motion machine. It’s one of those things that crackpots keep trying to do, no matter how much you tell them it could never work. If the spec defines precisely what a program will do, with enough detail that it can be used to generate the program itself, this just begs the question: how do you write the spec? Such a complete spec is just as hard to write as the underlying computer program, because just as many details have to be answered by spec writer as the programmer. To use terminology from information theory: the spec needs just as many bits of Shannon entropy as the computer program itself would have. Each bit of entropy is a decision taken by the spec-writer or the programmer.

So, the bottom line is that if there really were a mechanical way to prove things about the correctness of a program, all you’d be able to prove is whether that program is identical to some other program that must contain the same amount of entropy as the first program, otherwise some of the behaviors are going to be undefined, and thus unproven. So now the spec writing is just as hard as writing a program, and all you’ve done is moved one problem from over here to over there, and accomplished nothing whatsoever.

This seems like a kind of brutal example, but nonetheless, this search for the holy grail of program quality is leading a lot of people to a lot of dead ends. The Windows Vista team at Microsoft is a case in point. Apparently—and this is all based on blog rumors and innuendo—Microsoft has had a long term policy of eliminating all software testers who don’t know how to write code, replacing them with what they call SDETs, Software Development Engineers in Test, programmers who write automated testing scripts.

The old testers at Microsoft checked lots of things: they checked if fonts were consistent and legible, they checked that the location of controls on dialog boxes was reasonable and neatly aligned, they checked whether the screen flickered when you did things, they looked at how the UI flowed, they considered how easy the software was to use, how consistent the wording was, they worried about performance, they checked the spelling and grammar of all the error messages, and they spent a lot of time making sure that the user interface was consistent from one part of the product to another, because a consistent user interface is easier to use than an inconsistent one.

None of those things could be checked by automated scripts. And so one result of the new emphasis on automated testing was that the Vista release of Windows was extremely inconsistent and unpolished. Lots of obvious problems got through in the final product… none of which was a “bug” by the definition of the automated scripts, but every one of which contributed to the general feeling that Vista was a downgrade from XP. The geeky definition of quality won out over the suit’s definition; I’m sure the automated scripts for Windows Vista are running at 100% success right now at Microsoft, but it doesn’t help when just about every tech reviewer is advising people to stick with XP for as long as humanly possible. It turns out that nobody wrote the automated test to check if Vista provided users with a compelling reason to upgrade from XP.

I don’t hate Microsoft, really I don’t. In fact, my first job out of school was actually at Microsoft. In those days it was not really a respectable place to work. Sort of like taking a job in the circus. People looked at you funny. Really? Microsoft? On campus, in particular, it was perceived as corporate, boring, buttoned-down, making inferior software so that accountants can do, oh I don’t know, spreadsheets or whatever it is that accountants do. Perfectly miserable. And it all ran on a pathetic single-tasking operating system called MS-DOS full of arbitrary stupid limitations like 8-character file names and no email and no telnet and no Usenet. Well, MS-DOS is long gone, but the cultural gap between the Unixheads and the Windows users has never been wider. This is a culture war. The disagreements are very byzantine but very fundamental. To Yale, Microsoft was this place that made toy business operating systems using three-decades-old computer science. To Microsoft, “computer sciency” was a bad word used to make fun of new hires with their bizarre hypotheses about how Haskell is the next major programming language.

Just to give you one tiny example of the Unix-Windows cultural war. Unix has this cultural value of separating user interface from functionality. A righteous Unix program starts out with a command-line interface, and if you’re lucky, someone else will come along and write a pretty front end for it, with shading and transparency and 3D effects, and this pretty front end just launches the command-line interface in the background, which then fails in mysterious ways, which are then not reflected properly in the pretty front end which is now hung waiting for some input that it’s never going to get.

But the good news is that you can use the command line interface from a script.

Whereas the Windows culture would be to write a GUI app in the first place, and all the core functionality would be tangled up hopelessly with the user interface code, so you could have this gigantic application like Photoshop that’s absolutely brilliant for editing photos, but if you’re a programmer, and you want to use Photoshop to resize a folder of 1000 pictures so that each one fits in a 200 pixel box, you just can’t write that code, because it’s all very tightly bound to a particular user interface.

Anyway, the two cultures roughly correspond to highbrow vs. lowbrow, and in fact, it’s reflected accurately in the curriculum of computer science departments throughout the country. At Ivy League institutions, everything is Unix, functional programming, and theoretical stuff about state machines. As you move down the chain to less and less selective schools Java starts to appear. Move even lower and you literally start to see classes in topics like Microsoft Visual Studio 2005 101, three credits. By the time you get to the 2 year institutions, you see the same kind of SQL-Server-in-21-days “certification” courses you see advertised on the weekends on cable TV. Isn’t it time to start your career in (different voice) Java Enterprise Beans!

(Part two will appear tomorrow).

Not loving your job? Visit the Joel on Software Job Board: Great software jobs, great people.

We’re So Vain! You’re probably thinking this blog is about you!

Jonathan osman — Mon, 03 Dec 2007 07:06:58 PST

Yesterday I was reading a agent’s blog on another site and she was wondering what her car said about her. She owns two cars: a Prius and a Lexus SUV. Her “first impression” vehicle was the Lexus and when she feels the client has bought into her enough, she shows up in the Prius. I have nothing wrong with [...]

Israel's Cyber Shot at Syria

(author unknown) — Mon, 26 Nov 2007 08:52:49 PST

Our friends at Av Week have this story so wired, I couldn’t wait to post this update. And, as you well know, I’m a bit obsessed with it. It now seems that one of Israel’s first shots in its raid into Syria in September was a fusillade of 1s and 0s. From Aviation Week: The U.S. was monitoring the electronic emissions coming from Syria during Israel’s September attack; and—although there was no direct American help in destroying a nuclear reactor—there was some advice provided beforehand, military and aerospace industry officials tell Aviation Week & Space Technology. That surveillance is providing clues about how Israeli aircraft managed to slip past Syrian air defenses to bomb the site at Dayr az-Zawr. The main attack was preceded by an engagement with a single Syrian radar site at Tall al-Abuad near the Turkish border. It was assaulted with what appears to be a combination of electronic attack and precision bombs to enable the Israeli force to enter and exit Syrian airspace. Almost immediately, the entire Syrian radar system went off the air for a period of time that included the raid, say U.S. intelligence analysts. There was “no U.S. active engagement other than...

SR-71: Now, That Was Some Airplane

wwds — Mon, 19 Nov 2007 13:09:19 PST

Note: At the Air Force Academy, part of our training was that we had to fly (as passengers) in every AF airplane. One plane, however, was off limits and that was the SR-71. If memory serves, we could barely get close enough to even touch it, it was so classified. I did get to break the sound barrier in an F-4 Phantom once, mostly because the pilot had just returned from VietNam and had no fear nor regard for orders, “don’t be breaking the sound barrier for these cadets,” he was told. Yeah, right. He was a stud, and no one could make him not do something. We broke the sound barrier at 100 feet over the desert in Nevada, ~~no shit.~~ Years later, I flew faster than Mach 1 in that little British-French, pointy nosed commercial plane, the Concorde. I paid dearly ($$) for the experience and it wasn’t even a thrill. Anything that has to do with the SR-71 interests me…so this first person account by another grad who got to fly it, was of particular interest. In it, he says he was flying at Mach 3.5. To keep it in perspective for you, airliners today travel at about 550-600 mph–Mach 3.5 is 2,663 mph. Rifle bullets are about 1,600 mph.

by Brian Shul, from his book SLED DRIVER

In April 1986, following an attack on American soldiers in a Berlin disco, President Reagan ordered the bombing of Muammar Qaddafi’s terrorist camps in Libya. My duty was to fly over Libya and take photos recording the damage our F-111s had inflicted. Qaddafi had established a ‘line of death,’ a territorial marking across the Gulf of Sidra, swearing to shoot down any intruder that crossed the boundary. On the morning of April 15, I rocketed past the line at
2,125 mph.

I was piloting the SR-71 spy plane, the world’s fastest jet, accompanied by Maj. Walter Watson, the aircraft’s reconnaissance systems officer (RSO). We had crossed into Libya and were approaching our final turn over the bleak desert landscape when Walter informed me that he was receiving missile launch signals. I quickly increased our speed, calculating the time it would take for the weapons-most likely SA-2 and SA-4 surface-to-air missiles capable of Mach 5-to reach our altitude. I estimated that we could beat the rocket-powered missiles to the turn and stayed our course, betting our lives on the plane’s performance.

After several agonizingly long seconds, we made the turn and blasted toward the Mediterranean. ‘You might want to pull it back,’ Walter suggested. It was then that I noticed I still had the throttles full forward. The plane was flying a mile every 1.6 seconds, well above our Mach 3.2 limit. It was the fastest we would ever fly. I pulled the throttles to idle just south of Sicily, but we still overran the refueling tanker awaiting us over Gibraltar.

Scores of significant aircraft have been produced in the 100 years of flight following the achievements of the Wright brothers, which we celebrate in December. Aircraft such as the Boeing 707, the F-86 Sabre Jet, and the P-51 Mustang are among the important machines that have flown our skies. But the SR-71, also known as the Blackbird, stands alone as a significant contributor to Cold War victory and as the fastest plane ever-and only 93 Air Force pilots ever steered the ’sled,’ as we called our aircraft.

As inconceivable as it may sound, I once discarded the plane.
Literally. My first encounter with the SR-71 came when I was 10 years old in the form of molded black plastic in a Revell kit. Cementing together the long fuselage parts proved tricky, and my finished product looked less than menacing. Glue,oozing from the seams, discolored the black plastic. It seemed ungainly alongside the fighter planes in my collection, and I threw it away.

(more…)

Tags: SLED DRIVER

Success

(author unknown) — Sun, 25 Nov 2007 21:00:00 PST