the_codist()

Why Is Your Computer System Down?

Today I went to a local Texas Drivers License office to replace the license I lost at the airport recently (thanks, TSA). On arriving there was a sign, nicely laminated as it had obviously been well used, saying all the computers were down and they could do nothing for anyone.

Give me a break. It's embarrassing to admit you work in an industry where things fall apart at random times. Would you go to a doctor who said he would keep you alive 98% of the time?

Of course this is a "new" system which was promoted in April, and then was supposedly shut down with a virus or worm (articles seem to be light on the details) which caused 4 weeks delay in processing licenses.

I asked the information person if this happened frequently and she didn't say how often but apparently often enough to warrant the nearly permanent sign. Oddly enough certain offices in the state still had service but it seemed hit or miss.

I imagine it was built by some big firm at $500 per hour, who then promptly hired coders off the street at $20. When I worked for a small consulting firm we would be routinely clobbered by one of the big boys who would brag about having 100 people here tomorrow. Today I assume some of these big outfits do outsourcing to squeeze even more profit out of big projects, particularly state projects where the clueless agencies have no idea what a good project team looks like.

I haven't seen many details of what the "new" system looks like, but anything developed today shouldn't suffer from random fluctuating downtime on a statewide basis. Sure, even mighty Google and Amazon have their issues, but those are worldwide 24x7 systems; even in a state as large as Texas, these systems are only used at most 10 hours per day, 5.5 days per week, leaving plenty of time for testing and maintenance during off-peak hours.

There is no way of knowing if it's crappy software or poor systems architecture or crummy management or cheap hardware or what this was but it still reeks of incompetence. I remember a story of a California attempt to build a new drivers license system, after tens of millions in development costs it was rolled out to the whole state, at which point it became obvious that the architecture was non-scaleable and apparently unfixable as well.

Why do we tolerate such garbage in our industry? Programming by now is more than 50 years old, hardware is unbelievably faster and cheaper than ever and in business systems there are few unknowns anymore. Yet today numerous Texas Drivers License offices' computers were down, like some bad stereotype from a 1960's B-movie.

Tomorrow I will call first. Maybe I will get lucky and the computers won't go down the moment I step into the office.

Update: 24 hours later, they are still down.

Update: 48 hours later, still down, plus they were up for a couple hours yesterday, but when it went down they lost all the transactions.

Waiting For Bubba

I hate to keep harping on the App Store, but it's such an amusing pastime.

My latest game, Quantum Pool, was submitted on May 13. Last week on May 19 and 20 it was tested at Apple on the 3.0 beta on three devices for about 10 minutes. That's it. No testing on the existing platform (2.2.1) yet. This seems to indicate that there are multiple people involved in the process.

Funny, I'd always thought only one person worked at the App Store Review team, some guy named Bubba who does reviews in his spare time and sometimes pushes a few random approval/disapproval buttons in order to go home early. I had also thought he might have a mean streak, where he denies apps for random reasons just to remind people who is the boss.

After all He Is The Bubba, He Has The Power.

Sometimes a huge firestorm erupts and his power is momentarily vanquished (NiN/Eucalyptus) but generally he's in charge.

Since Apple doesn't reveal the inner workings of this mysterious group, it leaves one to speculate and create elaborate fantasies of what goes on behind those closed doors.

Ok, so maybe I am wrong and there is no Bubba.

I guess I will have to wait some more. Maybe I will be visited by Godot in the meantime.

Update: Bubba awakens! Quantum Pool has been approved. Funny thing they never tested it on iPhone 2.x and didn't touch it again after May 20th (at least that was visible). Oh well, never got to talk with Godot.

Programmers Like Being In Control - Until You Hit Cancer, When You Aren't

My mom was just diagnosed with lung cancer, despite being a lifelong non-smoker. Although both my dad and her's smoked much of their lives, neither ever got the disease. We are still waiting for the oncologist appointment to get the details. At 78 lung cancer is common (although only about 15% happen to non-smokers) but although there are many treatments depending on the precise variety and phase, the prognosis is rarely good.

As a programmer I am always confident in my ability to code anything, debug anything broken, find some way to make things work. Programmers in general like to be in control, our comfort zone increases the more source we can look at, the better tools we have, the more expressive and productive our languages, frameworks and environments are, and the more we can manage how the code is written. It's when something gets in our way, keeps us from doing it the best way we known how, or simply impedes our progress that we get antsy, angry or antagonistic. We live in a world of our own making, where we can actually change everything to suit the coding need. Create a new language, write a new framework, even code a new operating system if necessary. Of course the reality is there will be limitations of money, time, management, etc. Still we linger in our desire to make things work in whatever manner we desire.

In facing her cancer, I look at an enemy I can do nothing about. The source code is mostly unknown or hard to decipher (understand what DNA really does is like comprehending 100,000 pages of APL code, character wrapped) , the debuggers are terrible and expensive, gray areas abound, there is no way to reboot, you can't easily make changes to see what happens without possibly losing the patient. Biological systems are oddly non-logical; it's like every copy of the program is slightly different. Sometimes you have no clue what is actually wrong. In my mom's case she had 0 symptoms until a few weeks ago, and then only what appeared to be a simple cold. A program that runs perfectly for 78 years and then crashes? No such thing.

Feeling helpless as your code crashes is usually followed up by figuring out what went wrong. But with life it's more like seeing your plane's wings fall off and knowing you can't stick them back on real quick, or wait until you hit the earth and start up the debugger.

I enjoy watch House on TV since the stories are based on real doctor's experiences (there is a diagnostician running the show, although the timeframes are often compressed) and always thought diagnostics was like debugging code, but when you face real people the analogy loses its punch. Of course if I was a doctor I would know more (I started college as pre-med but made a D in anatomy) but even talking with friends who are doctors, there is such a huge body of things that can go wrong, and limited ability to always get the real problem, that sometimes you have to hope for luck.

At least with my mom the diagnosis is fairly common and understood. My dad died of hemochromatosis, which is where the body piles up iron in all the organs until everything shuts down. Before the mid-80's many doctors didn't even know it was a problem or would recognize it; his final doctor was a recent graduate and knew its signs but it was too late. Today it's one of the easiest diseases to treat if you discover it, basically regular blood-letting. In his case the lack of knowledge was fatal. If I have source code I can find a problem I never knew existed, if it's closed-source I can at least ask someone who does have it (and hope they want to help). In medicine it's like much of the code is encrypted or lost.

So here I am not in control and have to depend on other's ability and on medicines still primitive solutions to cancer. Cut it out or blast it with radiation and fatal drugs; like attempting to build a transtator with stone tools and bear knives (Spock quote). In my almost 28 years of professional programming I've seen generations of changes happen in our industry but the pace of medicine never moves at that same speed. Most people who get lung cancer will never see 5 more years. My code in Deltagraph has now been continuously maintained for more than 20 years now and has to be incredibly nasty by now, yet it still works even though the original development team wanted to chuck it after 5 years. There are Cobol programs still in use after 40 years or more. Sure people live longer than that but not with the same productivity and with constant updating.

So meanwhile my sister (a VP at IBM) and I will see what happens and how we will support our mom. Coming from a profession were anything is possible since we can control our own universe it will be hard to have no control over anything but have to continue on faith, hope and love.

My App Rejection #3, Now Apple Forbids Mentioning iPhone 3.0 In Help File

After two bouts of cutting features based on a complaint of first "ridiculing public figures" and then "objectionable content" (namely John Hancock), now I get a call telling me that I can't mention iPhone 3.0 in my help file.

Like it isn't all over the internet and Apple's Website. All I said was that I would add features to the email part of the app when 3.0 ships in the summer.

WTF????

Now I get to resubmit and go back to the back of the bus. The first submission took 7 days, the second 9, the third 12 and I'm guessing the next will be 2 weeks. All for a $0.99 app that lets you put your picture on high-denomination fake money.

I guess not being a famous rock star is a serious drawback here - Trent Reznor's NiN app was approved 3 days after the initial rejection after the huge public outcry.

In addition, now I have to test with iPhone 3.0 Beta, which is tough since I only have one iPhone (running 2.2.1) and one iPod Touch (running 3.0 beta). I can't fully test the app since the Touch has no camera, so I am effectively stymied. If I submit it again with only the help file altered and without a full iPhone test, it might get rejected for some camera issue.

I'm beginning to think this app won't ship until the day when there are real $1 Trillion Dollar bills. Meanwhile I am making no money. My next app, Quantum Pool is almost ready; at the rate things are approved it might be a long time before I see any income and it's not a pretty thing.

Maybe writing for OSX is a better idea after all. Harder to market and lower volume but higher paying per copy, and no approvals needed.

I never thought this would become a soap opera. Argh is becoming my favorite word lately.

The App Store Is a Classic Example Of A Broken Business Process

In the last 10 years or so, I've worked on or witnessed a number of broken technology business processes. Most of the symptoms and causes appear over and over again, and it appears from the outside that the iPhone App Store process (if you look at it as a whole) seems like a colossal disaster in the making.

What is a technology business process? It a series of actions taking by various people in a business to perform a business function involving (or sometimes avoiding) some technological support.

A perfect example was the CheapTickets refund process I worked on in the late 90's. The process took 3 months to get refunds to people with unused tickets, and during that whole time no one actually knew the status of anything, the 5 people who worked on it were way overworked, and the company was losing every customer who ever needed a refund (plus those that they told). Eventually the costs of losing customers overcame the unwillingness to spend any money and they begrudgingly hired my employer (and thus me) to fix it (but only me, they were being cheap). After about 8 weeks or so I had built a new process and the software to make it work, trained the employees and it now took about 1 week with full status reporting.

Why did it get so screwed up? When a process (or company in this case) starts they don't plan on growth very well, so things sort of slide along for a while. Processes which are not immediately important often get overlooked. Then when it starts to be a pain, managers often start by making the existing employees work harder. If that doesn't work they raid other departments in the company (if politically possible) but that screws up other processes and generally creates stress in the new employees (how do I do this) and the old (I have to teach people AND do all this work). If the workers in the process complain management may point to budgets being limited or other "excuses" to keep things from bubbling up the chain. Spending money on tools (which might make people more productive) meets the same denial.

What often happens in these high pressure process environments is that the workers start to create their own "tools" to make their lives easier. A shared spreadsheet here, a couple databases there, and soon the process is being supported by unsupported tools and applications which although they seem to help the workers, actually makes the process even less effective. Management often approves of these "self-sufficient" initiatives as they seem to cost no money and make it look like progress is being made.

In the CheapTickets case, the poor workers had utilized spreadsheets, databases, the SABRE reservation system, boxes, files and post-it notes to try to make sense of the mess. Until I showed up no one had spent any time understanding what the process was, much less how it needed to work. By spending time with each employee, seeing where all the data came from and to, following the process through all its evil steps it was clear that it could be streamlined and simplified. Since I understood both the business and knew what technology could (and could not) do it was fairly easy to build something that worked for everyone involved.

Another big one I did was for Sabre's Corporate Air Pricing Group, same kind of kludgy manual process but on a larger scale, which cost Sabre millions a year in Airline penalties simply because they had no audit trail to prove if they were at fault in ticket errors or the Travel Agencies were. Yet despite the $ the IT people had no interest in such a "minor" process. So the CAP department hired us (and I wound up working with another guy on it from my employer). Same problem (no one cared, employees do the best they can, customers are pissed, and the company is losing $), and same result (orderly automated processes, full audit trail, better management, and tons of saved money).

So what do these two examples have to do with the iPhone App Store? Even though I am outside I do know a few folks inside, though mostly I can simply see the end result. Initially when they started the App Store team I know they raided other departments for personnel. They also seemed to predict the volume but apparently did not plan for it.

From what I can tell, the process isn't very automated, the reviewers are overworked, there is little supervision or approvals internally, and the lack of bug fixes on vital web apps used by iPhone Developers dragging on for months sure seems like a good sign no one is working on them (I still can't even log into the Apple Bug Reporter web app despite numerous emails to them, but I do get helpful automated responses telling me to file bug reports in the Bug Reporter!). Apple in the past has always been very supportive of its developers (I started as a Mac developer in late 1984) and I've always gotten good help from the Developer Tech Support group (where I worked for a bit in 1995). Apparently the App Store team is isolated from the rest of the employees, plus they are forbidden to talk to anyone even at Apple about what goes on there.

I'd say either the management of this group is incompetent or overwhelmed, or maybe they are neither but their management has hamstrung them with the usual excuses (you can't exceed your budget, sorry); either way the end result reeks of CheapTickets' refund process or Sabre's Air Pricing (Corporate was bad but the general group was even worse). For a company with $29 Billion in cash, complaining of budget troubles is rather silly. I imagine that Steve either doesn't care since it doesn't impact products directly, or with him being out he's not getting the full picture.

Business processes are everywhere, you interact with them everyday. Most companies big and small have terrible ones. Fixing them is not always expensive (Sabre's ROI was about 6 months) but it often scares management types since you do have to spend some money (try fixing your dead car with no money) and it means change (people hate change) and it might make them look bad (you had to hire someone to fix your processes, you must be incompetent).

Fixing stuff like this is actually fun, when they are willing to accept that it's necessary (like an alcoholic admitting it); generally the workers are happy someone is listening, the customers (if you get to interact with them) are happy the company cares, and then when its working you get all sorts of thank yous (unless you are incompetent and screw it up).

Right now I'd love to get inside Apple and take a look. Fixing a bad process is a process itself and requires understanding of a lot of areas, both business and technical, and being a patient interviewer. Plus you have to be a good storyteller and management soother.

Of course fixing the App Store would make me happy too. Maybe my damn apps might get approved in less than geologic time.

One can always hope. That is itself a process.