Daniel Schneller's Blog

Removing Xcode 3 shared build settings from Xcode 4

2012-01-11T07:00:00.000+01:00

This is about me getting a substantial amount of grey hair over the past couple of days, trying to hunt down a setting that would cause the current version of Xcode 4 to build my iOS projects to an unexpected, but not unfamiliar, taken over from Xcode 3, location, but not presenting any obvious way to revert that.

A little history

In Xcode3 you could use the preferences dialog to configure custom build output folders. This was necessary when you wanted to organize a somewhat more complex software into several cross-referencing Xcode projects and at the same time retain some sanity when linking and packaging it. Clint Harris Tutorial on shared libraries describes it in more detail.

The preferences dialog looked like this (image copied from Clint’s site, because I didn’t have any Xcode3 installation lying around anymore):

In my venerable Xcode 3 setup I configured the shared build output to /Users/ds/Documents/cc/SharedBuildOutput.

Introducing Xcode 4

In March of 2011 Apple released Xcode4, a major overhaul of the IDE with many new features and, of course, new configuration. At the time of me writing this in January 2012 the latest Xcode release is 4.2.1 and I installed most releases as upgrades on my MacBook Pro. The settings dialog for build locations looks like this in my current installation:

As you can see, there is no trace of the previously configured shared build output setting, let alone my custom directory.
Ok, so apparently this is the new way of doing things, and starting with new projects will surely apply.

Not so quick…

Despite what (I for) one would expect, nevertheless, this is what a brand new (iOS) project’s actual build setting look like (I filtered to just show the relevant parts):

Switching from the Combined to the Levels view shows that the setting is clearly not coming from the project itself, which explains why even a freshly created project could know about this old folder:

This clearly goes against the current Xcode standard of placing build products into so-called “Derived Data” subfolder folders with (partly) random names. Build scripts can use these by referring to them using the environment variables, but more often than not, leaving the (now already) trodden path of the new defaults will cause more trouble than it’s usually worth.

“How do I fix this?”, you might ask. Well, that’s what I asked myself, too, and it took me a few days to finally figure it out. Remember that Xcode 4 no longer shows this path in its Build Locations preferences, so this is obviously not getting us any further.

Circumventing the problem…

Changing the folder at the project level works, overriding the default. As soon as you have created a new project, you can change the “Build projects path” and “Intermediate Build Files Path” to simply build. This should be the same value that’s the default in a fresh Xcode 4 installation.

I like to keep my build settings in a separate .xcconfig file that makes it easier to share and change settings with other project contributors using version control. If you do that, you could also use that to override the old default by putting these lines it it:

SYMROOT = build
OBJROOT = build

However, that, too, requires you to remember making the change for each and every new project. Surely it would be better to get rid of that now-unwanted setup.

…or solving it

Albeit there is no UI preference setting to modify the SYMROOT and OBJROOT settings in Xcode 4 anymore, apparently it still honors them from the previous Xcode 3 installation. So in order to reset them, we need to know where they are stored. I did not know anything more than that it must be a user-specific setting, because a newly created account did not use it. So – after deleting everything from ~/Library/Developer/Xcode/DerivedData – I tried (in this order):

$ grep -r SYMROOT ~/Library/Developer 2>/dev/null
$ grep -r SYMROOT ~/Library/Application\ Support/Developer 2>/dev/null
$ grep -r SYMROOT ~/Library/Preferences 2>/dev/null

Well, actually I tried searching for Documents/cc/SharedBuildOutput first, but that also matched several archived applications. The last command revealed this:

Binary file ./com.apple.dt.Xcode.plist matches

Converting the first match to XML with plutil -convert xml1 -o tmp.xml com.apple.dt.Xcode.plist and searching inside it I found this:

<key>IDEApplicationwideBuildSettings</key>
<dict>
    ...
    <key>SYMROOT</key>
    <string>/Users/ds/Documents/cc/SharedBuildOutput</string>
    ...
</dict>

Instead of removing these two lines and converting the file back to its binary format, I deleted that entry with the handy little PrefSetter application by navigating to the com.apple.dt.Xcode domain and searching for SYMROOT in there. That way there is no risk of making any typos, damaging the file.

Do not forget to save the change!

Success!

Restarting Xcode and looking at the build settings of the very same “Test” project created earlier proves that the change had the desired effect:

Of course, as a last resort (maybe there are more problems with your Xcode settings?), you could also just delete – or rename – the com.apple.dt.Xcode.plist file and have Xcode start afresh, even though I noticed when I tried this, that several settings were left unchanged, e. g. Source Trees. These are apparently stored somewhere else…

SOAP web services with iOS

2012-01-02T09:00:00.000+01:00

Fresh on the codecentric blog is my new post about using SOAP web services from an iOS client application.

It features a short comparison of the current state of frameworks and tools with the Java world, and then focusses on the sudzc open source library that takes a very interesting approach in generating web service client artifacts by transforming the service's WSDL into Objective-C classes using XSL transformations.

The post is available in German as well.

Why good metrics do not equal good quality

2011-10-17T21:43:00.000+02:00

A short while ago I posted an article on the codecentric blog about why good metrics can be, but need not be equal to good software quality. As I wrote earlier, I will add links to this blog whenever I post something of interest to the company site.

The post is available in both English and German at http://blog.codecentric.de/en/2011/10/why-good-metrics-values-do-not-equal-good-quality.

List all open Safari tabs across windows

2011-08-02T17:50:00.000+02:00

Over the course of a workday I tend to accumulate lots of browser windows, and even more tabs inside them. Up to now, I would often lose track of what which tabs were open in which window and in which space. In the end, I would often just open a page again in a new tab of the window I happened to be in at the moment, increasing the overall clutter.
With the advent of persistent application state across reboots or application restarts as well as fullscreen apps in Mac OS X Lion that situation has gotten even worse.

The "Window" menu in Safari does not help too much, because it only shows the tabs of the currently focussed window. Today, while wondering why a website was not displaying correctly, I accidentally found a remarkably simple (and built-in!) way of showing all open tabs across all open Safari windows.

Just hit Cmd-Alt-A or pick "Activity" from the Window menu in any Safari window to open or focus the Activity overlay window.

It is usually used to determine what servers different parts of any given website are loaded from. However, even if you are not interested in that, this little popup contains a list of all currently open tabs, regardless of the window they are in.

Double clicking any entry in there will take you to the correct window and focus the desired tab. Nice and easy, not even an extension needs to be installed :) The only drawback I see is that you cannot sort that list, clicking the "Address" column header has no effect.

A final tip: When you open the window, it might have one or more of the entries expanded, making the list overly long. Even though there is no collapse-all feature I could find, you can rather quickly do so manually by selecting the top entry in the list and then repeatedly hit the left and down arrow keys. Left arrow will close the currently selected item, down arrow move to the next entry. Even with lots of tabs listed, within a few seconds you should make your way all the way down, leaving a nice clean list of tabs.

com.apple.dock.extra.xpc needs to take control

2011-07-26T17:29:00.001+02:00

After my recent Mac OS X 10.7 Lion upgrade I was greeted with a dialog box after reboot telling me that

com.apple.dock.extra.xpc needs to take control of another process for debugging to continue. Type the name and password of a user in the "Developer Tools" group to allow this.

No problem, I thought, and entered my credentials. Alas, to no avail. My name and password were rejected, which had me a little panicked at first, because I thought the OS upgrade might somehow have botched my account, potentially locking me out of my Mac later.

Hitting cancel would only get rid of the message for a few seconds, then it would reappear, again and again.
Turns out the solution was quite easy: Go the Mac App Store and download Xcode 4.1. Once the installer has finished downloading, run it to replace the Snow Leopard version of Xcode 4.0 with a Lion compatible one. Once that is done, the message will not appear again.

Please note: Maybe I could somehow have fiddled with my accounts privileges and group memberships, but as I needed Xcode anyhow, installing the correct version seemed the logical thing to do.
And by the way: Should the Xcode 4.1 installer seem to take forever, make sure there not this little popup hidden behind some of the other Windows on screen:

The installer will stall, until you close iTunes and the iTunesHelper (which may be running, even if iTunes is not. Use Activity Monitor to quit iTunesHelper if needed.)

Update: Apple has released an update to the Xcode 4.1 installer (4.1.1), which will fix this (iTunes) problem. As the release notes say, if you have already installed it, there is no need to update, as the included software is the same as before:

Lion Finder Source List Icon Size

2011-07-21T19:05:00.002+02:00

[Update]As can be read on Mac OS X Hints here, this setting also applies to the side bar in Mail.[/Update]

Apple has - as was to be expected - slightly modified the appearance of many Mac OS X controls in 10.7 "Lion". Some of those changes have caused protest and debate around the net, but I believe this is just the same as it is with face-lifted car designs, which means in a few weeks everyone will have gotten used to the new style and consider the previous version old-fashioned.

However, there is one particular little issue that I could tell I would not come to like immediately: The icons - and more importantly the font-size - in the Finder's left hand sidebar is way bigger than it was in Snow Leopard. This makes the source list look much more cluttered in my opinion.

At first, I headed for the Finder's View Options context menu, but there's nothing there to change the sidebars appearance. A little more digging then brought me to the General preference panel in System Preferences (previously called Appearance). There you can make the desired change:

The following pictures show the three choices (Small, Medium, Large) with Medium being the default. Set to Small the Finder looks much friendlier again.

Oh, and while you're at it, you might also want to change Show Scroll Bars setting to When scrolling, instead of Always or Automatically based on input device if you like me sometimes use an older mouse.

A Short Story on a Waste Of Time

2011-07-11T20:16:00.000+02:00

This is about wasting a lot of time, effort and some energy on an unfortunately not so successful transition from smaller to bigger disks. Actors include a few external drives, Time Machine, an iMac with a dying system disk and me, being a little stupid. Fortunately there were no really serious consequences, however if I ever face a similar situation again, I might come here and read up on how to migrate systems and backups more sensibly.

Previously on Lost Daniel’s machine

To understand the situation fully, this is what my drive and partition layout used to be:

320GB internal hard drive called “Snow Leopard internal”
2TB external USB drive
- 500GB partition called “TM500” for Time Machine
- 1.5TB partition called “TMRest” for media, disk images etc.
2x1TB in a Firewire 800 Western Digital MyBook II enclosure, configured as a 1TB RAID1, used for media and VMware instances

The partitioning on the USB drive came about by migrating (cloning) a Time Machine backup from an earlier 500GB ATA drive in an external Firewire enclosure. The enclosure had failed, and as I did not feel like buying a new one for an elderly PATA hard drive, I replaced it with a new USB model, including a 2TB drive. However, I did not want to “waste” the whole new thing for Time Machine. Even with the 500GB drive I had had well over 6 months of backup to go back through, which was more than enough for me. So after transferring the Time Machine data from the old drive to the new bigger one, I set up the remaining 1.5TB as a media partition.

Early signs of trouble

A few weeks ago my iMac’s internal hard disk started failing. It manifested itself a spurious beachballing during random operations, even ones that would not obviously encompass intense disk activity like surfing the web or writing emails. At first I suspected a recent Flash or Adobe Reader updater to be the culprit, because I had had headaches over these before. Only after a few days did I realize that the system was logging disk read errors into the kernel log:

kernel[0]: disk0s2: I/O error.

Googling for that message quickly confirmed my suspicion and revealed that the disk was failing, making it impossible for the OS to function reliably. Fortunately the drive was kind enough to fail slowly, instead of a big bang, with this early warning I could make sure Time Machine had a very recent backup of my data, before shutting down the computer and taking it in for repair.

The old 320GB disk that came with the computer had been working since 2008, and while I would not have minded it living a little longer, this turned out to be a fine opportunity to get some more room to breathe on the internal disk, without having to resort to putting the Movies folder and the iTunes library to an external Firewire drive. So I had the shop install a replacement 2TB disk in place of the faulty stock one.

Room to breathe

Upon return I booted from the Snow Leopard DVD and let the Time Machine restore run. A few hours later I was happy to boot up the Mac as if nothing had changed. Well, almost nothing. For one, applications that had been downloaded from the Internet caused the “This is potentially harmful, because it’s from the Intarweb” confirmation dialog to appear, even though I had run these apps for ages before the restore, but more importantly, the available disk space was about 1.800GB more than before. So far so good.

Happy about the newly won capacity I started moving back the iTunes library from the external drive to the internal one. After that, I began moving several hundred gigs of iMove projects, too.

Remember the backups

When finished copying lots of data - which had taken several hours - it occurred to me that now the 500GB Time Machine partition on the USB drive would be way too small now, even though the system drive was only about half full. I thought about my options for a moment and came up with the idea to reset the Firewire enclosure to work in RAID0 mode, effectively giving me 2TB capacity instead of 1TB with the former RAID1 configuration, and let Time Machine use that volume for backups. Sure, RAID0 increases the chance of fatal failure, but as I also back up my data to Carbonite, I figured this would be an acceptable risk.

I would then reformat the 2TB USB attached drive as a single large partition for miscellaneous use.

Lots of copying and moving

Because of the upcoming repartitioning and reformatting I had to do a lot of shuffling around the remaining 1.000GB of data that was still lying partially on any of the three external drive partitions. This is when I realized that hard drives may have grown in size, but they have not grown in speed nearly as much. Even without the bottlenecks of Firewire 800 and USB2.0 copying around a TB of data will take an uncomfortable amount of time, even with directly attached SATA drives and sequential access. For example, assuming a (very optimistic) average speed of 60MB/s which I could get from the FW800 drive to the internal volume, moving 1TB (which is approximately 1.000.000MB will still take at least 4.5 hours. With USB 2.0 you can feel lucky if you get half the speed on average, easily doubling that time.

So over the course of three days I moved several TBs of data between the drives, partitioning and formatting in between. At last I ended up with this setup:

2TB internal hard drive called “Snow Leopard internal”
2TB external USB drive, called “USB2MEDIA”
2x1TB in FW 800 MyBook II enclosure, configured as 2TB RAID0, called “TM2TBR0”

It would have been easier, had I not wanted to keep my 6 months of Time Machine backup history intact. Even though I have rarely needed it, I tend to rather err on the side of caution. Without that and the accompanying limitations with regards to what could be moved where and repartitioned how and when, I could probably have saved one or two large copy jobs.

Caught my mistake?

As I said earlier, after a few days and nights of shifting large amounts of data over way to narrow bandwidth channels, I was now sitting in front of my target configuration. Extra points to anyone who has caught my little oversight that made all this was somewhat in vain. The rest of you would probably have been just a confused as I have when I entered Time Machine for the first time after that, just to see that my backup history had shrunk down to only 2 weeks after all.

Turns out, I was a little too happy about all the space I had won on the internal drive - so I did not remember to turn off Time Machine after the first boot into the new system. Instead, I started importing all sorts of large media files from the external disks to the internal one. While I was doing so, Time Machine dutifully kicked in, noticed a lot of new material and started backing that up to the still 500GB volume. And because of all the new data and the much too small target drive, it did the only thing it could, remove old backups to make room for the new stuff.

In essence, I had lost my 6 months worth of history even before I came up with my elaborate (?) data moving and migration scheme…

So what?

Well, in the end I did not really lose anything very important - all my current files and some history are still there. Should I find out a day from now that I need to get back a version of a file from 6 months ago, I may be out of luck, but that’s about it. However, I am fairly sure this will not happen to me again, shouldwhen the next hard drive failure strikes. And now that you have made it through this text desert, maybe it won’t happen to you, at all :)

Using JMeter to measure binary protocols

2011-05-11T23:26:00.000+02:00

This is just a short post linking to an entry I recently posted on my company's blog. It is about Using JMeter to measure binary protocols, which I recently had to do in a customer project.

It is very well possible that in the future there might be more posts there than here, but I will make sure there are links posted to this blog. Please note, that the JMeter article is available in both English and German.

iOS Feature creep hurts usability

2011-02-15T00:07:00.001+01:00

I have been a user of the iPod Touch from day one. With the iPhones being tied to unreasonably expensive mobile phone contracts it was never an option for me to buy one just for fun. But when the iPod Touch was announced I knew this was my device. I have been updating iOS whenever there was an update available, and usually I was pleased with what Apple had refined and added.

However, one thing has been bothering me for some time now, and I did not know if it was me getting older, or if really something had changed with the OS.

Long Audio Files

One of the main purposes the iPod has for me is listening to podcasts and audio books. With these I tend to almost enjoy my commutes. But also when doing chores around the house or at the gym my (now 4th gen) iPod Touch is a trusty companion.

Nevertheless, in recent times I got annoyed regularly when trying to pause playback and resume later from the lock screen. I am not sure if this was possible with the first versions of iOS - when it was still called iPhone OS - but it feels that this feature has been there forever: Double click the Home button when the lock screen is up, and you get playback controls in the upper third of the screen. This always worked perfectly for me, but in the past couple of months I have regularly hit the skip back or forward buttons by accident. When you are in the middle of a 3 hour podcast or a 12 hour audio book, jumping back to the beginning and having no idea how far exactly you had already got into it, this can be rather unnerving.

Getting old and clumsy?

I did not know what caused this. At first, I thought I was just a little more clumsy than usual, but this happened time and again. So finally I asked my book writing partner in crime Udo to please take a screenshot of the playback controls on his 2G model which still had iOS 3.1.3 on it. This is his screenshot:

Looks normal, doesn’t it? But somehow different than 4.2.1:

As is clearly visible in the direct comparison, the skip buttons have been moved much closer to the center button for play and pause, apparently to make room for additional controls, as can be seen here:

Small change - big impact

This seemingly minor change has proven to be a real pain in the neck for me. I have heard from other people complaining about their accidental hitting the skip buttons, too. In my opinion this is a classic case of feature creep: Over time more and more options get added to what was initially a well designed and carefully layed out design, effectively making it more feature rich, but also less usable. I am sure the people at Apple who designed this were aware of the implications, but probably they thought it was still just usable enough to do it this way instead of adding another row for the additional buttons.

So now I know it is not me somehow losing precision with my tapping on the screen, but an actual software change that works against me in this case.

Feature creep

There are other instances where adding new features to an already polished system adversely affects its ease of use and works against the proverbial magic like simplicity in handling Apple devices. One example would be the multi-tasking on recent iOS versions. Its quick task switcher is brought up by double-clicking the home button. This is a completely counterintuitive gesture in my opinion, not even mentioning the fine differences between switching and closing apps by “deleting” them from the switcher bar.

Another case of increasingly bad UI is the much criticized notification system. Originally it was just used for very few types of alerts and informational popups. Nowadays every app is free to push modal popups right in your face. When it was initially designed I am sure one of the key assumptions was that there would be a very limited number of these very disruptive alert boxes. Over time it became (ab)used for a lot more, which turned its simplicity from a strength to a weakness.

What’s next?

In my opinion Apple has to rethink a lot of aspects in iOS. When it first came out Steve Jobs said it was “at least 5 years ahead of the competition”. It is 2011 now, almost 4 years into iOS’ life, and Android, Windows Phone 7 and most recently webOS are coming on strong, running circles around iOS in some areas. While I am still not yet sold on my company Android phone I have to admit the platform has its strong points, and iOS could definitely use some of its features. However, I would rather not have them if Apple’s adding them to one of the next versions of their mobile OS meant they would be bolted on to the existing UI concepts without some serious rethinking.

On Writing a Book, Pt. 5 - The Contract

2011-01-24T21:14:00.002+01:00

This is part five of an ongoing series about my experiences while writing the MySQL Admin Cookbook for Packt Publishing. All previous parts can be found under the mysql-admin-cookbook label.

After the initial discussions about the book’s outline which I described in part 2 of this series were underway, talks about the actual writing contract started. Naturally, neither Udo nor I had any experience with publishing companies and the contracts between them and their authors. Add to that the fact that we are both from Germany and had had little to no experience with English legal stuff of any kind - apart from the usual 5000 pages EULAs you just “agree to” by clicking the “go away” button as quickly as possible - and you will probably understand that we were a little nervous about what we should expect.

Contract outline

On Feb, 27th 2009 we were sent the first draft of our contracts as PDFs via email. It was twelve pages long and was structured as follows.

First, there was a definition of terms to be used throughout the document, clarifying what terms like “Editors”, “Electronic Form”, “Program”, “Rights” and so on were supposed to be understood as in context of the contract.

After that followed a series of paragraphs covering different aspects of the relationship between author and the publishing company. Contained therein were regulations about the writing schedule, copyright, what would happen in case the schedule was not met, about the process of editing, corrections and proofreading, the distribution of the finished book, warranties and payment terms. Moreover there were items about revised later editions of the book, marketing and support, regulations concerning what rights the author would have regarding using materials from the book and in which contexts and how many copies the author would receive for free after publication.

These descriptions were about 7 pages long. All this text seemed to be very much a standard template used for every book of that publisher.

After that followed a more individual part, detailing the exact names and addresses of the contract partners, amount and schedule of advance payments and later royalties depending on book sales and the terms of payment. Moreover the schedule for all the chapters was laid down with due dates for each of them. Since we were writing as co-authors, my contract mentioned only the chapters Udo and I had agreed on to be written by me, his contract naturally covered the other half.

Finally the format we would send our chapters in would be defined as either Microsoft Word Format or Open Office ODT. Graphics were expected in either TIFF or EPS as line drawing, screenshots as PNG or JPG.

Contract Details

Udo and I went through our respective PDFs on our own at first. Once we were through we sat together and compared notes and questions. There were quite a few points we were immediately sure and in agreement about - even though it was our first publishing contract - that could not be accepted from our side. Not yet knowing how much of a problem changing those would be for the publisher we agreed unanimously that should they not be possible to change we would not go ahead and abandon the whole project.

I will not go into extensive detail here, but let me just mention some parts we were not willing to sign as they were:

a phrase about liability in case the author(s) were to miss the schedule
a phrase which would make the author(s) responsible for anything in the book being “unlawful”, or “infringing on the copyrights of any third party”
a phrase coming down to a guarantee that nothing written in the book would cause any data loss or other damage to users’ data as a result of following instructions in the book.
a phrase that would require the author(s) to fully indemnify the Publisher fully against all losses, damages and costs, including any sums paid to settle claims, arising out of the breach of one of those warranties.

These terms would have - in the worst case - meant an incalculable legal risk to us. We suggested adding “to the best of the authors’ knowledge” as a limiting clause to all of the above items.

The only other “major” issue for us was an early termination clause from the authors’ side. The original contract proposal only had a provision for the publishing company to terminate the contract for certain reasons before the book would be finished. No such regulations was contained in the contract, for example in case of a severe illness or other personal events that would prohibit us from finishing up the writing.

All other terms were mostly fine with us, even though we, of course, had several detail questions about individual pieces. Those were all answered comprehensively and mostly met what we already had expected.

We offered to waive any advance payments in return for the addition of an early termination option from our side, assuming that this might be a problem for the publisher otherwise. Moreover we were looking at the whole project mostly from a curiosity and “new experience” type of view. None of us had any expectations or hopes to earn a lot of money, so we were quite ready to be paid on after actual copies of the finished book had been sold.

Publisher’s reaction

We had feared that we would be getting into some sort of lenghty discussion with the publisher about the changes we wanted to make to the contract, assuming that the initial version they had sent us was something that had been carefully designed and worked out by their legal team and was not subject to change very much.

Much to our surprise, however, all the changes we requested were applied without any further ado. We received a new version of the contract some time after we had sent out our feedback. In that new revision the first issue mentioned in the list above (liabilities in case of schedule missed) was deleted completely - we had merely asked for a clarification about what would be considered a “miss”.

The “to the best of the author’s knowledge” clause was added without further discussion to all the paragraphs we had asked for. Only in one case had there been a little misunderstanding as to how we wanted a particular sentence to be phrased, but that was fixed very quickly, too.

Once we had the proposed final contract in our hands, we let Udo’s sister, who is a lawyer, have another look at it and make sure we had not missed anything important. Once she gave her ok, we could sign the paper copies and send them back to the UK, starting the actual writing.

After the fact

Apparently the first contract draft was not as binding from the publisher’s side, as we had initially expected. We were somewhat nervous about the amount of discussion we would get ourselved into when proposing changes to it, that is why we were willing to give up the advance payments. Turns out that would not have been necessary at all. Some time after we had finished the book I heard - by pure coincidence - several people in podcasts and on the net claiming that one should never ever accept the first draft of any author contract. Judging from the very first and only one I read and having statements like this from experienced authors, I tend to agree :)

So if you are currently - or in the future - in contact with a publisher, I suggest you take a close look at their proposed contract and contact them about anything that is unclear. Ask questions and do not hesitate to demand changes whereever you do not like something about it or consider it too one-sided. I do not mean you should be needlessly aggressive, but in the end this is about what you will be investing a considerable amount of time and work into for the next several months, if not more, so I recommend being frank and not shy when it comes to the legal document covering that.

TotalFinder - A must have addition to Mac OS X

2011-01-18T01:29:00.000+01:00

As a former long time Windows user I have quite successfully (and since Steam has arrived completely) transitioned to the Mac platform. It took some time, but over the past two and a half years I have found replacements and substitutes for most programs I used regularly on Windows for everyday tasks and must say I don’t look back.

Ok, there are some things I installed to make the base system a little more comfortable, e. g. by installing Better Touch Tool or iStat Menus which allow for better leveraging of available hardware features. But other than that - I haven’t made up my mind about TextExpander yet - there is little left to be desired.

Except: File and window management… Say what you will about Windows, but one are where it still runs circles around stock Snow Leopard is window and file management. Aero Peek - docking Windows to the screen edges and having them become exactly half the screen size - is a welcome addition to BTT, and Exposé is also usable. But still, the Windows explorer IMHO is still a better file manager than the Finder.

I cannot even tell you the exact details of what I am missing, but the fact alone that there is no default system shortcut like Win+E which brings up a new Explorer/Finder window is a major downside of the Mac OS.

Quite recently, however, I came across TotalFinder in some podcast (sorry, cannot really remember, might have been Bits und so) and gave it a try. It’s author Antonin Hildebrand took the open source code that is used in Google’s Chrome browser to implement the tabs and put together a nice little piece of software that pimps your Goold-Ole-Finder.

There are lots of features, the primary one being the addition of said tabs to regular Finder windows, which itself already helps to tame the window clutter you normally end up with when using the Finder. Basically, when TotalFinder is installed, instead of opening a new window for each directory you want to use, a new tab is added to the last Finder window that was opened. Of course, you can still have multiple windows, and you can even tear off existing tabs and spawn them into new windows that way, just as you can do with websites in Chrome.

But my favorite feature of all is what Antonin calls the Visor: A configurable hot key slides up a (optionally multi-tabbed) Finder window from the bottom of your screen - similar to the way the iPhone brings up the keyboard. It works just like any other (Total)Finder window and automatically disappears once it loses focus or you hit the escape key (also configurable). IMHO this is insanely useful, because you never have to worry about filling up your desktop with countless Finder instances - they are all nicely kept in tabs, out of sight by default but just a single keystroke away and all at your disposal at the same time. This alone would be a great tool already.

But there’s still more… If you are like me, you like the classic two-column file manager paradigm of Directory Opus or Norton Commander fame, depending on which platform you grew up on. TotalFinder has that, too. Although it does not always work perfectly, probably because it has to be a rather hacky way to integrate this into the OS, I still find the “Dual Mode” a cool feature. It just splits a Finder window, including the Visor, in half vertically and mirrors the source list on both sides. That way you can easily choose two directories to be visible at the same time and move or copy files between them without switching windows or tabs. This, too, is just a shortcut away.

All in all, after two weeks of trying it for 8 to 10 hours a day on my MacBook Pro without a single crash, I can wholeheartedly recommend total finder. And prices between $10 and $15 should not really cause you much trouble, either. Depending on what you like, this is less than one or two iPad games - but TotalFinder is just so much more useful :)

iOS 4.2.1 battery drain on iPad

2010-11-28T18:12:00.001+01:00

The Battery Life Thread

Following the iOS 4.2.1 release in the news and in forums I came across a discussion thread in Apple’s forums that is about the issue of some users having extremely poor battery life after the upgrade from iOS 3.2. I have not yet upgraded to 4.2 and I might in fact hold on to 3.2 for a little longer for other reasons, so I don’t know yet, if I will be affected the same way. There are some users claiming that this is what is to be expected now with multitasking and several apps taking up resources at the same time. Reading through these, I could not help to notice that there seems to be a fair bit of confusion and misinformation going on about how multitasking might contribute to this issue

So from a developer point of view, let me just state a few things. Please note that the general tone of this is that the “Apple/iOS version” of multitasking is by no means comparable to what you know from Mac OS or Windows, so many analogies are just not correct.

iOS multitasking – A not-so-layman explanation

I would like to put a disclaimer here that have not yet had the chance to develop anything “big” for iOS, but I do believe I know enough about software development in general and the iOS environment in particular to clarify a few things. Should I be wrong in any respect, I would be glad to be corrected by someone with a deeper knowledge in the comments.

Switching apps in iOS prior to v4

In versions of iOS without multitasking (let’s just call them ‘old’) whenever you hit the Home button, the app currently running gets a signal from the OS that the user wants it to shut down. The app then has a chance to do some housekeeping, e. g. store some information in its private storage area in the flash memory about what pages are currently open in a browser, a reading position in a larger text or the like. It has only a very short time to do this, before the OS forcibly terminates the application and returns you to the home screen. That is to ensure the user does not get the impression the device is stuck or feels slow to respond.

At this point, there is no running trace left of the app. In particular, no background downloads etc. are possible in this state.

Please note that this applies to 3rd party apps only, things like he iPod app or Mail directly from Apple can play by different rules, but stay with me.

When launching any app in the old iOS versions, one of the first things most developers do is make it look for the information it might have stored when it was last quit. Using this information it can be then possible to return the app to a state that is close to what the user left it in – for example open the same browser pages again or jump to the last known reading position. How close it gets to restoring the precise situation as before depends in how well he developer did with this task. There is no additional help from the iOS with this. That is why some apps come up with their default view every time they are launched, while others quite accurately allow you to resume where you stopped.

The key point is: Once you are on the home screen, nothing can use up any battery that is not from Apple itself.

Putting apps to sleep with iOS 4

Now, with the advent of “multitasking a la iOS” a few things changed in that sequence described above. First, everyone might want to have a look at Steve’s keynote speech (available in iTunes as a free video podcast) where he explained how they did multitasking “right” to prevent the typical battery drain that is often seen on e. g. Android devices which feature something much closer to true desktop type multitasking.

One of the key points to realize here is that with iOS there is no such thing as a “generic” background application. Instead, there are a few (7 if I remember the keynote correctly) specific things that iOS (‘new’) allows developers to use. For example one of these is background audio playback, allowing other apps to behave similar to the iPod app. Another thing is finishing an up- or download that might be in progress when the home button is pressed. And a few more…

However, all these tasks are very specificly designed for by Apple and cannot be extended by app developers on their own – which is why certain kinds of background apps that exist on Android cannot currently be done with iOS.

Another thing to know is this: When the new iOS was introduced, a lot of apps got updates with release notes along the lines of “now supports multitasking”. What’s behind this is that developers have to explicitly opt-in to a different application lifecycle if they want to participate in the new multitasking world that is a little more complicated than what was described above.

Instead of just getting a note from the OS to shut down upon a home key press and then being terminated, the developer can choose to have his app “paused” instead. In essence, iOS will still tell the app about the home key event, but then it will not let it terminate, but just stop giving it any more resources, especially no more CPU time. In effect the program is still in memory at this point, but effectively just halted, as if you were hitting the Pause button on a DVD player.

The user can now start a different app from the home screen and work with it. The one she used immediately before is now displayed leftmost in the task switcher bar that comes up with a home button double click. Now this is important:

Just because an app is displayed in the task switcher does NOT mean, it is running! In fact, most apps are even ACTUALLY NOT running, but just consume some memory in their paused state. They do NOT cause any additional battery consumption, because they are halted!

The only way for an app to consume power/network bandwidth/any other resource is to use one of the aforementioned specially prepared iOS features. If you do not use an app with suchlike functionality, multitasking itself will NOT impact battery life.

Switching apps

What happens now, if you select an app from the task switcher? Let’s say you are currently writing a text in some sort of editor program and need to look up some things in a dictionary app you were previously using and that is now halted.

When you bring up the task switcher and hit the Dictionary icon, first of all the editor app is paused the same way. Once that is done, the dictionary will be unpaused and its GUI be animated to the front. Because it was just paused, it goes on whereever you left it. Simply speaking, it does not even know it was paused just a second ago (even though as a developer you do get notified and might want to do some additonal work like checking if the network is still there or the like to update your GUI).

At this time, the editor app is in that deep-freeze pause state. So in effect, iOS is not really multitasking in the sense that there are two or more apps running at the same time, it just makes switching between them much faster with this pause/resume trick. And because there is only one app running at a time, battery consumption is roughly the same as before, if all goes well.

Pause them… All?

There is one question that might come up in this: How many apps can be paused at the same time? Well, actually not too many, because iPads only have 256MB of RAM (not to be confused with the 16/32/64GB flash storage) and every paused app needs to fit into the remaining space that the currently running foreground app does not actually need itself. The iPhone 4 has 512MB, so it is less tense in there, but the principle stays the same.

Say if your editor was now waiting in the paused state while your dictionary is up, iOS might decide to really terminate it (just like the old iOS versions) if the dictionary app requested a lot of memory. If for example the dictionary had an integrated browser, surfing to graphics rich websites with it might require a lot of RAM.

In that case, in order to keep the foreground app running smoothly the OS might decide to reclaim the RAM the editor is currently taking up and just very uncerimoniously literally kill it in its sleep. This is why ‘well-behaved’ apps try to limit their memory usage to the minimum at all times to prevent others from being killed (and of course being killed themselves, because sleeping apps that use a lot of memory are an easier choice for the OS to kill).

If that happens, the app is still listed in the task switcher and will just be launched in the same way it would be in old iOS. From a user experience standpoint that is a very sensible choice Apple made here, because from the user’s perspective there should be no distinction between resuming an app from sleep or relaunching it – in the end the only thing that matters if that now she wants to use it again.

If the developer did his job well, the app will still resume to the state where the user left off, it just takes a little longer because the app is launched ‘cold’ and has to take care of that stuff itself, just like in the old iOS.

Explicitly terminating apps

Several times in the forums people point out that the price you have to pay for multitasking is manually teerminating applications when you want to preserve battery life. While for most apps we now know this is bogus advice, because they are not doing anything anyway when paused, it is nevertheless good to know that you actually can terminate an app manually.

When would thiis be helpful? Well, being paused and resumed as an app sounds fairly simple in general, but any developer with a little bit of experience will agree that most things are not as simple as they might at first seem. Without going into detail here, it is not very hard to have an app get into a f****d up state through any combination of programming mistakes.

In general people have learned that most problems can be cured by shutting down and restarting an application. While inconvenient, this used to work… That is, until iOS4 came along.

Because leaving an app by pressing the home button does not actually terminate it, any problematic state it might be in will be happily suspended by the OS and just as happily be resumed when you hit its icon again. BTW it does not matter whether you start it from the task switching bar or from the home screen.

If you have an app that is in such a pitiful state what can you do, short of going the Windows-Way of rebooting the whole OS? Well, you can remove it from the task switcher by holding on its icon in there until it shows the little red sign in the upper right corner. Pushing that sign will cause the app to be terminated, just as if you had been working with a memory hungry foreground app and the system had decided to evict it to free memory.

The fact that Apple itself calls all this “multitasking” leads to confusion and misconceptions all over the place, as Marco Arment of Instapaper fame also points out.

All nice and good, but I want my battery life back!

So where does this all leave us with respect to the problem of iOS 4 upgrades apparently sucking people’s batteries empty way faster than before?

Well, unfortunately all this knowledge is no silver bullet – it just helps us understand that this particular problem most probably does not have anything to do with iOS-style-appswitching at all…

If you are not using apps that do use any of e 7 background features – and you would probably know if you were, because these new features would usually be something you were looking for explicitly when buying the app – ~~using multiple apps ‘at once’~~ pausing and resuming, i. e. switching back and forth between multiple apps, will NOT cause significantly increased battery drain.

It also explains why nobody in the discussion forum so far has confirmed that removing apps from the task switcher (or not even starting any app after a reboot) really made any noticable difference.

This is the sound of inevitability

That all being said, of course life (and technology ;)) is even more complicated… Strange as it may sound, in fact, being a sibling/offspring of Mac OS X, iOS has always been a real multitasking OS from version 1 on the original iPhone, way before the Appstore even existed.

There are at any given time a lot of processes running, for example the home screen (“springboard”), background processes that monitor network signal strength, listen for incoming phone calls, push notifications etc. and several more.

And because that is so, there are a lot of variables in the game that can cause extensive battery drain. As with any piece of software there are certainly bugs in iOS. And as is often the case, bugs do not affect all users and device configurations the same way. If that were so, they would have been noticed by the iOS team and be fixed before the rollout of a new release.

But some problems only occur on certain hardware plattforms, certain timezones, certain configuration settings etc. Moreover problems might only arise on freshly installed systems, or only on updated ones, maybe even only if you went through multiple version uodates in a particular sequence.

The point is, that there is no way on earth all these combinations could be tested in advance. One can be very sure that a lot of tests were conducted before any serious update is rolled out.

However I have to also say that in my experience there have been a number of really serious quality problems with Apple products in the recent past (iPhoto 11 anyone?) and that Apple’s policy of being quiet about these until either being pressed very hard or until they just put out another update does not do their standing with customers very much good…

There is some discussion floating around on the net that a slightly higher energy consumption might come from the Find-My-iPhone feature that is now included for the iPad as well. Even though I did not yet see the effect myself I find it hard to believe that the huge drains we are talking about here are caused by this feature alone. But maybe making sure it is turned off can alleviate the pain until a real fix is found.

Conclusion

In the end, I assume there are one or more bugs in 4.2.1 that directly result in extremely quick battery depletion for some customers, but do not affect fine the majority of others. That is why there is so much heated debate about the whole matter.

While it does certainly not help anyone when people who do not have the problems just deny that there is one based on their own experience and try to make other people feel too stupid to use their devices, sadly those seem to be the loudest ones (see Phily_Phan’s posts, esp. the 3rd from the bottom).

I think I will try 4.2.1 regardless – if I am lucky, it will work just fine, but if not, I won’t hesitate to downgrade to he trusty 3.2 and wait for 4.2.2.

Collection Performance - Don't become too lazy

2010-11-17T00:33:00.002+01:00

Situation

A few weeks ago I was performance tuning some code that ran quite regularly and took more time than it should, judging from the complexity of what is was doing. As usual, by merely looking that the code there was nothing blatantly, obviously, complete and ridiculously wrong - after all, it worked correctly.

So I fired up trusty old JProfiler and had a look, just to find out that sometimes you can become just too lazy and reliant on libraries without realizing their characteristics deeply enough.

The Code

The following snippet shows the relevant method of the code under analysis. The surroundings are not really important, the heart of the matter is in here:

protected List<String> getCurrentChunkList() {
    List<String> all = getContext().getFullList();
    List<String> result = new ArrayList<String>(chunkSize + 2);
    int tempCount = 0;
    for (Iterator<String> allIter = all.iterator(); allIter.hasNext()
                && tempCount < chunkSize; tempCount++) {
        result.add(allIter.next());
        if (tempCount == chunkSize) {
            while (allIter.hasNext()) {
                String current = allIter.next();
                if (tCurrent.equalsIgnoreCase(result.get(result.size() - 1))) {
                    result.add(tCurrent);
                } else {
                    break;
                }
            }
        }
    }
    all.removeAll(result);
    return result;
}

Basically all this does is take a portion of a long list of Strings and transfer them to a partial list with some data specific fiddling at the end of a chunk. Once that has been completed, the elements transferred are removed from the primary list.

The whole thing is part of a batch processing system that is only allowed to run for a limited time slice at a time. So the overall list of items that still needs to be processed is stored persistently in a context object and a certain amount of those is extracted, the master list shortened and the resulting chunk returned to the actual processing in each run. Control is then returned to the surrounding framework that can assign a new time slice to this kind of job, or at different one at its discretion. This will be repeated until the master list is empty.

The all list contains about 10,000 elements, the chunk size is set to 1,000. Now, this does not look too bad, does it?

First run

When running the processing job for the mentioned 10,000 elements in chunks of 1,000 the whole process took 490 seconds; of that the above method accounted for a whopping 6,3% or 31 seconds! A little long considering that it is just calling a few supposedly well-optimized collection operations…

But looking at it more closely, these “few” operations are a few more. With 1,000 elements per chunk, there were

10 invocations of getCurrentChunkList
10,000 invocations of Iterator.hasNext
10,000 invocations of Iterator.next
10,000 invocations of ArrayList.add
10 new ArrayLists
10 new Iterators
10 invocations of ArrayList.removeAll

Not exactly minimalistic, but 31 seconds seems a little over the top.

The problem lies with the ArrayList.removeAll call - internally an ArrayList is using a regular array as a backend store. The implementation of removeAll is just a looped call to the remove method for a single element. remove will then internally iterate the array and do an equality check of the element to be removed, finally removing it from the array when found.

Unless the element to be removed is at the end of the backing store array, there is a lot of bookkeeping to do, because removing something from the middle of an array is not an easy thing to do - in any case a good chunk of the array needs to be shifted over to fill the gap.

While this is using the fairly efficient System.arraycopy in its implementation, it still adds up, because we the algorithm shown above removes the elements it picks for processing from the beginning of the list, leading several thousand array shifting and resizing operations.

For reference, this is the relevant part from the source of ArrayList (Edit: from Apache Harmony!):

public E remove(int location) {
    E result;
    int size = lastIndex - firstIndex;
    if (0 <= location && location < size) {
        if (location == size - 1) {
            result = array[--lastIndex];
            array[lastIndex] = null;
        } else if (location == 0) {
            result = array[firstIndex];
            array[firstIndex++] = null;
        } else {
            int elementIndex = firstIndex + location;
            result = array[elementIndex];
            if (location < size / 2) {
                System.arraycopy(array, firstIndex, array, firstIndex + 1,
                                 location);
                array[firstIndex++] = null;
            } else {
                System.arraycopy(array, elementIndex + 1, array, elementIndex,
                                 size - location - 1);
                array[--lastIndex] = null;
            }
        }
        if (firstIndex == lastIndex) {
            firstIndex = lastIndex = 0;
        }
    } else {
        throw new IndexOutOfBoundsException(
            // luni.0A=Index: {0}, Size: {1}
            Messages.getString("luni.0A", //$NON-NLS-1
                Integer.valueOf(location),
                Integer.valueOf(lastIndex - firstIndex)));
    }
    modCount++;
    return result;
}

Edit: Once I had posted this - including the previous source snippet from the DocJar Website which I came across when looking for "ArrayList.java source", I found it should also have worked when removing elements from the front of the master list - but this did not work, when I tried it out on my machine. Turns out, I grabbed the Apache Harmony implementation by accident, which is implemented more efficiently than the one Sun's JDK6 includes. That's where the following snippet is from, which I had used in my profiling and that does not have the "remove from front optimisation" as the implementation above.

public E remove(int index) {
    RangeCheck(index);

    modCount++;
    E oldValue = (E) elementData[index];

    int numMoved = size - index - 1;
    if (numMoved > 0)
        System.arraycopy(elementData, index+1, elementData, index, numMoved);
    elementData[--size] = null; // Let gc do its work
    return oldValue;
}

Possible optimisations

As one can see from the source of ArrayList.remove (Sun JDK6) above, the ideal case would be to remove elements from the end of the array, because that does not require any element shifting and array copying whatsoever. The remove operation can simply be done by setting the last element to null and decrementing the lastIndex.

So I changed to code of the initial method to the following:

protected List<String> getCurrentChunkList() {
    List<String> all = getContext().getFullList();
    List<String> result = new ArrayList<String>(chunkSize + 2);
    int tempCount = 0;
    for (int i = all.size() - 1; i >= 0 && tempCount < chunkSize; i--, tempCount++) {
        result.add(all.remove(i));
    }
    return result;
}

This code removes strictly from the end of the all list and adds each element to the current result chunk. Going this route also allowed for the special fiddling in the middle of the old method to be removed, even though that did not contribute any measurable amount of overhead to the method’s execution time. The call to removeAll is completely gone, because remove already takes care of that individually.

Second run

Now, with the same data set the profiling comes to a very different conclusion:

10 invocations of getCurrentChunkList
10,000 invocations of ArrayList.remove
10,000 invocations of ArrayList.add
10 new ArrayLists

With this new configuration the percentage of overall runtime for this particular piece of code was reduced to a mere 23ms, less than 0,005% of the total execution time.

Conclusion

Even though the Java Collections framework’s classes are well tested and contain lots of performance enhancing shortcuts, it is still vital to be aware of their characteristics. Even though the original method looked perfectly ok at first glance and for a few runs would not show up on the radar, the repeated execution time added up to quite a remarkable percentage of the overall process.

With just a few minor modifications - that made the code easier to understand, too, getting rid of intermediate Iterators and special handlings for edge cases - that single method’s performance was boosted significantly. Considering that it is part of a more general batch job framework, overall improvements are even greater than what this single isolated test case showed.

No Google Apps for me

2010-10-05T23:39:00.000+02:00

I am a huge Google fan and especially love GMail. I have been a member from very early on and gone through all their improvements. Currently there is a mail archive of about 1GB in my account, going back to February of 2005.

Recently my friend and colleague Udo made me aware of the fact that Google Apps for your domain offers a free standard edition for up to 50 users. That seemed to be the perfect solution for me, because I figured that would allow me to use a custom sender address when sending mail from my iPad or iPod Touch instead of my @gmail.com address. Right now, if you want push notifications for new mail, you need to set up an Exchange account for GMail on the iOS device, but this will not allow for custom sender addresses. In the end I am not even sure this would have worked, but for other reasons I decided to abort the setup half way through for a IMHO very stupid limitation.

But let's start from the beginning.

To set up Google Apps for your domain, you need to first verify that you actually own the domain you want to prepare. You can do so in multiple ways - put an HTML file up there or add a DNS record. Since my provider allows for completely custom DNS configurations I opted for a TXT record with a Google provided identifier value.

Once that had propagated the Google Apps setup wizard let me continue to set up the different services like Google Calendar and Google Mail. For each I set up the user accounts, a CNAME and the MX DNS records as per Google's instructions.

Some time (and DNS propagation) later I found my new inbox ready to use, but empty of course. So next I went ahead and looked for instructions on how to migrate my existing GMail archive from the regular GMail site to my custom domain...

At first I thought I was just overlooking something, but after a little searching around it dawned on me that there is no way to do that! There are workarounds described all over the place using POP3 fetching, but that could take days and sometimes abort with timeouts etc. - nothing I was too eager to try.

In the end I reverted all my DNS changes and deleted the Google Apps account without ever really using it for the sole reason that Google seems to be unable - or perhaps unwilling? - to transfer an existing account from GMail to GMail... Anyone care to explain this to me?

Code Generation with Xtext

2010-08-07T01:51:00.002+02:00

Recently I attended a local rheinJUG meeting in Düsseldorf. While the topic of the session was Eclipse e4, the night’s sponsor itemis provided some handouts on Xtext which got me very interested. The reason is that currently at work we are developing a mobile Java application (J9, CDC/Foundation 1.1 on Windows CE6) for which we needed an easy to use and reliable way for configuring navigation through the application.
In a previous iteration we had – mostly because of time constraints – hard coded most of the navigational paths, but this time the app is more complex and doing that again was not really an option. First we thought about an XML based configuration, but this seemed to be a hassle to write (and read) and also would mean we would have to pay the price of parsing it on every application startup.
Enter Xtext: An Eclipse based framework/library for building text based DSLs. In short, you just provide a grammar description of a new DSL to suit your needs and with – literally – just a few mouse clicks you are provided with a content-assist, syntax-highlight, outline-view-enabled Eclipse editor and optionally a code generator based on that language.

Getting started: Sample Grammar

There is a nice tutorial provided as part of the Xtext documentation, but I believe it might be beneficial to provide another example of how to put a DSL to good use. I will not go into every step in great detail, because setting up Xtext is Eclipse 3.6 Helios is just a matter of putting an Update Site URL in, and the New Project wizard provided makes the initial setup a snap. I assume, you have already set up Eclipse and Xtext and created a new Xtext project including a generator project (activate the corresponding checkbox when going through the wizard). In this post I am assuming a project name of com.danielschneller.navi.dsl and a file extension of .navi.
When finished we will have the infrastructure ready for editing, parsing and generating code based on files like these:

navigation rules for MyApplication

mappings {
 map permission AdminPermission to "privAdmin"
 map permission DataAccessPermission to "privData"

 map coordinate Login to "com.danielschneller.myapp.gui.login.LoginController" in "com.danielschneller.myapp.login"
 map coordinate LoginFailed to "com.danielschneller.myapp.gui.login.LoginFailedController" in "com.danielschneller.myapp.login"
 
 map coordinate MainMenu to "com.danielschneller.myapp.gui.menu.MainMenuController" in "com.danielschneller.myapp.menu"

 map coordinate UserAdministration to "com.danielschneller.myapp.gui.admin.UserAdminController" in "com.danielschneller.myapp.admin"
 map coordinate DataLookup to "com.danielschneller.myapp.gui.lookup.LookupController" in "com.danielschneller.myapp.lookup"
}


navigations {
 define navigation USER_LOGON_FAILED
 define navigation USER_LOGON_SUCCESS
 define navigation OK
 define navigation BACK
 define navigation ADMIN
 define navigation DATA_LOOKUP
}


navrules {
 from Login
  on navigation USER_LOGON_FAILED
   go to LoginFailed
  on navigation USER_LOGON_SUCCESS
   go to MainMenu

 from LoginFailed
  on navigation OK
   go to Login

 from MainMenu
  on navigation ADMIN
   go to UserAdministration
   with AdminPermission
  on navigation DATA_LOOKUP
   go to DataLookup
   with DataAccessPermission
  
 from UserAdministration
  on navigation BACK
   go to MainMenu

 from DataLookup
  on navigation BACK
   go to MainMenu


}

As you can see it is a nice little language for defining coordinates in an application, meaning a specific GUI for a certain task and the possible navigation paths between them. Optionally a navigation path can be tagged to require one or more permissions to work. So for example one possible navigation path shown in the above sample is from the applications main menu, identified by the identifier MainMenu and represented in code by the com.danielschneller.myapp.gui.menu.MainMenuController class in the com.danielschneller.myapp.menu OSGi bundle to a GUI identified as DataLookup, implemented by com.danielschneller.myapp.gui.lookup.LookupController in the com.danielschneller.myapp.lookup bundle.
For this path to be taken, the application must request the DataLookup navigation path and the currently logged in user be assigned the DataAccessPermission. What exactly that means is not the focus of this tutorial, suffice it to say that we somehow need to get the information contained in this specialized language into our Java application in some shape or form that can be evaluated at runtime. In the following example all information will be transformed into a HashMap based data structure. For our little mobile application this has several advantages over the XML option mentioned earlier:

No XML parsing necessary on application startup, saving some performance
Validation of the navigation rules ahead of time, preventing parse errors at runtime
No libraries needed to access the information – by putting everything in a simple HashMap we do not have to rely on any non-standard classes whatsoever

First thing I did when I started with Xtext was define a sample input file such as the one above. Then – following its general structure – I began to extract a formal grammar for it. Of course, the first draft of the sample data was not perfect, over the course of a few iterations I refined some of the syntax, but in the end this is the grammar definition I came up with. It is heavily commented to allow you to copy it out and still leave the documentation intact:

grammar com.danielschneller.navi.NavigationRules with org.eclipse.xtext.common.Terminals

generate navigationRules "http://com.danielschneller/fw/funkmde/navi/NavigationRules"

/*
 * The top level entry point for the file.
 * "Root" is just a name as good as any, but
 * makes the meaning quite clear.
 */
Root:
    // first thing in the file is a "keyword",
    // followed by an attribute that will be
    // accessible as "name" later and allow
    // definition of an ID type of thing.
 'navigation rules for' name=ID
 
 // after the keyword and "name" attribute
 // three sections follow, each assigned
 // to an attribute for later reference
 // (called "mappingdefs", "transitiondefs"
 // and "ruledefs").
 // Their types are defined later in the file.
 mappingsdefs=Mappings
 transitiondefs=TransitionDefinitions
 ruledefs=NavigationRules
// semicolon ends the definition of "Root"
;



// mappings section >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
/*
 * Definition of the "Mappings" type used in
 * the "Root" type. 
 */
Mappings:
    // first the keyword "mappings" is expected,
    // then an open curly
 'mappings' '{'
 // after that a collection of "Mapping"s is
 // expected. The "+=" means that they will
 // all be collected in a collection type element
 // called "mappings" for future reference.
 // The "+" at the end means "at least one, but
 // more is just fine".
 (mappings+=Mapping)+
 // finally the "Mappings" type requires a closing
 // curly brace.
 '}'
// semicolon ends the definition of "Mappings"
;

/*
 * Definition of a single "Mapping", those we are
 * collecting in the "mappings" attribute of the
 * "Mappings" type.
 */
Mapping:
    // each mapping starts with the keyword "map"
    // and is followed by an element of type "MappingSpec"
 'map' MappingSpec
;

/*
 * Definition of a "MappingSpec" element. This is
 * actually just a "parent type" for two more specific
 * kinds of "MappingSpec":
 */
MappingSpec:
    // no keywords are defined here, a "MappingSpec"
    // can be either a "PermissionMappingSpec" or a
    // "CoordinateMappingSpec". Any of these will be
    // fine where a "MappingSpec" is asked for.
 PermissionMappingSpec | CoordinateMappingSpec
;

/*
 * Definition of a "PermissionMappingSpec" element.
 */
PermissionMappingSpec:
    // first the keyword "permission" is required.
    // then a "name" attribute is expected of type ID.
    // Following the name the "to" keyword is expected,
    // followed by a string that is stored in the "value"
    // attribute
 'permission' name=ID 'to' value=STRING
;

/*
 * Definition of a "CoordinateMappingSpec" element.
 * The definition is very similar to the "PermissionMappingSpec"
 * but has more attributes.
 */
CoordinateMappingSpec:
    // first the keyword "coordinate", then an ID stored as "name",
    // the keyword "to", followed by a string stored as "controllername",
    // next the keyword "in" and finally another string, memorized as
    // "bundleid"
 'coordinate' name=ID 'to' controllername=STRING 'in' bundleid=STRING
;
// <<<<<<<<<<<<<<<<<<<<<<<<<<<<< mappings section



// >>>>>>>>>>>>>>>>>>>>>>>>>>>>> navigations section
/*
 * Definition of the "TransitionDefinitions" type used in
 * the "Root" type. 
 */
TransitionDefinitions:
    // first, this element is introduced with the "navigations"
    // keyword, followed by an open curly brace.
 'navigations' '{'
    // after that a collection of "TransitionDefinition"s is
 // expected. The "+=" means that they will
 // all be collected in a collection type element
 // called "transitions" for future reference.
 // The "+" at the end means "at least one, but
 // more is just fine".
 (transitions+=TransitionDefinition)+ 
 // the element ends with a closing curly brace
 '}'
;

/*
 * Definition of a "TransitionDefinition" element. This
 * one is very simple.
 */
TransitionDefinition:
    // the keyword "define navigation" is required first,
    // then a "name" attribute of type ID is expected.
 'define navigation' name=ID
;
// <<<<<<<<<<<<<<<<<<<<<<<<<<<<< navigations section


// >>>>>>>>>>>>>>>>>>>>>>>>>>>>> navrules section
/*
 * Definition of the "NavigationRules" element.
 */
NavigationRules:
    // Element starts with the keywords "navrules" and
    // open curly.
 'navrules' '{'
 // collection attribute called "rules", consisting
 // of one or more occurrences of a "Rule" element.
 (rules+=Rule)+
 // element finishes with a closing curly keyword
 '}'
;


/*
 * Definition of a "Rule" element as used in the "NavigationRules"
 * element.
 */
Rule:
    // first the "from" keyword, then a reference  to one of the
    // coordinate mappings defined earlier. This time no new
    // definition of a coordinate is required, but one of those
    // that have been listed before. So the type here is put in
    // square brackets
 'from' source=[CoordinateMappingSpec]
 // following the source specification, one or more "Destination"
 // type elements are expected, collected in a collection attribute
 // named "destinations"
 (destinations+=Destination)+
;

/*
 * Definition of a "Destination" type. These are collected
 * in a "Rule".
 */
Destination:
    // first comes an "on navigation" keyword. After that a
    // reference to one of the Transition elements defined
    // in the "navigations" section is required and stored
    // in the "transition" attribute.
    // after that follows a "go to" keyword and a reference
    // to a coordinate mapping, stored in the "target" attribute.
    // finally - as with the "destinations" collection attribute
    // in the "Rule" element - a "permissions" collection is
    // defined to store none or more (*) "PermissionReference"
    // elements.
 'on navigation' transition=[TransitionDefinition]
 'go to' target=[CoordinateMappingSpec]
 (permissions+=PermissionReference)*
;

/*
 * Definition of a "PermissionReference" type. This is used
 * in the "permissions" collection of a "Destination".
 */
PermissionReference:
    // first, a "with" keyword is expected. After that a
    // "permission" attribute stores a reference to one of
    // the previously defined permission mappings from the
    // "mappings" section.
 'with' permission=[PermissionMappingSpec]
;
// <<<<<<<<<<<<<<<<<<<<<<<<<<<<< navrules section

This is what XText can digest and create an editor plugin and outline view for. Just save this as navigationRules.xtext – when you created the XText project in Eclipse using the wizard it should have been prepared for you.
Copying and pasting this into a .xtext file in Eclipse will provide you with syntax highlighting, code completion and syntax checking, making it easy to play around with grammar files.
Once done, right click the .mwe2 file lying next to the grammar file in the Package Explorer view and select Run As MWE2 Workflow from the context menu. This will take a moment and generate several classes, both in the current (XText) project and the accompanying ...ui project.
Next, right click the Xtext project and select Run As Eclipse Application from the context menu. This will bring up another Eclipse instance with the newly created support for navigation rules files (with a .navi suffix) installed.
To try it out, just create a new project and in that a new file. Make sure its name ends in .navi. When asked, make sure to accept adding the Xtext nature to the project. You will be presented with a new, empty editor that already has an error marker in it. This is because according to our grammar definition, an empty file does not comply to all the rules we specified. Try hitting the code-completion shortcut (Ctrl-Space) twice and see what happens:

The first code-completion fills in the navigation rules for part. According to the grammar this is the only valid text at the beginning of a file, so it is automatically inserted. Hitting Ctrl-Space again will tell you that now you need a Name of type ID. Just go ahead and try out the completion. It will help you create a syntactically sound navigation rules file. Notice that the Problems View tells you what is currently wrong. Also notice, that one you reach a part where references are expected by the grammar (e. g. when defining source and destination coordinates in a navigation rule) you will get suggestions based on what you entered earlier.
This is what the whole sample from above looks like in the editor:

While you are still fleshing out and fine tuning your grammar definitions, you will probably close this Eclipse instance and reopen it, once you repeated the Run As MWE2 Workflow steps in the main instance. In the long run I suggest you create a Feature and an Update Site project to allow easier distribution and updates of the intermediate iterations.

Generating Code

Now, as we have a complete Xtext DSL defined and in place let’s have a look at the Code Generation side of things. This part is completely optional: You are free to include the necessary Xtext libraries into your applications runtime (although they seem to be numerous) and just use them to dynamically load and parse .navi files on-the-fly. This would probably be a good idea if you were writing an Eclipse based application anyway. However, when targeting a very limited platform like JavaME this option is not viable. Instead we will now create a code generator that provides a transformation from the DSL syntax into more classic Java terms – specifically we will create a HashMap based data structure that carries all the same information, but in Java terms.

This is a sample of what the generated output is going to look like:

public class NaviRules {

    private Map navigationRules = new Hashtable();

    // ... 

    public NaviRules() {
        NaviDestination naviDest;
        // ========== From Login (com.danielschneller.myapp.gui.login.LoginController)
        // ========== On   USER_LOGON_FAILED
        // ========== To   LoginFailed (com.danielschneller.myapp.gui.login.LoginFailedController in com.danielschneller.myapp.login) 
        naviDest = new NaviDestination();
        naviDest.action = "USER_LOGON_FAILED";
        naviDest.targetClassname = "com.danielschneller.myapp.gui.login.LoginFailedController";
        naviDest.targetBundleId = "com.danielschneller.myapp.login";
        
        store("com.danielschneller.myapp.gui.login.LoginController", naviDest);
        
        // ========== On   USER_LOGON_SUCCESS
        // ========== To   MainMenu (com.danielschneller.myapp.gui.menu.MainMenuController in com.danielschneller.myapp.menu) 
        naviDest = new NaviDestination();
        naviDest.action = "USER_LOGON_SUCCESS";
        naviDest.targetClassname = "com.danielschneller.myapp.gui.menu.MainMenuController";
        naviDest.targetBundleId = "com.danielschneller.myapp.menu";
        
        store("com.danielschneller.myapp.gui.login.LoginController", naviDest);
        

  // =============================================================================
        
        // ========== From LoginFailed (com.danielschneller.myapp.gui.login.LoginFailedController)
        // ========== On   OK
        // ========== To   Login (com.danielschneller.myapp.gui.login.LoginController in com.danielschneller.myapp.login) 
        naviDest = new NaviDestination();
        naviDest.action = "OK";
        naviDest.targetClassname = "com.danielschneller.myapp.gui.login.LoginController";
        naviDest.targetBundleId = "com.danielschneller.myapp.login";
        
        store("com.danielschneller.myapp.gui.login.LoginFailedController", naviDest);
        
        // .... and so on ...
    }
}

The support class NaviDestination is omitted but is generally just a value holder struct type class.
When creating the Xtext project using the wizard earlier we created a third Eclipse project, ending in ...generator. Its src folder contains three subdirectories called model, templates and workflow. Put the sample .navi file into the model directory. It will serve as the input for the generator.

Create the first template

Code generation is based on templates. Xtext leverages the Xpand template engine. In the templates directory create a new Xpand template using the context menu. Call it NaviRules.xpt, open it and insert the following:

«REM»
    import the namespace defined in our DSL model
«ENDREM»
«IMPORT navigationRules»

«REM»
    Define a template called "main" for elements of
    type "Root". The minus sign at the end takes care
    of not adding a newline at the end of it.
«ENDREM»
«DEFINE main FOR Root-»

«ENDDEFINE»

As there is only one instance of a Root element in a navigation rules file, this will be the main entry point - hence the name. There is no need to call it main, but it seems fitting.
Now between the DEFINE and ENDDEFINE insert what is to be generated: As shown above, we need a new Java source file called NaviRules.java:

...
«DEFINE main FOR Root-»
«FILE "NaviRules.java"-»

«ENDFILE-»
«ENDDEFINE»
...

Again, the contents to be generated is put in between the FILE and ENDFILE brackets. Anything not enclosed in «» will be used verbatim in the output file. So first of all, put in the static parts of the Java file. What I did was first write the source for a single navigation rule by hand, made sure it compiled and then copied over the relevant parts into the template piece by piece:

...
«FILE "NaviRules.java"-»
import java.util.*;

public class NaviRules {

    public static class NaviDestination {
        String action;
        List requiredPermissions = new ArrayList();
        String targetClassname;
        String targetBundleId;
        NaviDestination() {};

        public final List getRequiredPermissions() {
            return new ArrayList(requiredPermissions);
        }

        // let Eclipse generate getters, setters,
        // equals and hashCode methods for this
    }

    private Map navigationRules = new Hashtable();
    
«ENDFILE-»
...

Now, this is nothing special so far. To fill in the elements from the navigation rules DSL file put in the following:

...
    private Map navigationRules = new Hashtable();

    public NaviRules() {
        NaviDestination naviDest;
«REM»
    Iterate all elements in the "rules" collection attribute
    of the "ruledefs" attribute of the "Root" element. Call
    each iterated element (which is of type "Rule") "rule" and
    expand the "ruletmpl" template for it here.
«ENDREM»
        «FOREACH ruledefs.rules AS r»«EXPAND ruletmpl FOR r»«ENDFOREACH»
    }
...

In the class constructor we first define a local variable naviDest of the previously declared type. Then - as the comment states - the FOREACH instruction will iterate over all Rule type elements. This might not seem to be completely obvious at first. Remember at this point in the template the current scope is the "Root" element from the navigation rules file. It has an attribute called ruledefs as per the grammer definition. This attribute is of type NavigationRules which in turn has a collection attribute called rules, containing of Rule type objects. Inside the loop the current element can then be adressed by the template variable name r. The loop body (between FOREACH and ENDFOREACH) contains another Xpand instruction to expand a template called ruletmpl which will be declared next.
Don't worry, even though this is a little difficult at first - switching contexts between the Java and the template scopes is made significantly easier in Eclipse, because the Xpand template editor will syntax color (static parts are blue) and also assist you with code completion inside the Xpand template parts. Ctrl-Spacing your way through it will make things more obvious than they are when reading an example.
Now for the ruletmpl template. Place it below the ENDDEFINE statement belonging to the main template:

...
«ENDFILE-»
«ENDDEFINE»

«DEFINE ruletmpl FOR Rule-»
        // ========== From «source.name» («source.controllername»)
        «FOREACH destinations AS d»«EXPAND destTmpl(source) FOR d»«ENDFOREACH»
        // =============================================================================
        
«ENDDEFINE»

You see the same idea used again: Static parts that get transferred into the output file 1:1 and Xpand statements that fill in data from the navigation rules definition file. In this case you see references to the attributes of the Rule element. As per the FOREACH instruction in the previous template, the one at hand will be repeated for every instance of Rule in our source file. Inside this definition the current scope is that of Rule, so with «source.name» the name attribute of the CoordinateMappingSpec object referenced as source in a Rule is taken first, then the controllername attribute likewise.
Next up another FOREACH loop iterates the one or more possible Destinations of each Rule. Instead of just applying a template (destTmpl) for every Destination we also pass in the corresponding CoordinateMappingSpec stored in the source attribute of the Rule. This is then used in the following template:

...

«DEFINE destTmpl(CoordinateMappingSpec source) FOR Destination-»
        // ========== On   «transition.name»
        // ========== To   «target.name» («target.controllername» in «target.bundleid») 
        naviDest = new NaviDestination();
        naviDest.action = "«transition.name»";
        naviDest.targetClassname = "«target.controllername»";
        naviDest.targetBundleId = "«target.bundleid»";
        «FOREACH permissions AS p»«EXPAND permTmpl FOR p»«ENDFOREACH»
        store("«source.controllername»", naviDest);
        
«ENDDEFINE»

«DEFINE permTmpl FOR PermissionReference-»
        naviDest.requiredPermissions.add("«permission.value»");
«ENDDEFINE»

In this innermost templates the attributes of the CoordinateMappingSpec objects source and target are accessed and put into place to be assigned to the members a NaviDestination Java object instance per Destination. There is only one more (very simple) template for the PermissionReference elements. With this, the Xpand file is complete.

Set Up The Generator Workflow

The wizard initially created a NavigationRulesGenerator.mwe2 file in the workflow folder. Open it and replace its contents with the following:

module workflow.NavigationRulesGenerator

import org.eclipse.emf.mwe.utils.*

var targetDir = "src-gen"
var fileEncoding = "Cp1252"
var modelPath = "src/model"

Workflow {
    component = org.eclipse.xtext.mwe.Reader {
        path = modelPath
        // this class has been generated by the xtext generator 
        register = com.danielschneller.navi.NavigationRulesStandaloneSetup {}

        load = {
            slot = "root"
            type = "Root"
        }
    }

    component = org.eclipse.xpand2.Generator {
        metaModel = org.eclipse.xtend.typesystem.emf.EmfRegistryMetaModel {}

        expand = "templates::NaviRules::main FOREACH root"
        outlet = {
            path = targetDir
        }
        fileEncoding = fileEncoding
    }
}

The most interesting parts of this workflow file are the load section in the Reader component and the expand and outlet sections in the Generator component:
The first one will connect a so-called slot with the Root element from our navigation rules.
The second one will trigger the evaluation of the main template in the NaviRules.xpt file in the templates folder and feed any Root instances it finds in the *.navi files from the src/model (modelPath) into it.
Now it is time for some actual generation.

Run the generator workflow

Right click the MWE2 file you just edited and select the Run As MWE2 Workflow command from the context menu. The Eclipse console will show this output:

0    [main] DEBUG        org.eclipse.xtext.mwe.Reader  - Resource Pathes : [src/model]
431  [main] DEBUG xt.validation.ResourceValidatorImpl  - Syntax check OK! Resource: file:/Users/ds/ws/ws36_xtext/com.danielschneller.navi.dsl.generator/src/model/MyApp.navi
1013 [main] INFO         org.eclipse.xpand2.Generator  - Written 1 files to outlet [default](src-gen)
1014 [main] INFO  .emf.mwe2.runtime.workflow.Workflow  - Done.

Then have a look at the newly generated contents of the src-gen source folder. If everything went alright, you should find a fresh NaviRules.java file placed there, based on the contents of your navigation rules file and the Xpand templates. Try and make some changes to the template, then re-run the workflow. You will see the changes reflected in the generated source file.

Generate a second source File

In the templates directory add another Xpand template file Navigation.xpt with the following content:

«IMPORT navigationRules»;

«DEFINE main FOR Root-»

«FILE "Navigation.java"-»
public final class Navigation {

«FOREACH ruledefs.rules.destinations.transition.collect(e|e.name).toSet().sortBy(e|e) AS t»«EXPAND actionTmpl FOR t»«ENDFOREACH»

    private final String name;
    
    private Navigation(String aName) {
        name = aName;
    }
    
    public String getName() {
        return name;
    }

}
«ENDFILE-»
«ENDDEFINE»

«DEFINE actionTmpl FOR String-»
    /** Constant for Navigation «this» */
    public static final Navigation «this» = new Navigation("«this»");

«ENDDEFINE»

This is a template for a type-safe enumeration that can be used in Java 1.4 - remember I had to do this for JavaME.
Notice the FOREACH loop in this case. It demonstrates that not only simple iterations are possible, but that Xpand allows more complex operations as well. In this case it will collect the names of all the navigation transitions from all the Destinations in the navigation rules. These are of type String. They are made unique by converting them to a Set datastructure and then finally sorted in their natural order. The resulting list of sorted strings is then iterated, each one - called t - is passed to the actionTmpl template. It is very simple, just placing the string itself («this») into a single line of Java source code.
Of course, strictly speaking this is a rather complicated procedure to get the same information we could also have taken from the TransitionDefinitions element in the rules definition. However I think it serves as a nice example for additional Xpand capabilities. For a full description of its possibilities, have a look at the Xpand Reference in the Eclipse documentation.
To use the new template, add another section to the MWE2 workflow definition:

component = org.eclipse.xpand2.Generator {
    metaModel = org.eclipse.xtend.typesystem.emf.EmfRegistryMetaModel {}
    expand = "templates::Navigation::main FOREACH root"
    outlet = {
        path = targetDir
    }
    fileEncoding = fileEncoding
}

Running it again will produce a slightly different output, making clear that two files have been generated. This is what comes out in the src-gen folder as Navigation.java:

public final class Navigation {

    /** Constant for Navigation ADMIN */
    public static final Navigation ADMIN = new Navigation("ADMIN");

    /** Constant for Navigation  BACK */
    public static final Navigation BACK = new Navigation("BACK");

    /** Constant for Navigation DATA_LOOKUP */
    public static final Navigation DATA_LOOKUP = new Navigation("DATA_LOOKUP");

    /** Constant for Navigation  OK */
    public static final Navigation OK = new Navigation("OK");

    ...

More...

This was just about my first experiments with Xtext. I am sure there is plenty more to be done with it. For more reading, please have a look at this very nice Getting started with Xtext tutorial by Peter Friese of Itemis.

[SCR] Found components with duplicated names inside their bundle!

2010-08-06T18:22:00.004+02:00

Today I was briefly confused by an error message issued by the OSGi Equinox runtime’s Declarative Services runtime which I did not understand immediately. For your – and my own – reference find the solution here.
First, this was the error message I got:

1281104579615=1::[SCR] Found components with duplicated names inside their bundle!
 This component will not be processed: Component[
    name = networksimulation
    factory = null
    autoenable = true
    immediate = true
    implementation = com.danielschneller.sim.network.NetworkStateSimulation
    properties = {devicevendor=DanielSchneller, simulation=true}
    serviceFactory = false
    serviceInterface = [com.danielschneller.network.NetworkState]
    references = {
        Reference[name = LOG, interface = org.osgi.service.log.LogService, 
        policy = static, cardinality = 0..1, target = null, bind = null, unbind = null]
    }
    located in bundle = com.danielschneller.sim.network_1.1.0.qualifier [49]
]

Apparently the component name networksimulation was used by two components in the same bundle. Well, that’s why you are encouraged to use package-name like identifiers; however this was not the problem. Changing it to com.danielschneller.sim.net.simulation which was guaranteed to be unique in my case still left the error the same:

1281104579615=1::[SCR] Found components with duplicated names inside their bundle!
This component will not be processed: Component[
    name = com.danielschneller.sim.net.simulation
    factory = null
    ...

To make it short, this is what happened: I had added this component’s description to a bundle that already contained some more using the Eclipse wizard for new service components.
That very fine wizard dutifully added the new filename to the Manifest is the Service-Component: line. Unfortunately in this case it led to:

Service-Component: OSGI-INF/*.xml, networksimulation.xml

At launch time apparently the wildcard was expanded, already including the new file, and then added at the end again. Apparently there is no sanitizing done (like adding the filenames to a Set instead of a List), which results in a second attempt of registering the same component a second time. ~~I will go and file a bug with Eclipse for this,~~ There already is a filed, but unfixed bug for this (Eclipse Bug #278540) but till then just make sure that when you use wildcards in the Service-Component Manifest header it does not cause any duplicates.

Java Object Initialization Order - Know your JLS!

2010-07-19T09:06:00.002+02:00

Recently I came across an interesting problem whose solution eluded me at first glance. Consider these three classes:

package com.ds.test;

public class Upper {
 String upperString;

 public Upper() {
  Initializer.initialize(this);
 }
}

package com.ds.test;

public class Lower extends Upper {

 String lowerString = null;

 public Lower() {
  super();
  System.out.println("Upper:  " + upperString);
  System.out.println("Lower:  " + lowerString);
 }

 public static void main(final String[] args) {
  new Lower();
 }
}

package com.ds.test;
public class Initializer {
 static void initialize(final Upper anUpper) {
  if (anUpper instanceof Lower) {
   Lower lower = (Lower) anUpper;
   lower.lowerString = "lowerInited";
  }
  anUpper.upperString = "upperInited";
 }
}

What output is to be expected from running the Lower class?

In this very reduced example it is much easier to get a view of the whole situation - in reality where this occurred there was a lot more code to distract one's attention...
Anyway, this is what the output looks like:

Upper:  upperInited
Lower:  null;

While the little example uses Strings, the real code of Initializer had a delegate object registered with the equivalent of the Lower class - at least that was the intention. For some reason however did this not work when running the application. Instead, the default path was taken - the one for the delegate object being not set (null).
Now, change the code of Lower slightly:

package com.ds.test;

public class Lower extends Upper {

 String lowerString;

 public Lower() {
  super();
  System.out.println("Upper:  " + upperString);
  System.out.println("Lower:  " + lowerString);
 }

 public static void main(final String[] args) {
  new Lower();
 }
}

The output is now:

Upper:  upperInited
Lower:  lowerInited

Notice the difference in the code?
Yes, the lowerString field is no longer explicitly set to null. Why would this make a difference? Isn't the default value for reference type fields (such as String here) null anyway? Of course, it is. However it turns out that this tiny little change - which apparently would not change the code's behavior in any way - makes this thing fly or not fly.
So what is going on? It becomes clear when looking at the initialization order:

main() calls the Lower constructor.
An instance of Lower is prepared. That means, all fields are created and populated with default values, i. e. null for reference types, false for booleans and so on. At this time, any inline assignments to the fields have not taken place!
The super-constructor is called. This is mandated by the language spec. So, before anything else happens, Upper's constructor is called.
The Upper constructor runs and hands a reference to the freshly created instance to the Initializer.initialize() method.
The Initializer attaches new Strings to both fields. It does so by using a somewhat dirty instanceof check - not a particularly good design pattern, but possible, nevertheless. Once that has happened, both the upperString lowerString references are no longer null.
The Initializer.initialize() call finishes, as does the Upper constructor.
Now it becomes interesting: Construction of the Lower instance continues. Assuming there is no explicit =null assignment in the lowerString field declaration, the Lower constructor resumes execution and prints out the two Strings that are attached to the fields.
However, if there is an explicit assignment to null, execution has a slightly different flow: Just after the super constructor is done, any variable initializers are executed (see section 12.5 of the Java Language Spec), before the rest of the constructor is run. In this case the String reference that was previously assigned to lowerString is now overwritten with null again! Only then does the rest of the constructor continue execution, now printing lowerString: null.

Apart from being a nice example for why it is handy to be aware of some of the minutiae of object creation (or knowing where to look in the JLS, printed or online) this shows why it is a bad idea to write the Initializer like this. It should not be aware of Upper's subclasses at all! Instead, if for some reason initialization of certain fields cannot be done in the Lower class itself, it will just require its own variant of some sort of initialization helper. In that case, it would really make no difference if you used String lowerString; or String lowerString = null; - just as it should be.

Set Scaled OS X Desktop Wallpaper via Automator

2010-06-02T23:48:00.001+02:00

I regularly enjoy the pictures Romain Guy posts on his blog. Often they make for great desktop backgrounds, but at a whopping 20 megapixels they are just a little too big for my iMac's 1920x1200 screen.
When today I came across his most recent picture called False Kiva I went through my usual list of steps:

Go to the "original size" version of the picture on Flickr
Drag it to the desktop
Open the file with a double click
Scale it to 1920x1200 in Preview.app
Save it to the Pictures folder
Set it as the wallpaper

It occurred to me, that apart from liking and downloading a picture, all the remaining steps could be automated with a little Automator service. So I recorded this sequence of steps and saved it as a service.

These are the steps in textual form:

Service receives selected image files in Finder.app
Copy Finder Items To Pictures
Add Text to Finder Item Names; Add Text_1920x1200 after name
Scale Images To Size (pixels) 1920
Set the Desktop Picture

After that I can choose to trash or keep the original file, a copy that does not take up more space than necessary for my screen now lives in the pictures folder.

On Writing a Book, Pt. 4 – The Tools (II)

2010-05-16T23:16:00.001+02:00

This is part four of an ongoing series about my experiences while writing the MySQL Admin Cookbook for Packt Publishing. All previous parts can be found under the mysql-admin-cookbook label.

This part will be about more software used in the process of writing the book. The last episode covered writing tools, file/version management and backups. What's up now is graphics programs, virtualization and PDF handling.

Outlining

For outlining and structuring thoughts I like mind-maps. I know they are not for everyone, but if you like them and do not want to spend a lot of money on MindManager, have a look at FreeMind. It is a free, open source, Java based mind mapping application with a large set of features and a really nice UI. In fact, I am using it right now to structure the writing of this series of blog posts. It comes in readily installable versions for Windows, Mac OS X and Linux, which makes it really a good fit if you regularly switch platforms and want to be able to add a thought or two to an existing map without having to dual-boot or fire up a virtual machine.

Graphics and Illustrations

As with any technology, some things in database land are much easier to show as a diagram or schematic drawing than explaining them verbosely in text. Not to forget the occasional – well, the regular – screenshot to show GUIs or console printouts.

Screenshots

As a matter of fact in the first drafts of most chapters Udo and I had a lot of textual presentations of query results and plain text output from the mysql command line client, but the publishing company wanted more screenshots instead to make the book appear less text heavy. There have been mixed reactions to this, but generally I think less would have been more. Several of the screenshots merely show console windows which are not always good to read due to scaling and anti-aliasing.

To take screenshots, Mac OS X is pretty well suited without any additional tools. Even though there are some nice programs that offer advanced features like Layers which can automatically create a PSD file with one layer per open window, usually screenshots were taken of a single Terminal.app window or specific dialogs. For that the built-in features were completely sufficient:

⇧+⌘+3 produces a snapshot of the whole screen
⇧+⌘+4 produces a crosshair that you can drag over a specific area of your screen
⇧+⌘+4 followed by the space bar turns the crosshair into a camera to snapshot a single window

The only adjustment needed was to disable the drop shadow that is otherwise put behind the windows in the screenshot files by default:

defaults write com.apple.screencapture disable-shadow -bool true

That change gets effective after logging out and back in again or by killing the SystemUIServer process.

The screenshots are placed on the desktop by default. Depending on the OS X release the filename and format may differ. For the book we wanted PNG files, because JPGs would have caused compression artifacts. PNG is the default in Mac OS X 10.6 Snow Leopard. Generally this command allows you to change the format of screenshot files:

defaults write com.apple.screencapture type png

Instead of png you can also choose from tiff, gif, jpg, pdf, if I am not mistaken.

For Windows I used is a handy little tool called Screenshot Pilot which has a slightly “nostalgic” GUI style, but it works really well. It is far more pleasant to use than the otherwise necessary Windows style screenshot taking by hitting Alt-PrtScr to copy the current Window to the clipboard and then paste it into mspaint.exe and save it as a file.

Illustrations

Very early in the process we asked Packt about how to go about illustrations. They told us that anything would be fine that worked for us, because they had professional illustrators who would re-draw anything we sent them to give the book a uniform style. For example they would accept drawing made with any of the Office tools, bitmap graphics or just scans of hand-drawn sketches. I was very relieved when I heard this, because if there's one thing that eats up time even more quickly than text formatting it certainly is graphics work. I think of myself as nothing close to an artist, so knowing that someone else with probably lots of experience and routine would take care of this was very comforting and would relieve us of the necessity to find a set of good graphics tools.

Or so we thought at the time... To give you an idea, this is one of the sketches I sent along with a chapter of text:

While certainly no masterpiece I think the idea is brought across well enough for someone to redraw this with a little style. I scribbled together a few of these and sent them off in good faith.

Later in the process, this is what I got back for the sketch above:

Frankly, I was very much disappointed and certainly did I not want graphics of this style and quality to go into the book. Just look at the tunnel entrance and exit symbols… So in the end I decided to take care of the graphics myself. I fooled around a little with OpenOffice's drawing tool but did not manage to create anything that I even remotely liked. Fortunately a friend of mine owns a copy of OmniGraffle, a professional vector based diagramming tool for the Mac, and he let me spend some time on his machine remotely. With this I managed to create new versions of the illustrations in a style that I liked and with a little more attention to detail. This is my version of the same idea:

Making these took quite a lot of time. Once finished with the bulk of the work I sent it off to the publisher in EPS format, because I hoped that would be the easiest way of making sure they had the best possible quality to work with without requiring them to organize a Mac. Turns out the EPS is not as portable as I thought and had hoped. There were lots of troubles with line and arrow styles, transparencies etc. In the end I had to download a trial version of Corel Draw for Windows, because that's what Packt's graphics people use.

With trial and error I managed to get the illustrations to Corel Format with the correct fonts and no strange rasterizing artifacts for partially transparent areas. All in all, had I known I would have to take care of this myself from the beginning, I would have asked for at least one more month in the schedule and prepared them along the way. The same goes for the very few charts included in the book. I made these with a trial version of OmniGraphSketcher, also from OmniGroup. The idea behind this tool is to actually draw diagrams, instead of having them generated by a spreadsheet tool from pure numbers. While I generally like the idea, the tool is still lacking in my opinion. For example getting the X- and Y-scales right was more complicated than I would have thought. Anyway, I will keep an eye on the program - it surely has the potential to become a valuable tool and I will most certainly check it out again if/when the need arises.

A few more words on Corel Draw: I have no option but to rant about it! The last version I remember to be really good was Corel Draw 5, about a millennium ago or so. After that I had a look at Corel Draw 7 and did not like it at all. In my opinion it has suffered form the same feature-creep that most applications that were once lean, fast and powerful, getting bigger and especially buggier all the time (Firefox, Nero anyone?). I am glad I have no need to work with this software on a regular basis. From real crashes - which I saw quite a few of - to annoying user experience problems; this is not a program you would enjoy to work with. The stupidest thing in my opinion was that for virtually any file operation the open or save dialogs would open in my home directory, instead of remembering which directory I had last used. An estimated three hours of my lifetime could have been saved, had I not had to spend them navigating to the right path time and again. Also what is it the myriad of file format versions? As you save a file you have to decide what version (7, 8, 9, 10, 11, 12) you would like to use. Of course they are all called ".CDR", and the program does not remember what you chose for the last file... Better make sure to include the format version in the filename, because once on the disk you have to easy way to tell which is which!

Test setups / Virtual Machines

As it is easy to imagine, for a database cookbook lots of experiments and trying out stuff is required in order to make sure that everything you claim actually works. Anyone who kept reading up to this point is probably technical enough to understand that no matter how well you plan and how carefully you write down any sort of script, source code sample or the like, it will not work unless you copy and paste it from the actual command line window or source code editor. There just is no other way. And also it is (almost) guaranteed that you will break any code by making "one last beautifying change" in the word processor; be it a missing semicolon, a space introduced into a regular expression or a capitalization change, something will go wrong!

The only chance to make sure your readers are not too likely to find a bug in your scripts or command lines is to try them all - each and every one of them. Of course, some of them by design make permanent changes to the test setup, so if anything goes wrong and you cannot just take a screenshot and put it into the manuscript you might be in for varying amounts of resetting the machine and trying again. More than once I made a simple mistake in that, too and ended up with an ever so slight difference in the test setup, messing up the test again. In that light, one wonders why any functioning production systems exist in the first place...

Fortunately this is not 1995 anymore and there are very capable virtualization solutions available for reasonable prices to ease the pain with repeated tests and different setups.

As Mac OS X is not the primary platform most MySQL administrators use - neither as a client, nor as a server - I need a way to test with different versions and setups of Windows and Linux. One way would have been to use bootcamp to dual-boot into Windows, but that would have caused unacceptable roundtrip times. Because performance was rarely an issue, I decided to buy VMWare Fusion 3. Though not very cheap it certainly is a product worth its money. Here you can see my Virtual Machine Library - some of the test VMs have been deleted since the book was finished - but you can still see the Ubuntu, Vista and Windows 7 VMs I primarily used to take screenshots on.

Apart from being able to run different operating systems and MySQL versions, the killer feature for me was snapshots. Basically they allow you to clone a virtual machine and all its state into one or more named copies and reverting back to these "save-games" at a later time, discarding any changes that happened in the meantime. Though not particularly suited for long term operations - performance is degraded quite a bit - they are ideal for what I call "destructive" tests: Those that actually modify system state, like configuration files, database contents, software installations etc. For every VM I would first set up a well-known "good state" and take a snapshot of that. If in the course of writing a recipe I needed to make several attempts to get everything just right I could mess everything up with no worries and go right back to the beginning, without any risk of accidentally missing something important.

If you regularly have to do any sort of software testing you will appreciate the benefits virtualization immediately. The fact that I chose VMware is not too important. As it is often with software, personal preference plays a huge role. Udo got a good deal on Parallels Desktop which is another commercial virtualization solution for the Mac. However you could just as well go with the free VirtualBox from Oracle, available for Windows, Linux and the Mac. They all offer very similar feature sets and would all have been equally suited for the tasks I performed.

PDF Tools

In the later stages of production annotated PDF documents superseded the previously used OpenOffice and Word document files. The reasons for and problems with that will be focused on in a later installment of this series.

Mac OS X incorporates very strong PDF handling facilities right in the operating system's core. Much of the Mac's graphics are based on PDF internally, so it is not too surprising to find advanced PDF handling features in the built-in "Preview.app". Out of the box any Mac can open PDF documents, inspect their meta-data and also create new PDFs from any application without the need for special PDF printer apps or the like. Before the book writing project I had never even bothered to install the Adobe Reader - I just did not have any use for it. Considering the fact that compared to Preview.app it takes ages to load and is a never-ending source of security problems I had no intention of changing that.

But then Packt started to send us PDF versions of the first formatted and laid out chapters, complete with annotations and comments. I blogged about a rather unpleasant episode regarding Preview.app's shortcomings earlier.

I won't repeat everything here, but suffice it to say that you should not trust it to show all the annotations and comments a document contains!

So I needed to download and install Adobe Reader to make sure I could at least see all annotations and meta-information embedded in the PDFs from the publisher. I have not checked again since, but when I last downloaded the installer from their website it was not the latest version. In order to keep your system (reasonably) secure against the many Adobe Reader based exploits circulating the net, make sure to manually run the updater again and again until it tells you there are no more updates left! If there had been updates like this in 1995, I am sure they would have been implemented like this...

The same goes for Acrobat Professional which in the further course of getting the book done was needed as well - again, details will follow later. For reasons beyond logic there is no trial version on Adobe's page for Acrobat Pro as a standalone application on the Mac - only for Windows. As a Mac user you need to download the full CS4 suite to try it... Fortunately I did not have to jump through these hoops, because a version 8 DVD was included in the software bundle with Fujitsu's ScanSnap. That one, too, had to be updated in what felt like 100 individual runs of the Adobe Updater.

While being considerably slower than the Preview.app accompanying Mac OS X, Acrobat and Adobe Reader were much better at handling document commenting, revising and especially comparing! Yes, it is possible to compare two PDFs and have Acrobat highlight changes between the two - a feature that became indispensable in the later project phases:

With that I will conclude this second part of the tool descriptions. I have probably forgotten one or two little utilities that came in handy, but these were the companions that accompanied me along the way constantly. In the upcoming episodes I will refer to them as needed, but you should have a general idea on what kinds of tools it took to complete the project. Be advised though that once my little saga here is complete you will find that quite a few of them shouldn't have been necessary for me to have, but that is for another time.

The next part will resume the more chronological style of describing my experiences. Stay tuned :)

On Writing a Book, Pt. 3 – The Tools (I)

2010-05-09T23:50:00.001+02:00

This is part three of an ongoing series about my experiences while writing the MySQL Admin Cookbook for Packt Publishing. All previous parts can be found under the mysql-admin-cookbook label.

Even though I said I would be presenting things in mostly chronological order, I think after the previous - rather dry - part, a little more technical and fun information would be nice for a change: The tools used to create the MySQL Admin Cookbook (well, at least those used by Udo and me). To give a detailed account of what software products we used during the whole experience I will split this topic up into multiple posts. Otherwise it would just become either way too long or I would have to leave out too much stuff than I am willing to.

Writing Tools

Packt provided us with both Microsoft Word and OpenOffice.org document templates. Authors are free to choose which format they want to use - if you own MS Office and like Word, you can use that, but if – like us – you did never buy a copy from Microsoft and don't want to do so for the book, OpenOffice is fine as well.

I have been using a Mac as my home computer for quite some time now, and I remembered OpenOffice as being rather slow and not too stable. Admittedly that was the result of only a quick glance at it, because I had bought iWork with the Mac and had been using Pages to write my everyday stuff. I was not too impressed with OpenOffice, but quickly learned that the compatible Mac version called NeoOffice was way more usable.

The document template provided by Packt contained several paragraph and character styles with names like “Code in Text”, “Keyword”, “Command Line” etc. Packt recommended writing with these styles to get a basic idea of what the book's pages would look like in print later. Also the page size was set close to the actual books format, in order to allow for a good estimate of the page count.

Even though NeoOffice was generally doing its job, I soon started looking for alternatives for two main reasons:

Speed

Even though more responsive than OpenOffice for the Mac OS, NeoOffice still did not feel as snappy and responsive as OOo on Windows did. After several writing session during which I cursed more than once about lagging responses to my clicks and typing I decided I had to find something else for the remainder of the book if I wanted not to kill myself or the Mac.
Distractions

While at first glance it might seem like a good idea to write in a way that let's you keep a close eye on the page count and do some basic formatting “in-line” while writing, it turns out to be a huge distraction. Even though I tried very hard to not get lost in fine-grained formatting, I regularly found myself going back a paragraph, because I had forgotten to use the “Code in Text” or “Keyword” template for some SQL example or other element. Also I regularly spent way too much time tweaking table column widths or just plain fighting one or the other obvious bug in the word processor.

So in addition to the speed issue mentioned earlier I figured it would take me far too long to finish a chapter in this manner. It is quite remarkable how much more productive I got once I stopped letting my thoughts and actual writing be interrupted by this constant formatter “background thread”.

I decided I would first write down the bulk of a chapter's content as a plain text file, using only the formatting tools available there: None. The only thing I did was separating paragraphs with blank lines in between and occasionally inserting easy to find tags in the form of ***INSERT REFERENCE HERE***. As nowhere in the real contents of the book you would find “***” this was very easy to locate later on using simple text search.

However, TextEdit - the Mac's default onboard text editor - even though more comfortable than Windows Notepad did not convince me as a heavy duty writing tool. After trying several options, including WriteRoom and a custom Pages template, I finally decided to buy myTexts which I also blogged about already. It combined the no-distractions-full-screen-writing with a nice automatic file handling which even relieved me of thinking about saving my material. Almost everything after the first chapter was written with this simple but powerful tool.

I kept one file per recipe written instead of one per chapter, even thought the tool would certainly have handled that well. But by keeping the individual recipes separate it became a lot easier to navigate and keep of track what recipes had already been written by just looking at the side-panel.

Once finished, I would just copy and paste the text in its entirety into the NeoOffice document and started a little formatting session. Once I had amended the original document template with some personal keyboard shortcuts for the style templates most commonly used I could manage to format a single recipe in maybe 5-10 minutes. Often I would first write several recipes in myTexts and then bring them over to NeoOffice in one go. I cannot provide any hard timings or comparisons as to how much time I saved writing in this manner, but I can firmly state that material written like that tended to be a lot more “production ready” than what I had written directly in NeoOffice earlier.

In fact, I have not since changed my writing habits again and continue to work like this whenever producing any sort of text longer than an email. I have since bought a license of WriteRoom as well and use them both sort of on a daily-mood basis. As a matter of fact, this very post is being written in WriteRoom right now :-)

Version Control

Both Udo and I being software developers we were already accustomed to the concept of using a version control system (VCS). More than once in my day job was I glad to have easy access to prior versions of files I had locally changed, but not necessarily for the better. So it was very natural for us to start using a VCS right from the beginning. Even before we started the actual writing we checked in all documents and files we had exchanged with Packt so far as well as the outline and all reference materials into a hosted subversion repository.

There are numerous choices for free or paid hosting services. A nice one that allows you to start for free and only pay if you need more than 200MB of space is http://www.unfuddle.com. We created a free account with a Subversion repository and used it till the end of the project. We were expecting to have to upgrade to a paid plan somewhere along the way, but in fact we were able to squeeze everything in this free entry level account until the book was finished. As for the choice of Subversion over other version control systems, it was merely a matter of what was available, and as we were dealing with ODT and other non-mergable file types we figured it would not make much of a difference. Of course, personal preferences vary - any other VCS, be it distributed or centralized, would work just fine. The key aspect just was having a (one more) off-site backup and the aforementioned ability to go back in time should the need arise.

Whenever we sent of a finished chapter or anything else important to Packt we tagged it in Subversion - which turned out to be really helpful during the final stages of the project. More details on how and why will be presented in one of the later incarnations of this series.

As for the Subversion client, this is a matter of taste as much as the VCS itself. If I remember correctly, Udo used TortoiseSVN on Windows, while I – after some attempts with various Mac utilities – stuck with Versions, a very nice and “Macintoshy” application.

Backup

If you have ever written a dissertation, thesis or the like you should already be paranoid enough to not trust any single machine to keep your data safe, be it from thieves, fires or even (most dangerous of all) your own stupidity. If you have not, you should quickly become paranoid, because even if you trust (which you should not) that your hardware will not fail anytime soon, and even if your house is right next to the police and fire department headquarters this will not prevent you from accidentally deleting the most recent version of the chapter you were just about to send off to the publisher. Or you overwrite it with an older version. Or you try to move a picture around in the word processor and it crashes horribly, damaging the file, or ... well, you get the point.

The following are the backup measures I take. It is, of course, up to you to decide which of these or others you want to employ, but regarding backup, my take is “the more the better”. I did not include the SVN repository here again, but of course it is a backup, too.

File Name Rotation

The simplest way to guard against yourself destroying your own work accidentally is to regularly save to a new file name. This of course does not play too well together with the myTexts approach where you do not actually see the regular “Save File As” dialog, but it is a good idea anyway. I got used to just append a number to the file name and count it up every hour or so. Of course I would save much more often than once per hour, having Cmd-S hardwired into my spine after every paragraph or so. But using multiple files locally is also a good protection against faulty software corrupting a file.
Time Machine / External Hard Drive

As a Mac user this is a no-brainer. Time machine runs every hour or so, backing up everything that has changed since the last cycle to an external hard drive. As a Windows user you do not have the luxury of having this built in to the OS, but a local external hard disk is a good insurance against a failing internal hard drive.
Dropbox

The free Dropbox service is easy to set up and automatically uploads any files you save in a specified local folder to their servers. Access is protected and it operates silently in the background. This is also great if you work on different computers, because it will keep them all in sync when they are connected to the same account. The free plan gives you 2GB of space, certainly more than you would ever need for your book texts.
Offsite Online Backup

I own a Carbonite subscription which will transparently backup all my files to their cloud service. There are a lot of these services with varying prices and services, but they all send your computer's files (not only one folder like Dropbox) to their servers. While not strictly necessary in addition to Dropbox, it does not hurt either. Any form of cloud-based storage, even if you just copied your files to an FTP server regularly, will protect you from fire, lightning strike or floods, thieves or any other physical damage to or loss of your equipment. I would never work seriously for extended periods of time on a machine that does not use some form of online backup.

One can hardly stress the importance of backups too much. For a very good concept called “3-2-1 backup strategy”, go to http://www.dpbestflow.org/backup/backup-overview#321 and start to live by it.

So much for now. This has already gotten longer than I had planned… The remainder of the tools and programs I used while writing the book will get their mention in the next part of this series.

On Writing a Book, Pt. 2 – Outline and Schedule

2010-04-30T00:53:00.003+02:00

This is part two of an ongoing series about my experiences while writing the MySQL Admin Cookbook for Packt Publishing. All previous parts can be found under the mysql-admin-cookbook label.

While last time I focused on the initial contact with the publishing company (just referred to as "Packt" from now on), this issue is about the process of putting together an outline proposal and coming up with things to write about in the first place. As from this point on in the process Udo was involved with everything, I will be referring to "us" and write "we" most of the time from now on.

The Publisher's Expectations

The only thing we knew about the books would-be content was a chapter template for Packt's cookbook series as well as the general description provided by Sarah earlier:

As I'm sure everyone is aware, MySQL is a relational database management system. Administrators of MySQL will be tasked with things such as maintaining the database, tuning the server, managing users etc etc.
This cookbook will have all the MySQL recipes an administrator could dream of, spanning from creating tables to managing views, from improving performance to securing the database, from using monitoring tools to using storage engines. DBAs of all levels will be catered for with recipes of varying difficulty, allowing the reader to administer MySQL to their hearts' content.

Before actually beginning the outlining process there were a few more emails sent back and forth, mostly questions on our part, for example whether we should include programming related materials or tool descriptions in addition to more "real" database themes, what spectrum of experience to expect from potential readers and so on.

Packt did not provide a list of topics they had in mind - instead we were completely free to make suggestions for what we deemed interesting for MySQL administrators. There were no hard rules on what topics would be out-of-scope, but in general we were asked to find a balance and suggest roughly the same number of topics that would be interesting for relatively inexperienced readers, people already familiar with MySQL and experienced, professional DBAs - with a general tendency toward the latter. At the time I was a little disappointed as to how little guidance they provided, but in hindsight I believe this was definitely for everyone's benefit. Packt's position on this apparently is that their authors generally are closer to their designated audience than someone who can only have so much of an insight into a lot of technical topics; something I can certainly agree on :).

As for the MySQL versions targeted, they naturally asked us to concentrate on the most up-to-date version at that time - which was MySQL 5.1. In our day jobs we are still stuck with 5.0, but it was a good opportunity to get a feeling for what changes we will have to make to our processes and practices when eventually we decide to tackle the upgrade. Considering the fact the writing the whole thing would take its time this was of course a reasonable decision. Midway during the process we thought about looking at 5.4, too, but decided against it - checking everything on 5.0 and 5.1 was enough work already.

The cookbook format as it was described to us would allow for "spin-off" recipes: A general recipe with detailed instructions, followed by shorter, more concise ones that described a variation of the original baseline, sometimes more advanced, sometimes just with a different focus. So in general Packt recommended to go for a greater number of shorter individual recipes, because that would allow the reader of the book to skim through the table of contents and not have to interpret too much into the titles in order to grasp what each recipe would do for her.
While we actually ended up with a few of these in the final book I did relate to the idea too much. As a matter of fact I think we could have better used those pages for genuinely different content, but that might simply be a matter of personal preference.

Brainstorming

On January 23rd, not just yet one month after the initial contact, we sat down together in front of Google Docs and held a little brainstorming session on topics that occurred to us while talking about our experiences with MySQL. Of course some of the "bigger" milestones from our day jobs sprang to mind first, but over the course of a couple hours with some snacks and drinks we came up with quite a lot of individual topics.

Looking back at it I think we might have better used a mind-mapping application like FreeMind, because when working with text documents one is always tempted to format or move around stuff, trying to find structure where there usually is none intended in brainstorming. On the other hand, brainstorming sessions are usually short, so maybe that made up for it :-)
There was lots of random talk about other stuff, too, but we consciously decided not to try and concentrate and focus too hard. Instead whenever someone had an idea - however stupid it might seem at the time - we added it to the list in a very rough sketch sort of way, not thinking about it at all.

Structuring the results

When we were finished, we had compiled a list of about 125 ideas, some being duplicates of others on second thought, some being broad enough to be split again into two or more more specific ones. We went through them all and tried to categorize them logically into rough areas such as "replication", "schema related", "indexing" and so on. We did not try to come up with a list of categories or chapters first, but just made them up on the fly. I think, that worked out very well. The result - after some more minor tweaking - can be found in the final book now as the 9 chapters and the appendix.

The latter one was not really a part of the cookbook format's description, but after combing the list a few items remained that proved very resistant to the idea of being formulated as a "How to do xxx" recipe.

For a short moment we considered dropping these ideas as being "not right" for the cookbook, but quickly decided to propose the idea of an appendix titled "Good to Know" as sort of a collection of interesting and relevant ideas one might find useful despite them being not strictly step-by-step instructions.

During the course of this sorting phase we also spent some time thinking about what level of experience a reader would be expected to have for each of the proposed recipes. We categorized everything as "A" (novice) through "D" (professional) to get an idea if we were anywhere near a uniform distribution among these. After that first pass we saw we had come up with only a very few "A" level topics, roughly the same (higher) amount of "B"s and "C"s and also quite a few "D"s. We took one or two more passes, sometimes splitting up more complicated recipes into two more simple ones to get a few more novice level recipes. In the end we reached an approximate bell curve with a slight tendency toward the more complicated stuff. We decided against trying to cut on the more complicated topics, because we came to the conclusion that this was what we would expect from a book, were we to buy it ourselves.

The end result was compiled into an email and sent to Sarah shortly afterwards on January 26, 2009. At that point we considered it to be sort of a pick list for Packt to choose from. It did not contain estimated page counts at that time and we surely expected a reasonable number of items to be combed out by Packt.

Publisher’s Feedback

Just a day after I had sent the email to Sarah, we got this reply:

Hi Guys,
Thanks for the outline. I've had a look through and its looking good. You have a good mix of basic and advanced recipes which will keep all readers happy.

I don't really have any suggestions to make at this stage. I am happy with the outline and so have presented it to my colleagues who will give us their feedback and suggestions which will help us to finalise the outline.

If possible, could you provide me with the expected page counts of each chapter. This only has to be a rough estimate at this stage, but you will find it useful to know what to aim for when writing each chapter.

[...]

That same email had a contract proposal attached, but I will spare that for a later post, because that's a little story of its own.

Page Count Estimate

The page count issue turned out more difficult than we had imagined. The initial description of the project called for about 300 pages. That's one of the reasons why we assumed the publisher would cut down the number of recipes from our proposed outline quite a bit. Even though we had no per-recipe estimate at that time, we had sort of a "feeling" going in the direction of around 400+ pages, just from the number of topics.

Even before we had sent the estimate Packt came up with a proposed schedule. Unfortunately there was little information on what that was based on, so we assumed it was calculated based on 300 pages with one page per author per day. That would have meant almost no buffer and a writing phase of approximately 5 months.

We had sat down again, thinking about how long each recipe would probably be. This turned out to be quite difficult with no experience to draw from. So we first were quite generous with the estimates. When looking at the final page count we were at about 500 and decided that this was a little over the top, especially when every recipe was intended to provide a quick solution to a clearly defined problem. So we thought about every single one again, this time considering how much we would be willing to read ourselves were this someone else's book. In the end we came up with around 420 pages.

So if the number of pages were to come closer to the 400 we had in mind, that would have meant about 7 months of writing, still with basically no buffer.

I decided to write a sample piece to get a feeling for how long an actual recipe and a chapter introduction would be, and if we had the same level of detail in mind as Packt did. If you'd like to read it, just go ahead: It ended up almost unchanged as the introduction of Chapter 3 (Tools) and the recipe on Sharing MySQL GUI Tools' connections.

In the meantime we had received another email, informing us about the acceptance of the outline proposal we had sent. No word on any cuts or redactions. So along with that content sample we sent our concerns about the schedule and the roughly 100 page difference we expected.

This is what we got back:

[...]

All recipes have been accepted, but if you feel that you need to remove some of them, that would be fine. The outline was approved by the editorial team here, so none of us have extensive technical experience. Therefore, we rely on an author's technical experience when talking about the specific details of the content. We would ideally like to keep the page count as close to 300 pages as possible, as this is what all books in the cookbook series are based on.
Do you imagine that all the recipes in the book will be as detailed as your sample one? I expect that different levels of recipe may need different amounts of explanation, depending on the topic. If you feel that you may need to delete some less-important recipes from the outline, then that would be fine. As long as the page count didn't go over 350 pages, that would be ok.

[...]

The schedule is just based on the first drafts of the chapters. Once a first draft chapter has been written, we will send it to a technical reviewer and, once all first drafts have been written, you will start writing the final drafts based on mine and the reviewers' feedback. [...] On the whole, we estimate the first draft writing stage to take 6-7 months, whilst the final draft stage takes 2-3 months. [...]

However, we do allow 30 days 'breathing room' in our editorial timeline and the schedule will be fairly flexible if needs be.

[...]

At that time we were reluctant to accept the schedule, because we were not quite sure the "1 hour per day and page" rule of thumb would hold very well and we did not want to reserve every moment of free time for writing.

Some more emails flew back and forth.

On February 10, 2009 I wrote:

Sarah,

[...]

We definitely see your point regarding the general 300 to 350-page rule for the cookbooks [...]. Unfortunately we do not see a way to keep all recipes in the outline and stay below that mark. Leaving out details is a path we would be reluctant to follow, because we fear that would mean giving up the really important stuff.

[...] option might be to cut a lot of the easier recipes (probably most if not all marked A and several designated B) and concentrate on the more ambitious ones that are not covered elsewhere at a comparable level of detail or in similar contexts. To do this we would generally assume a slightly higher level of expertise regarding the readership.

[...]

On the very same day I got this reply:

Hi Daniel,

[...]

I would be reluctant to lose the easier recipes as it would cut out a great deal of the audience. I think it's important to include a wide range of recipes to cater for all levels of experience.

Therefore, if you agree, I would like to keep all of the recipes that you mentioned in the outline. I have spoken to the cookbook series editor and he agrees that all the recipes are invaluable and so would be happy to extend the page count if necessary. Obviously, this would mean extending the schedule too. [...]

I'm glad that you both have put so much thought into this and can see that you are committed to quality, which is exactly what I like to see! :-)

[...]

So in the end getting the outline and page count agreed on took more than four weeks, which we would not have believed up front. In retrospect I believe our initial estimates were always a little too high and conservative. Probably if we were to tackle another one of these, both the estimates would be more accurate and the whole process of agreeing on the outlines and schedules would take a lot less time to get done with.

What’s next?

The next part will probably concentrate on the matters of getting our contracts ready - a rather dry and boring experience, but an important one nonetheless.

On Writing a Book (Pt. 1)

2010-04-23T22:22:00.003+02:00

This is the first part of an ongoing series about my experiences while writing the MySQL Admin Cookbook for Packt Publishing. All parts can be found under the mysql-admin-cookbook label.

In a few past posts I wrote about different aspects of my book writing project already. However those posts were not very contiguous, and I was asked a few times what my general experiences were and what caveats to look out for when taking on such a project for the first time. It might be of interest to some if I write down my experiences in a little more structured form.

This will be a series of several posts, because for one I do not think it would be feasible to have a single very long one, and also because this allows me to span writing it over a longer time. I will try to keep everything roughly chronological, but will deviate when I think it makes things clearer or better to understand.

Please be aware that I do not have extensive experience with multiple publishers, their processes and so on. This is solely about my personal experiences while writing a single book, with a single publisher, over a time of roughly 15 months. So please do not take anything for granted – should you go on that route and write something yourself, you might find yourself in completely different circumstances and witness the whole matter completely differently.

As for getting in contact with the publishing company, I already wrote down my experiences with that here: http://www.danielschneller.com/2009/12/51-weeks-since-my-book-writing.html

However I will go into a little more "historical detail" for starters.

Chapter 1: Getting contacted by the publisher

I do not know how common this is, but in my case the idea for writing a book did not come from me. Instead on December 24th of 2008 I received an email from Kshipra Singh, Author Relationship Manager with Packt Publishing titled "Author MySQL book-Packt Publishing".

Not knowing that name, never having heard of the company and generally being in the progress of getting my GMail inbox cleaned up I almost wanted to delete this mail unread, assuming it was some sort of junk. However, as the subject seemed to have a least some connection to my interests and because I really did not have to do much (Christmas Eve, empty office, fresh coffee, no bugs to fix, …) I opened it anyway.

This is what I read:

Hi Daniel,

I am writing to you for Packt Publishing, the publishers of computer related books.

We are planning to publish a book on MySQL, titled, The MySQL Admin Cookbook. While trying to find some potential authors for this site I came across your blog and saw your interest in MySQL. I also saw your blog entries on various facets of MySQL. These give me an impression that you could be a potential author for this book. The book is planned to be a cookbook with an admin recipe which will help DBAs to successfully maintain and administrate a MySQL database. DBAs of all abilities will benefit from this book as they learn tasks from creating tables to managing views, from improving performance to securing the database, from using monitoring tools to using storage engines. The reader is expected to have existing knowledge of MySQL and will already have their MySQL installed and ready to go.

To give you an idea about the way things work at Packt:

The editorial team at Packt works with the authors through out the project.
We pay a royalty of 15% and an advance against the same
The marketing team at Packt ensures that the book is well promoted.
We donate a percentage of sales coming from the book to the Open Source project on which it is based and have donated more than 100,000 dollars to various projects since inception in 2004.

Does it sound interesting to author such a book?

I had to read this twice to understand that this was not some random spam, but an actual offer, based on what I had written on the blog. I have to admit, I was rather surprised and also a little flattered, being not really sure about what to do next.

Then it occurred to me that maybe someone had accidentally mistaken me for a full-time DBA. So I wrote back that I am actually a software developer with only a strong side-interest in database matters and asked whether they still wanted me to write something. In fact at that time I was under the impression that the cookbook format was some sort of a compilation of independent articles, contributed by a series of individual authors and that I was offered a chance to write a recipe or two.

Very quickly however it turned out what they had in mind would be a one-, maybe two-author book.

Over the course of the next days between Christmas and the end of year several mails went back and forth, mostly questions on my part about what this whole book writing business was all about, how much time would be needed and so on. I also forwarded the whole conversation so far to Udo Schwedt, a colleague and good friend of mine with his fair share of MySQL knowledge under his belt, too. Turned out that he was very much interested in the prospect of co-authoring, however we were both quite skeptical about the "about 1 hour per day for the next 6-8 months" estimate that was mentioned in one of the early mails from Packt. In the end we decided to go forward together and get some more details - at this point everything was still in its very early stages with no obligations whatsoever.

Back then the Packt website looked a little different and the pages on authoring we looked at have been changed quite a bit. Here is a link to the current version: http://authors.packtpub.com/content/writing-process

I have to say that this current page contains a lot more details than the old one – at least as far as memory serves... You should read it to get a a quick overview of the whole process. But beware: Turns out there is still a lot more to it than one might expect from this rather streamlined description :-)

On January 5th we received a mail from Sarah, the acquisition editor, which included more information in the form of attached Word and PDF files: Author guidelines, a description of the Cookbook series and a description of what the "MySQL Admin Cookbook" (the title being already set) was planned to be like:

Page Count: 300

Executive Summary

A cookbook focusing specifically on administrative tasks with MySQL.

A concise collection of admin recipes, which will have no direct competition from the lengthy reference guides currently in the MySQL market.

MySQL is, and will continue to be the world's most popular open source database.

Description

As I'm sure everyone is aware, MySQL is a relational database management system. Administrators of MySQL will be tasked with things such as maintaining the database, tuning the server, managing users etc etc.
This cookbook will have all the MySQL recipes an administrator could dream of, spanning from creating tables to managing views, from improving performance to securing the database, from using monitoring tools to using storage engines. DBAs of all levels will be catered for with recipes of varying difficulty, allowing the reader to administer MySQL to their hearts' content.

Why a Cookbook?

There is a wealth of MySQL books on the market, but the majority of them are lengthy reference guides which cover anything and everything to do with MySQL. Our closest competitor - MySQL's Administrator's Bible - spends half of its time introducing the reader to MySQL. Our readers will already be up and running and will just want to know how to get things done as quickly and painlessly as possible. A cookbook is the best approach for this, as they can pick and choose which recipes are most applicable to them and won't have to faff around scouring the pages of a very, very, very long (and boring) reference guide.

Target Audience

DBAs who are tasked with maintaining the administration of MySQL.
Style/Approach

A cookbook, with many an admin recipe.

Prerequisites

The reader will have existing knowledge of MySQL and will already have their MySQL installed and ready to go. DBAs of varying abilities should benefit from this book.
[...]
So how will our book be different?

Quite simply, ours will not be as long, will not try to cover every single aspect of MySQL and will focus specifically on admin issues, whilst assuming that the reader has already installed MySQL and is raring to go and do something practical with it. The ranking of each of these books show that MySQL books sell incredibly well, despite there being so many comprehensive books on the subject.

Armed with this knowledge we decided to go on and started working out a draft of the book's outline – everything still without any bindings or obligations.

As for the outline: I think this will be the next issue in this series of posts...

Amazon + DHL delivery speed - just great!

2010-04-09T17:11:00.000+02:00

Instead of all the bashing that's usually to be found in blogs, I think one should give credit where credit is due.

Yesterday in the late afternoon I ordered some hardware from amazon.de with standard delivery options.

This is what my email inbox looked like today:

The oldest mail (the last one in the screen shot) is Amazon's confirmation of my order. Unfortunately this cannot be seen directly, but the mail arrived - seconds after I placed the order - at 17:50h (5:50pm).

The next mail - today at 09:53 (am) - informed me about the package leaving Amazon.

Only about 5 hours later I was informed that it had been delivered at my chosen destination!

With this kind of speed, I do not think I will be buying an Amazon Prime membership anytime soon, otherwise they would probably deliver the stuff, even before I ordered it ;-)

iTunes 9.1 to check for iPod Software Update on Apr 3rd

2010-03-31T00:10:00.002+02:00

When I just heard about iTunes having been updated to 9.1 I went ahead and installed it via Software Update on my iMac. When connecting my iPod Touch I was immediately curious about this:

Is it just me, or do I see a very special date for the next update check of the iPod's software version? Usually this is set to be something like every two weeks. I cannot be sure if this is the same date as before the update, but I think this might be a hint of the iPhones and iPod Touches being updated along with the iPad release as well.

Even when hitting the "Check for Update" button, the date does not change...

New blog template – and the problems entailed

2010-03-21T19:26:00.001+01:00

Last week I was happy to see that Blogger.com (where this blog is hosted) introduced a new template layout editor. While marked “beta” – or “Blogger in Draft” as they call it – I had thought about updating the layout of my site for quite some time, but never managed to sit down and do it. Now with this new editor, there seemed to be no reason not to go for a redesign.

You can see the result – I did not go crazy with the layout, but I think it is a nice, refreshed variation of the previous style. I replaced the header image and went for a slightly different color scheme. Temporarily I had picked a more colorful background image, but after some feedback decided against keeping it; it was just too noisy.

In general, I have to say the new editor is very nicely done and offers a very quick way to come to a reasonable design. However, there are some problems, too. Just to be clear: Even though I did not see any big-lettered warning signs in the new editor itself, by using the “Blogger in Draft” mode, I agreed to use pre-release software. So the following is clearly at least in part my fault – don’t get me wrong. However I thought I’d write it down for others to read and decide whether they would be fine trying it out, too.

Problem 1: Custom CSS got lost

Quite regularly I post source code or script snippets. In the old template I had included google-code-prettify to make those samples look nice and not take up too much space, but be limited to maximum box size and use scrollbars when necessary. To achieve this, a reference to a JavaScript and a CSS file needed to be included in the template, which I did. This got lost when I used the new graphical template editor and needed to be restored manually.

Problem 2: Google Analytics is broken

When I looked at my Google Analytics stats today, I saw a sharp drop in all numbers, starting last week when I switched to the new template. Apparently I am not alone with this and there seems to be some sort of problem with the tracking code embedded in the page.

According to the instructions, the JavaScript snippet for GA must be placed right before the closing body tag of any page. However when doing to in the new template, the final page (as seen in the browser) will contain a whole bunch of additional script code, before the body gets really closed. I am not sure what the problem with this is, but I suspect some sort of race condition or something like this, because my stats still show some people getting through.

Even though I do not really rely on these numbers for anything, this might be a more serious problem for other people. I hope Google will find a way to fix this soon.