Phil at Warrimoo

How many Linux installations are there?

2011-09-25T23:19:00.000+10:00

How hard can it be?

I'd like to know how many linux desktops/laptops, servers, phones, tablets and even routers there are.

What if each linux distro included a package that was part of a default desktop or server installation and this package submitted machine data to an online database?

A database already exists that oddly goes by this URL: linuxcounter.net

So, we have a database, we just need automated data to populate it.

We might need some rules:

1. Opt-in
2. Don't use serial numbers

Following these rules, a linux installation could send in these values:

1. CPU type
2. RAM
3. HDD size
4. linux kernel
5. Desktop, Server, Phone, Tablet, Router (based on installation)
6. Install date (of package)
7. Distro identifier

8. Number of user accounts
9. Public or private IP address (not actual IP address, just type)

A key could be generated using data 1 to 7 - things that are unlikely to change too often.

This data could be sent in monthly to either update a database entry, or to create a new one.

Old entries that don't get updated for 1 month get marked as stale.
Stale entries get deleted after 6 months.

A machine that undergoes a significant hardware or software change will automatically get a new entry and it's old entry will go stale and eventually be deleted.

At any time we will have a lower bound on active linux installations - I'd like that.

Approximate Distance between Two Points

2011-08-25T02:10:00.000+10:00

Is there an approximate way to calculate distance between two points?

Normally I would use

D = sqrt( dx^2 + dy^2 ).

But I didn't want to use sqrt().

Note: I have assumed that dx and dy are always positive by using absolute values.

Consider a case where dx >= dy:

After normalizing by dividing dx and dy by dx,

D' = D / dx = sqrt( 1 + (dy/dx)^2 )

Plotting this from 0 to 1 shows a flat-ish curve. It could be approximated by a straight line or, as I have done, by a quadratic.

D' = Ax^2 + B

Plugging in two extreme points (0,1) and (1,sqrt(2))

When x=0, D'=1 therefore B=1

And when x=1, D'=sqrt(2) therefore A=sqrt(2) - 1 (about 0.4142)

So, for dx>=dy (and dx!=0)

D = 0.4142 * dy^2 / dx + dx

And for dx<dy (and dy!=0),

D = 0.4142 * dx^2 / dy + dy (dx and dy are swapped)

Average error seems to be less than 1% - good enough for what I want and simple.

By using different 'tuning' points, better accuracy can be achieved

Another approach (using a linear approximation I suspect) can be found here.

PowerTop - Howto Enable Device Power Management by pressing the P key

2011-06-18T11:30:00.001+10:00

Have you ever ran powertop? If not, have a look at these links.

http://www.linuxpowertop.org/

http://www.lesswatts.org/index.php

When you run powertop as root (eg. sudo powertop), it might recommend something like this:

Enable Device Power Management by pressing the P key

So how do you do this when powertop is not running?

This forum made a suggestion that I have attempted to simplify.

find /sys/devices/pci* -path "*power/control" -exec bash -c "echo auto > '{}'" \;

Or you can use this slightly longer one that echo's values before and after it changes.

find /sys/devices/pci* -path "*power/control" -exec \
  bash -c "echo -n '{}' = && cat '{}' && echo auto > '{}' && echo -n '{}' = && cat '{}'" \;

Both have to be run as root or using sudo.

eg.

sudo find /sys/devices/pci* -path "*power/control" -exec bash -c "echo auto > '{}'" \;

Here is my bash script that I run with an extra powertop recommendation that may

not be applicable in your case.

#!/bin/bash

find /sys/devices/pci* -path "*power/control" -exec \
  bash -c "echo -n '{}' = && cat '{}' && echo auto > '{}' && echo -n '{}' = && cat '{}'" \;

F="/sys/module/snd_hda_intel/parameters/power_save"
echo -n $F = && cat $F
echo 1 > $F
echo -n $F = && cat $F

An idea: a DockBook

2011-05-31T00:29:00.002+10:00

A DockBook looks like a laptop or netbook or a portable DVD player or a TV - who knows.

Update 7/9/2011: It has sort-of been done.

It is probably a dumb device: a screen, a keyboard, a battery, USB ports, SSD, ... whatever you like.

On its own, it may do nothing (but it could have some functionality, but this is not the main function).

Where you might expect a trackpad to go, it has a square-ish hole - call it a 'docket'. In this docket you place a sabot that is specific to your phone. This sabot allows your phone to be held neatly and firmly in your DockBook.

A separate sabot could be provided for a second phone battery - again, an open interface.

A DockBook is just a collection of peripherals that allow you to do more with your phone.

Your phone provides CPUs, GPUs, touchscreen, broadband interfaces, WiFi, GPS, accelerometers etc. And your DockBook provides, perhaps, a full-size keyboard, large screen, camera, speakers, mics, power and other interfaces not found on your phone.

A DockBook could simply be a car-dock or an alarm clock. Perhaps a tablet too. The docket might have different forms for different Docks: a flat one for a netbook form or a vertical one for an alarm clock. The sabot might wrap around all of your phone or just its base.

Your phone is a powerful computer and phones will get more powerful. But for some tasks you need a keyboard, mouse, bigger screen, or a holder to keep it in a particular position.

A DockBook sabo provides a charging interface to your particular phone and Bluetooth provides communications to peripherals. Perhaps inductive charging?

While you are working you might get a call - just un-dock your phone and answer the call. Or you could answer it while docked and use a DockBook's microphone and speaker - hands free.

Imagine apps on your phone displaying specific buttons that can be touched making your docked phone a smart, reconfigurable touchpad - a menu system perhaps, or slider control. Perhaps a scrolling list of things to select?

Another DockBook could be a Digital SLR camera: your phone provides GPS, and internet connectivity and your camera dock provides lenses, flash and focus system.

What about a mouse dock? your phone becomes a touchpad for your PC or perhaps your phone's accelerometer can be used just like a mouse? While you are at your PC your phone is charging and enabling applications to work with your phone in a new way.

Imagine a docket that configures your car to your particular preference: temperature, seating position, mirrors and music.

It all needs to be open so companies and individuals can make a Dock for a market or niche use. It should work with any phone - not just an iPhone or an Android phone.

Some other docks: a games console like a wii or a PSP; a printer; braille terminal; a tablet; lego mindstorm robot; synthesizer; printing calculator; desk phone; chess board; ePaper book reader; projector dock; digital radio; car/plane/train seat dock;

Howto use nfs to access files on a Drobo-FS

2011-04-09T20:24:00.000+10:00

This should work for Linux clients and OS-X.

Notes

1. I am going to assume that, if you want to use nfs on your Drobo-FS then you know what you are doing and I can be brief.

2. drobo-fs is the dns name of your Drobo-FS or use it's IP address. In OS-X, I use drobo-fs.local

Steps

1. Install Unfsd on your Drobo-FS

See here

http://www.drobo.com/droboapps/apps-for-drobofs.php

Which links to this

http://support.datarobotics.com/ci/fattach/get/25295/1286306491/redirect/1/session/L2F2LzEvc2lkL0JYM3RrMXJr

2. If Linux, make a mount point

sudo mkdir /mnt/drobo

3. If Linux, add this to your /etc/fstab file

drobo-fs:/mnt/DroboFS/Shares/Public /mnt/drobo nfs rw,soft,proto=tcp 0 0

4. Mount the nfs directory on your client

If Linux

mount /mnt/drobo

If OS-X

mount -w -t nfs drobo-fs.local:/mnt/DroboFS/Shares/Public /mnt/drobo

Thanks to http://obasandbox.wordpress.com/tag/drobofs/

or in Finder

Connect to server nfs://drobo-fs.local:/mnt/DroboFS/Shares/Public

Test

In Linux, browse to /mnt/drobo and you should see the files in your Public share.

In OS-X, open the drobo-fs.local share.

Howto Get Long Battery Life on a Nexus One

2011-03-08T00:42:00.002+11:00

In case this is copied, go to the original site which should be http://philatwarrimoo.blogspot.com/2011/03/howto-get-long-battery-life-on-nexus.html

I would use my phone a lot (music, hotspot, not phone calls) and only get 15 - 18 hours from a fully charged battery.

I had read that some people got between 1 day and 1.5 days.

So, what is the upper limit? If you leave you phone in one spot and hardly touch it, how long will it run for?

The specs say 250/290 hours for 3G/2G.

To get this I assume you would need great signal levels, and virtually nothing running on the phone - in fact the phone would be the factory default.

But what could I get? I don't get great coverage at my place, especially inside where the signal is shielded by metal fly-screens, foil insulation, steel roof and two layers of brick - it is almost a Faraday cage.

I defaulted the phone. I didn't clear my SD card because I didn't want to.

I did make a few changes: I kept some applications that I do use often, and I disabled Picasa sync because I am in the middle of uploading lots of photos and that would distort the tests.

The big power consumers are believed by some to be the display, radio, GPS, WiFi, Bluetooth, and sync.

I have found that the main ones are the display and the radio (2G/3G).

In Australia the carriers all have GSM as well as either UMTS or HSDPA. My carrier only seems to have GSM and UMTS.

I decided to test what radio setting uses less energy. My options are GSM Auto (PRL), WCDMA Preferred, WCDMA Only and WiFi (for when I am at home). I don't want GSM only so I am not going to test that. I won't test Automatic either because I have no idea what it actually does.

GSM Auto (PRL) I believe prefers GSM rather than WCDMA, but it will switch to WCDMA if the signal is strong enough.

WCDMA Preferred is the opposite.

WCDMA Only is what it is.

When WiFi is on, I believe the 2G/3G radio switches off, so only the WiFi radio is on.

GPS will be left on (as per the default), as will sync, the default live wallpaper, widgets etc.

As I said earlier, Picasa sync will be off.

The Applications installed are:

4trak (since uninstalled)
adobe flash (since uninstalled)
anycutastrogoogle earth (since uninstalled)
espeakfunction express (since uninstalled)
gmailgoggleshotspot widgetlistenmapsmarketplugin launcher (i added this last night - since uninstalled)
realcalcringdroid (since uninstalled)
spareparts (since uninstalled)
terminal emulator (since uninstalled)wifi analyser (since uninstalled)

And this is what generally runs:

settingsgoogle servicesmapsgmail (regularly on and off)android keyboardandroid live wallpaper

The following Widgets are part of the default setup:

google searchpower controlyoutubeweather and newsmarket

I checked once and found the 107 MB used and 244 MB free.

I generally did not reboot between tests - perhaps I should have.

I ran the tests for different periods and only long enough to get a feeling for how long it would run for.

The phone was always in the same place and was only moved to check how it was going. Usually I checked a few times for just a few seconds.

Test Results

WiFi On: 4.25 hours, 82% remaining. I'd expect it to go for 24 hours.

From the graph you can see that something changed for about 30 minutes that consumed a lot of energy (about 10%). This has resulted in the poor expected battery life.

The graph also shows the screen being turned on a few time and once for about 15 minutes - I don't recall doing this!

When the screen turns off or the phone sleeps, WiFi still runs for up to 15 minutes.

WiFi wakes up when the radio signal drops or changes.

I should re-do this test.

GSM Only (PRL): 9.2 hours, 90% remaining. I'd expect it to go for 92 hours.

Battery drain was fairly constant and seemed to increase when coverage was poorer (at the beginning and end of the test period).

The screen was also on for 20 minutes at the start of the test - I am sure that this did not happen.

WCDMA Only: 3.2 hours, 91% remaining. I'd expect it to go for 36 hours.

Again something happened for about 40 minutes causing a drain of about 5% of the battery.

WCDMA Preferred: 2.25 hours, 92% remaining. I'd expect it to go for 28 hours.

The phone seems to wake up when there is poor or changing signal.

The observation to make is that there is a big difference in the expected battery life depending on which radio setting you choose. For me, GSM Only (PRL) would provide the longest battery life at home, and probably where the 3G signal is poor.

WiFi was the poorest, but at home it will generally be on the dock and charging.

For those that have good 3G coverage, the WCDMA Only option will probably serve you well.

Custom Test

I then decided to turn off the widgets, use a black live wallpaper that I made a while ago, select GSM Auto (PRL) and turn WiFi on.

WiFi On, GSM Auto (PRL), most widgets off, black live wallpaper: 4.3 hours, 92% remaining. I'd expect it to go for 54 hours.

Only the Power Control widget was running.

Wifi seems to stay on for about 15 minutes then sleeps. It wakes up when you turn the screen on.

The WiFi Sleep Policy is Never when plugged in so this means that it will sleep if it is not plugged in.

WiFi Power Hungry Bug

Some have reported that WiFi on anything but the default sleep policy setting causes Android OS to run at high levels. Normally it sits at 2%-ish, but when the fault occurs it runs at 40%, 50% and even higher - usually higher than the display.

Some get relief using the WiFi sleep policy: 'Never', and others find this to be the cause. By this I suspect they mean that if the setting is changed from the default (to 'Never', for example) something bad happens when you switch back to the default setting. This is my experience.

In one of my tests I used the WiFi sleep policy: 'When screen turns off' and Android OS sky-rocketed. I recommend you keep the default (Never when plugged in) until this is fixed.

Summary

1. When WiFi turns on, it stays on for 15 minutes.

2. For me, GSM Auto (PRL) is better than WCDMA Preferred or WCDMA Only.

3, The graph seems to show the screen on when it isn't.

4. The phone seems to wake up on signal strength change, or this could be base station changes.

5. If WiFi is asleep and the radio looses signal, WiFi will wake up.

6. The default Nexus One setup with all the widgets and live wallpaper seems gentle on the battery.

Update

I have used my phone with minimal apps installed, my special black wallpaper and very few gadgets. I have been getting at least 12 hours of use before the battery reaches 50%, so this is about 1 day.

I'll try to keep some records from now on to quantify this.

Android Easter Egg

2011-03-03T21:44:00.002+11:00

In case this is copied, go to the original site which should be http://philatwarrimoo.blogspot.com/2011/03/android-easter-egg.html

Goto Settings - About Phone
Scroll down to Android version
Tap quickly several times on 'Android version' and some art will appear.
'Zombie Art by Jack Larson'

Hey Google! I want a Google Ark!

2011-03-02T00:01:00.001+11:00

In case this is copied, go to the original site which should be http://philatwarrimoo.blogspot.com/2011/03/hey-google-i-want-google-ark.html

An appliance running Android or Chrome OS that I buy and plug-in to my network that makes a copy of any data I want to duplicate from the cloud:

email, contact, docs, pictures, movies, bookmarks, blogs, waves... you get the idea.

It might have 2 solid-state hard drives so I can mirror them and, at some later point in time, I can replace one with a bigger one and then the other to grow my local cache.

It could also operate as a web/DNS cache for all computers, phones and Chrome OS googleBooks - why get it from the web if I got it yesterday or last year?

It could come in various sizes for homes, hoarders, small and large businesses.

It supports apps too so that anyone can develop for it, especially other cloud backup apps. Perhaps a Cannon camera photo management app, or a Digital Ink notepad app. What about a Wordpress backup app or a local bazaar repository. Maybe a Windows system backup app or a Time Machine folder.

It would be very low power and quickly go to sleep when not in use.

It might have a place for hardware apps (via USB perhaps) so that I could plug-in a LTE Femto cell, web printer, or a weather station.

If you lost my email, you could recover it.

If I dropped my laptop I could restore it from the Ark.

Howto Sting Android Nexus One Droid Gingerbread

2011-02-28T19:48:00.004+11:00

In case this is copied, go to the original site which should be http://philatwarrimoo.blogspot.com/2011/02/howto-sting-android-nexus-one-droid.html

Recently a blog of mine was copied and added to another site. I assume it was by a robot so I want to see if this gets copied as well.

I have added the important bait words such as:

1. Howto
2. Android
3. Nexus One
4. Droid
5. Gingerbread

But this time I have added a link on the top to this blog to see how many people get here from there.

An image to begin with...

So far, the tally is obviously zero since I am just writing it and it has not been saved or published - really I am just filling the space to make it look big enough for the bot to grab - assuming that it has some size range that it is interested in.

Another unrelated image:

An a link to the site - they might appreciate it.

http://www.premium-joomlathemes.com/jd-droid-drupal-template-drupal-hotel-theme-for-lodging-accommodation-with-t3-framework.html

Now for some real nonsense:

This nonsense comes from a pseudo text generator called Lorem Ipsum.

You can find more details here: http://www.lipsum.com/

Please note: I have copied text from another site and provided a link to it with thanks. Some could learn from this example - not that it is a very good example.

A sample nonsense would help:

The standard Lorem Ipsum passage, used since the 1500s

"Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum."
Section 1.10.32 of "de Finibus Bonorum et Malorum", written by Cicero in 45 BC

"Sed ut perspiciatis unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, totam rem aperiam, eaque ipsa quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt explicabo. Nemo enim ipsam voluptatem quia voluptas sit aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos qui ratione voluptatem sequi nesciunt. Neque porro quisquam est, qui dolorem ipsum quia dolor sit amet, consectetur, adipisci velit, sed quia non numquam eius modi tempora incidunt ut labore et dolore magnam aliquam quaerat voluptatem. Ut enim ad minima veniam, quis nostrum exercitationem ullam corporis suscipit laboriosam, nisi ut aliquid ex ea commodi consequatur? Quis autem vel eum iure reprehenderit qui in ea voluptate velit esse quam nihil molestiae consequatur, vel illum qui dolorem eum fugiat quo voluptas nulla pariatur?"

Another Virtualised OS-X Example

2011-02-27T21:22:00.003+11:00

In case this is copied, go to the original site which should be http://philatwarrimoo.blogspot.com/2011/02/another-virtualised-os-x-example.html

To see if my first virtualised OS-X was a fluke, I'll have another go. This will just list the steps I take and there won't be much discussion.

>VBoxManage internalcommands listpartitions -rawdisk /dev/sda
Number Type StartCHS EndCHS Size (MiB) Start (Sect)
1 0xee 1023/254/63 1023/254/63 200 1
2 0xaf 1023/254/63 1023/254/63 119102 409640
4 0x83 1023/254/63 1023/254/63 11444 244330976
3 0xaf 1023/254/63 1023/254/63 23167 343011768

The OS-X partition I want is Number 2: type is 0xaf and the size is about right.

I have found my first fault: I created the vmdk file in VB 3.2.8. Under VB 4.0 the '-register' option is no longer used. VB 4.0 has a '-relative' option for Linux hosts that would make sense to use, but 'relative' and 'register' are too different for it to be a typo.

My source used '-register' so VB 4.0 must be different.

>VBoxManage internalcommands createrawvmdk -filename /home/phil/.VirtualBox/HardDisks/osxpart1.vmdk -rawdisk /dev/sda -partitions 2RAW host disk access VMDK file /home/phil/.VirtualBox/HardDisks/osxpart1.vmdk created successfully.

I should point out that the name of the vmdk file that I used is from the viewpoint of OS-X, not the actual partition number - use your own naming scheme.

The '-register' option adds the new vmdk file to the list of hard disks available - looks like I'll have to register manually somehow.

I added an entry in the VirtualBox.xml file for the new hard disk like this:

I got the UUID from the osxpart1.vmdk.xml file - I used the uuid.image number for the uuid attribute.

That did the trick.

I created a new VM, called it maxosxp1 and VirtualBox automatically selected Mac OS X - nice.

I told it to use the new osxpart1.vmdk file that I just created and left everything else as default for the time being.

I tried to run it, and it didn't work. But I have found that you need to do this to populate the XML or VBOX file to make the next steps easier.

Note: in VB 4.0, the file is now in VirtualBox VMs/(machine name)/(machine name).vbox. It is still an XML file so you can edit it with a text editor - but as before only when VirtualBox is not running.

To make it boot I need to add in the special Apple SMC key.

I added these to the ExtraData section:

<ExtraDataItem name="VBoxInternal2/EfiBootArgs" value=" "/><ExtraDataItem name="VBoxInternal2/EfiGopMode" value="4"/><ExtraDataItem name="VBoxInternal2/SmcDeviceKey" value="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc"/>

I then started the VM.

It was slow to boot, but it got there. As it was loading I could see VMWare and VirtualBox kernel extensions loading and I thought that it will not like that - running a VM manager in a VM.

It wanted me to identify my keyboard and that was it.

I logged in, hit (shift)-(right command)-F to enter fullscreen mode.

Apart from being slow and running warm, it looked like I was running OS-X on a PM rather than a VM.

To solves the high CPU load problem, just delete the right kernel extension.

sudo rm -fr /System/Library/Extensions/AppleIntelCPUPowerManagement.kext

I haven't done that yet as I might want to boot into this partition sometime soon to tidy it up.

So, no fluke.

Howto Vodafone UK Nexus One to Gingerbread

2011-02-26T20:18:00.004+11:00

In case this is copied, go to the original site which should be http://philatwarrimoo.blogspot.com/2011/02/howto-vodafone-uk-nexus-one-to.html

A friend made this great guide.

http://ossadmin.wordpress.com/2011/02/26/goodbye-vodafone-uk-nexus-one-rom/

I have to same phone with the same Vodafone UK ROM and I kept my log of what I did in case it might help others.

All of the screenshots are from the above blog.

Notes:

All the following stages are similar. The first is a little different, but the others are virtually the same - the differences being the URL to get the file, and the name of the file to copy to your phone's SD card.

Sometimes the phone will automatically reboot after the installation of the update.zip file - other times you will need to manually reboot it by selecting the right menu item. All of this should be obvious.

Just follow these steps in order. If you get lost or miss a step, just start again (I did that a few times).

MD5

I do not know the original MD5 sums of the files listed below, but so you can verify that you get the same file as I did, here are my MD5 sums:

2854b06b22b9.signed-passion-FRG83G-from-FRG83D.2854b06b.zip
23a17e5f70d794858e0fe7aba308c3d7

766a101ae02c.signed-passion-ota-60505.766a101a.zip
1b3a95a92b0fd319f0e730b80e1e03ac

81304b2de707.signed-passion-GRI40-from-FRG83G.81304b2d.zip

706a4631271b6ee2c819ab548394b71e

ecb21b752065.signed-passion-FRG83D-from-FRG83.ecb21b75.zip

0fb26549e9de782e86e665a68693c50a

Passion_Google_WWE_2.16.1700.1_FRG33_MFG_Shipment_ROM.zip

3bcdbb83a4b951f813ca88771d69894d

passimg.zip (the file extracted from the file above)

d90635c2d269c6a26f77ae1a681e5424

Stage 1: Revert to Android 2.2

1. Get http://shipped-roms.com/download.php?category=android&model=Passion%20(Nexus%20One)&file=Passion_Google_WWE_2.16.1700.1_FRG33_MFG_Shipment_ROM.zip

2. Extract PASSIMG_Passion_Google_WWE_2.16.1700.1_FRG33_release_signed.zip

3. Rename to passimg.zip

4. Connect a USB cable to your computer and turn on USB storage

5. From your computer, copy passimp.zip to the root directory of your SD card

6. Eject USB device phone from your computer

7. Turn off USB storage

8. Power off the phone using the Power button (POWER) - like you normally would do

9. Hold down the Volume Down button (VOL DOWN) and keep it held down

10. Press and release (POWER)

11. When you see the white screen you can release (VOL DOWN)

A blue progress bar will be shown as it loads passimg.zip

12. After a while it will ask: 'Do you want to start update?' Press the Volume Up button (VOL UP) to say yes

The update will take a while. Progress is shown with various coloured progress bars

13. When done it will ask: 'Do you want to reboot device?' Press (VOL UP) to say yes

The phone will reboot

14. Disconnect the USB cable

The phone will say 'Welcome to Nexus One'. You don't want to do anything here so for me I skipped the setup like this:

15. Touch the screen

16. Tap the Skip button

17. Tap the Skip button

18. Tap Next button

Now, you can check to see if it all worked:

19. Find and launch the Settings app

20. Goto About phone

You should see the following:

Android Version: 2.2 Baseband Version: 32.41.00.32U_5.08.00.04 Kernel Version: 2.6.32.9-27237-gbe746fb android-build@apa26 #1 Build Number: FRG33

Stage 2: Upgrade to 2.2.1

1. Reconnect USB cable

2. Turn on USB storage

3. From your computer, delete passimg.zip from the SD card

4. Get http://android.clients.google.com/packages/ota/passion/766a101ae02c.signed-passion-ota-60505.766a101a.zip

5. Copy 766a101ae02c.signed-passion-ota-60505.766a101a.zip to the SD card

6. Rename it to update.zip

7. Eject the USB device phone from your computer

8. Turn off USB storage

9. Power off the phone

10. Hold down the Volume Down button (VOL DOWN) and keep it held down

11. Press and release (POWER)

12. When you see the white screen you can release (VOL DOWN)

It will quickly do something and then present 4 options

13. Press (VOL DOWN) to select the RECOVERY option

14. Press (POWER) to select RECOVERY

The phone will restart into a black ! screen

15. Press and hold (POWER)

16. Tap (VOL UP)

17. When you get a black screen you can release (POWER)

18. Use trackball to highlight 'apply sdcard:update.zip'

19. Click the trackball to select this

The installation will start. Progress is shown by an orange progress bar

The phone will eventually reboot

20. Disconnect usb cable

21. Goto About phone

You should see the following:

Android Version: 2.2.1Baseband Version: 32.36.00.28U_4.06.00.12_7 Kernel Version: 2.6.32.9-27240-gbca5320 android-build@apa26 #1 Build Number: FRG83

Stage 3: FRG83 to FRG83D

1. Reconnect USB cable

2. Turn on USB storage

3. From your computer, delete update.zip from the SD card

4. Get http://android.clients.google.com/packages/ota/passion/ecb21b752065.signed-passion-FRG83D-from-FRG83.ecb21b75.zip

5. Copy ecb21b752065.signed-passion-FRG83D-from-FRG83.ecb21b75.zip to the SD card

6. Rename to update.zip

7. Eject the USB device phone from your computer

8. Turn off USB storage

9. Power off phone

10. Hold down the Volume Down button (VOL DOWN) and keep it held down

11. Press and release (POWER)

12. When you see the white screen you can release (VOL DOWN)

It will quickly do something and then present 4 options

13. Press (VOL DOWN) to select the RECOVERY option

14. Press (POWER) to select RECOVERY

The phone will restart into a black ! screen

15. Press and hold (POWER)

16. Tap (VOL UP)

17. When you get a black screen you can release (POWER)

18. Use trackball to highlight 'apply sdcard:update.zip'

19. Click the trackball to select this

The installation will start - progress is shown by an orange progress bar

The phone will eventually reboot or you need to select REBOOT

20. Disconnect usb cable

21. Goto About phone

You should see the following:

Android Version: 2.2.1Baseband Version: 32.36.00.28U_4.06.00.12_7 Kernel Version: 2.6.32.9-27240-gbca5320 android-build@apa26 #1 Build Number: FRG83D

Stage 4: Upgrade to 2.2.2 and FRG83G

1. Reconnect USB cable

2. Turn on USB storage

3. From your computer, delete update.zip from the SD card

4. Get http://android.clients.google.com/packages/ota/passion/2854b06b22b9.signed-passion-FRG83G-from-FRG83D.2854b06b.zip

5. Copy 2854b06b22b9.signed-passion-FRG83G-from-FRG83D.2854b06b.zip to the SD card

6. Rename to update.zip

7. Eject the USB device phone from your computer

8. Turn off USB storage

9. Power off phone

10. Hold down the Volume Down button (VOL DOWN) and keep it held down

11. Press and release (POWER)

12. When you see the white screen you can release (VOL DOWN)

It will quickly do something and then present 4 options

13. Press (VOL DOWN) to select the RECOVERY option

14. Press (POWER) to select RECOVERY

The phone will restart into a black ! screen

15. Press and hold (POWER)

16. Tap (VOL UP)

17. When you get a black screen you can release (POWER)

18. Use trackball to highlight 'apply sdcard:update.zip'

19. Click the trackball to select this

The installation will start - progress is shown by an orange progress bar

The phone will eventually reboot or you need to select REBOOT

20. Disconnect usb cable

21. Goto About phone

You should see the following:

Android Version: 2.2.2 Baseband Version: 32.36.00.28U_4.06.00.12_7 Kernel Version: 2.6.32.9-27240-gbca5320 android-build@apa26 #1 Build Number: FRG83G

Stage 5: Gingerbread

1. Reconnect USB cable

2. Turn on USB storage

3. From your computer, delete update.zip from the SD card

4. Get http://android.clients.google.com/packages/ota/passion/81304b2de707.signed-passion-GRI40-from-FRG83G.81304b2d.zip

5. Copy 81304b2de707.signed-passion-GRI40-from-FRG83G.81304b2d.zip

6. Rename to update.zip

7. Eject the USB device phone from your computer

8. Turn off USB storage

9. Power off phone

10. Hold down the Volume Down button (VOL DOWN) and keep it held down

11. Press and release (POWER)

12. When you see the white screen you can release (VOL DOWN)

It will quickly do something and then present 4 options

13. Press (VOL DOWN) to select the RECOVERY option

14. Press (POWER) to select RECOVERY

The phone will restart into a black ! screen

15. Press and hold (POWER)

16. Tap (VOL UP)

17. When you get a black screen you can release (POWER)

18. Use trackball to highlight 'apply sdcard:update.zip'

19. Click the trackball to select this

The installation will start - progress is shown by an orange progress bar

The phone will eventually reboot or you need to select REBOOT

20. Disconnect usb cable

21. Goto About phone

You should see the following:

Android Version: 2.3.3 Baseband Version: 32.41.00.32U_5.08.00.04 Kernel Version: 2.6.35.7-59423-g08607d74 android-build@apa28 #1 Build Number: GRI40

Stage 6: Delete phone data (but not your SD card)

NB: I'm not sure this is required.

1. Goto Settings - Privacy - Factory data reset

If you want to keep your SD card data, DO NOT select 'Erase SD card'

Stage 7: Login and restore your apps

Once you enter your Google account details, your Google apps and settings should be restored. Your other apps will also be restored but y.our data will not be restored - use your backup

Any app not installed from the market will need to be manually restored.

I may have interrupted the restoration process by switching to a different WiFi Access Point - you probably should not do that.

Our Rain Water Tank Needs Treatment Again

2011-02-20T13:08:00.000+11:00

The last treatment to increase the pH was 29/1/2011 - about 50 days ago.

Our tanks are still full due to the substantial rain we have had.

The pH before treatment was 6.6 which, being below 7, is right for the Blue-Green stain to re-appear indicating that our copper pipes are dissolving again.

I treated it with 1L of lime and the pH rose to 10.6. This will drop as the lime spreads throughout the tank; as we consume the water; and as we collect more water.

In case it is useful, the water meter is on about 104kL.

Boot a Real Mac OS-X Partition under Linux and VirtualBox - What Works

2011-02-19T20:30:00.003+11:00

VirtualBox 4.0

Works (does something)

Nat networking
Safari (I have edited this post using Safari on the OS-X VM)
iTunes (starts - that is all I have done)
iPhoto (starts - that is all I have done)
About
System Preferences
Finder
Finder sees other SMB and AFS shares
Shutdown (mostly)
1 CPU runs flat-out - to 'fix' this I ran this command. When the VM is idle the host load is much reduced:

sudo rm -fr /System/Library/Extensions/AppleIntelCPUPowerManagement.kext

By removing /System/Library/Extension
Calculator
YouTube
Sound - I have tried IntelHD/ALSA
USB with VirtualBox extras
Disk Utility
Intel PRO/1000 MT Desktop (Bridged adapter, eth1)
Time Machine backup to Time Capsule if a network bridge is available

Does not work (or I can't get it working)

glxgears (X11 crashes repeatedly)
Shutdown (rarely)
V8 Benchmark runs too slow for Safari's liking

VirtualBox 3.2.8 OSE

Works (does something)

Nat networking
Safari
iTunes (starts - that is all I have done)
iPhoto (starts - that is all I have done)
About
System Preferences
Finder
Finder sees other SMB and AFS shares
Shutdown (sometimes)
1 CPU runs flat-out
Calculator
YouTube
Disk Utility

Does not work (or I can't get it working)

Bridge Networking
Audio
glxgears (X11 crashes repeatedly)
Finder can not connect to other SMB and AFS shares
Shutdown (sometimes)
V8 Benchmark runs too slow for Safari's liking

Boot a Real Mac OS-X Partition under Linux and VirtualBox

2011-02-19T17:19:00.001+11:00

This should be easy, but Apple make it hard.

I have a MacBook Pro. I have OS-X Leopard 10.5.2 installed as well as Ubuntu 10.10 linux.

I wanted to get my OS-X partition to run as a virtual machine.

After many false starts I got something to work.

First, the specs.

Hardware: MacBook Pro 4,1, early 2008, 2.4GHz, 200GB, 15"
http://en.wikipedia.org/wiki/MacBook_Pro
Host OS: Ubuntu 10.10
VM Manager: VirtualBox 3.2.8 OSE r64453
Guest OS: OS-X 10.5.2 Leopard (I don't bother to update it as I don't use it much and it have virtually nothing installed)

Overview

1. Get VirtualBox to list the physical partition in the Virtual Media Manager.
2. Setup the VM
3. Tweak it.
3. Run it.

1. Setup Disk Image

Please proceed at your own risk. Only do this if you have backed-up your disk partitions (with Time Machine for example, but don't forget about your linux partition) or your don't care about any data on your disk.

I used this information to create my VMM vmdk file.

http://ejs.seniejitrakai.net/2010/11/01/running-virtualbox-guests-from-physical-drive/

In my case, my hard drive is /dev/sda under linux.

phil@max:~$ VBoxManage internalcommands listpartitions -rawdisk /dev/sda

Oracle VM VirtualBox Command Line Management Interface Version 3.2.8_OSE(C) 2005-2010 Oracle CorporationAll rights reserved.
Number Type StartCHS EndCHS Size (MiB) Start (Sect)1 0xee 1023/254/63 1023/254/63 200 12 0xaf 1023/254/63 1023/254/63 119102 4096404 0x83 1023/254/63 1023/254/63 11444 2443309763 0xaf 1023/254/63 1023/254/63 23167 343011768

The OS-X partition in my case is number 3. It is about 23GB.

Then I made the vmdk file like this:

phil@max:~$ VBoxManage internalcommands createrawvmdk -filename /home/phil/.VirtualBox/HardDisks/example.vmdk -rawdisk /dev/sda -partitions 3 -register

Oracle VM VirtualBox Command Line Management Interface Version 3.2.8_OSE(C) 2005-2010 Oracle CorporationAll rights reserved.
RAW host disk access VMDK file /home/phil/.VirtualBox/HardDisks/example.vmdk created successfully.

This made this file - yours will be different, but it should have the same structure:

# Disk DescriptorFileversion=1CID=279c9811parentCID=ffffffffcreateType="partitionedDevice"
# Extent descriptionRW 1 FLAT "example-pt.vmdk" 0RW 409639 ZERO RW 243921336 ZERO RW 23437500 ZERO RW 75243292 ZERO RW 47448016 FLAT "/dev/sda" 343011768RW 262184 ZERO
# The disk Data Base #DDB
ddb.virtualHWVersion = "4"ddb.adapterType="ide"ddb.geometry.cylinders="16383"ddb.geometry.heads="16"ddb.geometry.sectors="63"ddb.uuid.image="99ca08b3-6d2e-4ed4-8790-2430cfe5a57a"ddb.uuid.parent="00000000-0000-0000-0000-000000000000"ddb.uuid.modification="00000000-0000-0000-0000-000000000000"ddb.uuid.parentmodification="00000000-0000-0000-0000-000000000000"

Now if you run Virtual Media Manager, you should now see a new hard drive entry for your physical partition.

That wasn't that hard, thanks to Sun/Innotek for making this easy.

2. Setup VM.

The key fields are these:

OS Type = Mac OS X
Use an existing hard disk - the one I created above: example.vmdk
The rest are the current defaults (which I will list below).
I also enabled 3D acceleration, but I can pick either.

Too easy, right?

3. Tweak VM

Apple OS-X checks to see if a special key is in a chip. VMs don't currently emulate this chip with the key, so you need to get the key off your machine and add it manually to the VirtualBox XML file that describes the VM.

The key is really a plea from Apple to not steal their OS. Here is their EULA for 10.5.

http://images.apple.com/legal/sla/docs/macosx105.pdf

It contains this clause:

A. Single Use. This License allows you to install, use and run one (1) copy of the Apple Software on a single Apple-labeled computer at a time. You agree not to install, use
or run the Apple Software on any non-Apple-labeled computer, or to enable others to do so. This License does not allow the Apple Software to exist on more than one
computer at a time, and you may not make the Apple Software available over a network where it could be used by multiple computers at the same time.

I think this complies: One licensed and installed copy of OS-X on an Apple-labeled computer. I can't run it twice because that would do bad things to the file system.

So, I'll continue...

In my case the XML file is /home/phil/.VirtualBox/Machines/example/example.xml

This is what it might look like:

<?xml version="1.0"?>
<VirtualBox xmlns="http://www.innotek.de/VirtualBox-settings" version="1.11-linux">
  <Machine uuid="{0db68b53-a452-40bd-bb83-ad8952411320}" name="example" OSType="MacOS" snapshotFolder="Snapshots" lastStateChange="2011-02-19T08:34:33Z">
   <MediaRegistry>
   <HardDisks/>
   <DVDImages/>
   <FloppyImages/>
   </MediaRegistry>
   <Hardware version="2">
   <CPU count="1" hotplug="false">
   <HardwareVirtEx enabled="true" exclusive="true"/>
   <HardwareVirtExNestedPaging enabled="true"/>
   <HardwareVirtExVPID enabled="true"/>
   <PAE enabled="true"/>
   <HardwareVirtExLargePages enabled="false"/>
   <HardwareVirtForce enabled="false"/>
   </CPU>
   <Memory RAMSize="1024" PageFusion="false"/>
   <Firmware type="EFI"/>
   <HID Pointing="USBTablet" Keyboard="USBKeyboard"/>
   <HPET enabled="true"/>
   <Chipset type="ICH9"/>

--- SNIP ---

</Machine>
</VirtualBox>

As you can see, the whole configuration of a VM is, more-or-less, understandable.

But wait! To do the next bit, you have to run the VM once. Until you do this scary thing, the XML file is missing bits that make the next part easier.

So, start the VM. When you see lots of text you can close the window and shutdown the VM.

This is all safe it seems.

My example VM XML file now looked like this - the highlighted bits are what we want to see:

<?xml version="1.0"?>
<VirtualBox xmlns="http://www.innotek.de/VirtualBox-settings" version="1.11-linux">
  <Machine uuid="{57d6a1c9-833f-4fb7-b765-f4c37793bc56}" name="example" OSType="MacOS" snapshotFolder="Snapshots" lastStateChange="2011-02-19T08:27:47Z">
   <MediaRegistry>
   <HardDisks/>
   <DVDImages/>
   <FloppyImages/>
   </MediaRegistry>
   <ExtraData>
   <ExtraDataItem name="GUI/LastCloseAction" value="powerOff"/>
   <ExtraDataItem name="GUI/LastGuestSizeHint" value="1024,768"/>
   <ExtraDataItem name="GUI/LastNormalWindowPosition" value="400,61,1024,815"/>
   <ExtraDataItem name="GUI/LastWindowPostion" value="37,53,1024,815"/>
   <ExtraDataItem name="GUI/MiniToolBarAlignment" value="bottom"/>
   <ExtraDataItem name="GUI/SaveMountedAtRuntime" value="yes"/>
   <ExtraDataItem name="GUI/ShowMiniToolBar" value="yes"/>
   </ExtraData>
--- SNIP ---
  </Machine>
</VirtualBox>

To tweak it, I followed these instructions:

http://www.takwing.idv.hk/tech/virtual/vb320_macvm/page3.html

Modify your VM's xml file by adding the following line to the section:

<ExtraDataItem name="VBoxInternal2/SmcDeviceKey" 
value="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc"/>

Mine then looked like this:

<?xml version="1.0"?>
<VirtualBox xmlns="http://www.innotek.de/VirtualBox-settings" version="1.11-linux">
  <Machine uuid="{57d6a1c9-833f-4fb7-b765-f4c37793bc56}" name="example" OSType="MacOS" snapshotFolder="Snapshots" lastStateChange="2011-02-19T08:27:47Z">
   <MediaRegistry>
   <HardDisks/>
   <DVDImages/>
   <FloppyImages/>
   </MediaRegistry>
   <ExtraData>
   <ExtraDataItem name="GUI/LastCloseAction" value="powerOff"/>
   <ExtraDataItem name="GUI/LastGuestSizeHint" value="1024,768"/>
   <ExtraDataItem name="GUI/LastNormalWindowPosition" value="400,61,1024,815"/>
   <ExtraDataItem name="GUI/LastWindowPostion" value="37,53,1024,815"/>
   <ExtraDataItem name="GUI/MiniToolBarAlignment" value="bottom"/>
   <ExtraDataItem name="GUI/SaveMountedAtRuntime" value="yes"/>
   <ExtraDataItem name="GUI/ShowMiniToolBar" value="yes"/>
   <ExtraDataItem name="VBoxInternal2/SmcDeviceKey" value="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc"/>
   </ExtraData>

--- SNIP ---

</Machine>
</VirtualBox>

4. Run it.

That's it.

Let me know if it worked for you, and if you have a different hardware, host, guest configuration.

If it didn't work, then I probably can't help - but you you want me to explain anything that I did in more detail then please ask.

BTW, if this kills you disk, damages your data, or generally make your life a misery then you will recall that I warned you, right?

Our (my) New Robot

2011-02-03T21:46:00.000+11:00

I decided that there were more important chores for the children (cleaning windows, sweeping the verandah, painting the roof, ...) so I needed a robot to vacuum the floors.

Blue-green stain in the bath is back

2011-01-29T09:54:00.000+11:00

Blue stain is back.

We have had a lot of rain recently and the tanks are full with about 30 kL each.

Tanks read an uncalibrated pH of 5.7 as measured at the kitchen tap.

By uncalibrated I mean that the pH meter has not been used for a while and I should calibrate it before use - but I didn't.

I treated one tank with 500ml lime and instantly it rose to 9.2.

Mac to Linux: Ubuntu is better than good enough.

2011-01-21T23:55:00.000+11:00

When I purchased my first Mac in 2005, my intention was to get some good hardware, avoid any Microsoft tax and run Linux.

The closest I came was Ubuntu 9.04 in 2009, but I was tied to iTunes for my Ipod Touch and a little stuck with iPhoto.

I tried again in 2010 but I could not get Ubuntu 10.04 working well enough.

More recently I purchased a Nexus One Android phone and gave away the iPod. To do this I had to divorce iTunes. This turned out to be easy. I hardly needed any apps or games and I never purchased any music. So all I had to do was copy music files from the iTunes directory into the Nexus One.

A good friend (who is really my beta tester for... well... everything) got Ubuntu 10.10 working nicely on his Mac and I quickly tagged along. I have not run OSX for 2 weeks now and I don't miss it.

Ubuntu is better than good enough.

Here is his How-To.

I am now moving my iPhoto pictures to Picasa and onto the web using 20GB of Google storage for $5/year. Then I'll clean up my files on OSX and do something else with the partition. I still have another small OSX partition to allow me to upgrade any Apple firmware and to check hardware such as the battery. It also means I have OSX should I need it.

I wonder if OSX will run under VirtualBox?

Solar Panel Update

2011-01-02T04:53:00.001+11:00

After a year, the performance did not seem to match what was estimated. We generated less than 2000 kWh whereas it should have been closer to 2700 based on mean solar hours. The data for the last 9 month indicated that this year would be no better.

I suspected a problem.

I contacted the supplier and they asked me to keep daily records which I more-or-less did for several weeks. I also obtained the solar radiation figures from the Bureau of Meteorology.

I was supposed to send them in, but I have not had time to do this.

Then, about a week ago, I noticed that the inverter was off - the circuit breaker had tripped and so had the one in the meter box.

Resetting them made no difference, so it seems that my inverter has died.

Last week, the inverter was replaced. It has a different UI which includes the energy generated so far for the current day down to 10Whr. Very useful.

Barlow St. Sydney Tram Tracks

2010-10-21T22:35:00.000+11:00

Energy Australia are doing some work in Barlow St. Sydney.

Their contractors(?) have uncovered some old Tram tracks. I managed to get a photo of one the rails and a partly visible sleeper.

I asked the friendly contractors for a small piece of rail - worth a try I think.

Blue-Green Water Results

2010-10-20T22:32:00.000+11:00

After almost 4 weeks, our blue-green stain has not returned.

Treated Tank Water Alkalinity

The addition of 1000 mL of lime raised the pH to nearly 11. Over time this has reduced to below 10 since about 10,000 L of new rain has been collected.

Rain Water Alkalinity

I also noted that the rain water had a pH of about 6. This was a little surprising since our tank water was pH 4.5.

Stale Treated Tank Water Alkalinity

In monitoring the treated tank water I have found that the alkalinity of the water decreases to nearly 7 when left for 24 hours or more. I suspect that this is due to the water absorbing CO2 from the air and this may also explain why rain water has a higher pH compared to our tank water - if left, the pH increases as the water absorbs CO2.

CO2 Absorption of Large and Small Bodies of Water

To help confirm this theory, I left two containers of treated tank water to stand. One was over 300mL (pH=9.94) and the other was less than 60 mL (pH=9.83). Both containers had similar surface areas. The idea is that given the similar surface areas, the two systems would absorb CO2 at the same rate but due to the different amounts of water, the larger body would change pH more slowly. This is what I observed: the smaller decreased pH to 7.17 in less than 24 hours. The larger body had a pH of 7.36. Not much difference, but the temporal reading indicate that the larger body of water lags the smaller.

Note: Although the two samples should have started with the same pH I suspect the smaller sample had already absorbed significant CO2 before I measured it. This was about 15 minutes.

Boiled Water

I also boiled some treated water, let it cool and checked its pH. The water was originally about 9.6 pH. After boiling it dropped to about 8 pH. I have no theory for this at present.

Blue-Green Water

2010-09-25T20:44:00.003+10:00

We collect rainwater from our roof into two large 30,000 litre tanks. A pump takes the water from near the bottom of one tank and feeds the house.

From the time all the down-pipes drained into the tanks, we have not run out of water. Previously less than one third of the roof was connected to the tanks and we ran out of water three times in about three years (all in the August - October months).

The Stain

At some point we noticed that our bath would become stained with a blue-green film. Over time the thickness of the film increases. The stain was on the taps, tiles, bath-base, shower curtain, and anything left in the bath for a long time. Being white, it was most noticeable on the bath-base.

The Experiment

Late last year, we ran an experiment: we cleaned the blue-green film from the bath and switched to town water.

The stain did not return.

I had a number of theories.

1. It was a bio-slim similar to that which would grow in a sand filter. I had experimented with filtering washing machine water through a sand filter and it develops a blue-grey slim on the surface of the sand.

2. It was some other mould.

I initially thought that the town water - being high in chorine - was keeping the mould/bio-film under control. But bleach/exit mould would not shift it so these theories did not make sense.

After a recent bath cleaning, I did some more research. The only mention of blue-green stains was in the context of copper stains - but we had rainwater which was 'pure' water I thought.

Rainwater is Acidic

It turns out that rainwater is acidic due to CO2 - carbonic acid.

Acidic water dissolves the protective copper oxide layer inside the pipes and in an alkaline environment, copper hydroxide will come out of solution.

The bath is probably an alkaline environment due to the soaps and shampoos.

I checked the toilet cistern - no blue-green stain. Probably because it is not an alkaline environment.
I checked the washing machine - no blue-green stain, but there is a yellow-brown stain (could this be some other copper product?).

Data Collection

I needed to measure pH. So I purchased a pH meter, buffer solutions and de-ionised water.
I also purchased some universal paper, but this does not seem to work very well.

All samples were taken after running the taps for a period long enough to ensure all stagnant water was flushed.

Tank Water

Water in tanks: 4.5 pH - acidic
Water from kitchen tap: 4.9-5.0 pH
Water from bath tap: 4.89 pH
Hot water from kitchen tap: 5.35 pH at 50 degrees C
Hot water from bath tap: 5.4 pH at 47 degrees C

Our rainwater is very acidic, but the more copper pipe it travels through the less acidic it gets.
Also, the hot water is less acidic than the cold water.

In both cases it seems that the acidity is being reduced as copper is being dissolved and this process is accelerated by high temperature.

Town Water

At meter: 7.75 pH

Water from kitchen tap: 7.75 pH

Town water is alkaline.

Stagnant Water

0 min: 5.11 pH
11 min: 5.56
13 min: 5.7
15 min: 5.7
17 min: 5.73
20 min: 5.82

It again looks like the acidity drops when the water is sitting in the pipe. I suspect that this is because the acidity is dissolving the copper pipe and it seems to happen quickly.

Research

I found that blue-green water is a mystery. There does not seem to be a single cause, but acidity, O2, CO2 and temperature are all suspects.

The CSIRO has done some research.

They suggested that Microbiologically Induced Corrosion (MIC) should be researched but they did not seem to offer a cause.

A massive Thesis by Owais E. Farooqi might be interesting to others as it contains a lot of data, statistical analysis and mathematical modelling of various forms of copper corrosion.

I did find one paper on copper corrosion in particle accelerators.

The paper shows that:

Increasing temperature increases copper solubility.
Very low O2 (less than 30 ppb) and very high O2 concentrations ( greater than 1000 ppb) decrease copper corrosion, but lower than 30 ppb is best.
pH less than 7 (caused by CO2) increases copper corrosion.

So, I need to increase pH to 7.5 or higher. A pH of 9 seems ideal. Town water seems to be above 7.5, so this indicates that the town water minimises copper corrosion.

At pH greater than 7, the corrosion due to temperature is minimised.

At pH greater than 8.5 and less than 9.5 minimises corrosion.

Treatment

My inlet is very low in the tank. This is supposed to be anaerobic so it should be low in O2 - but how low? I lowered the inlet pipe some time ago. Could this be contributing to copper corrosion by inadvertently ending up with a O2 concentration between 30 and 1000 ppb?

According to wikipedia, fresh water has 6mL per litre or 6,000,000 ppb.

After some experimentation I decided that 500ml of lime power added to one tank would be my first step.

This seemed to take the pH from about 5 to 5.8 almost immediately. But I needed the pH to be over 7.5 and ideally 9. So I added more lime and the pH went to 10.8.

I can not take the lime out, so I will monitor the pH over the next few weeks and see what it does.

Testing

The bath is virtually clear of the blue-green stain so time will tell if the stain re-appears. If it does not return I may have solved our problem.

How to remove commas from quoted strings in csv files

2010-08-26T22:54:00.001+10:00

I needed to remove commas (,) from within double quotes (") in a Comma Separated Variable (CSV) file.

For example, you can use cut -d, -f2,4,5 to extract fields 2, 4 and 5.

But if there is a comma in the text of a field like this "hello, world" you are generally stuck.

Also SQLite can also import csv files, but again commas in quotes cause problems.

I was stuck until I managed to create this sed script that will work in many cases.

# This bash function uses sed to remove up to 4 individual commas
# sequences from a quoted string in a csv file.

# eg. "Hello,, World, nice day." -> "Hello World nice day."

function removeCommas(){
  while read data; do
   echo "$data" | sed -e 's/^/,/g' | sed -e "s/$/,/g" \
   | sed -e 's/$,\"[^,]*$,*$[^,]*$,*$[^,]*$,*$[^,]*$,*$.*\",$/\1\2\3\4\5/g' \
   | sed -e 's/^,//g' | sed -e 's/,$//g'
  done
}

A friend mentioned that the sed commands can be combined as follows:

   echo "$data" \
   | sed \
   -e 's/^/,/g' \
   -e "s/$/,/g" \
   -e 's/$,\"[^,]*$,*$[^,]*$,*$[^,]*$,*$[^,]*$,*$.*\",$/\1\2\3\4\5/g' \
   -e 's/^,//g' \
   -e 's/,$//g'

How it works

The engine uses sed.

This sed statement takes input from stdin and replaces the regular expression 'from' with 'to' for any and all occurrences of 'from'.

sed -e 's/from/to/g'

First, I add a comma to the start and end of each line with this:

sed -e 's/^/,/g' | sed -e "s/$/,/g"

These make the function work for special cases and they get removed after the commas have been removed.

The 'from' string starts of by finding the beginning of a quoted string ',\"' then all non-comma characters '[^,]*'.

This pattern is enclosed in brackets '$' and '$' to assign the matching pattern to, in this case, part 1.

Then it matches the next one or more commas ',*'. This is not in brackets since we don't want them.

The next part of the pattern '$[^,]*$' is like the first: it finds a string of non-comma characters and keeps the values as part 2.

Then we skip over any commas again.

This sequence can be repeated as many times as you like. I did it 3 times.

The pattern ends like it starts. But this time it reads any character including commas up to the trailing quote using '.*'. Then it reads the trailing quote '\"' and comma ','. All of this makes part 5.

This means that it will only filter the first 4 sequences of commas. To make it do more, repeat the middle pattern ',*$[^,]*$' and increase the output parts (below).

The 'to' expression is simply the concatenation of the 5 parts of the quoted string that are guaranteed to not contain a comma (well, except possibly for the last part), where '\N' is the nth part.

\1\2\3\4\5

Testing

I tested the function with this.

function testthis(){

echo -n "\"$1\" --> \"$2\"..."

echo "$1" | removeCommas | grep -qE "$2" && echo "Pass" \

|| echo "FAIL"

}

# test removeCommas

testthis "this test should fail to test the 'tester'" "anything but this"

testthis "" ""

testthis "," ","

testthis ",," ",,"

testthis ",\"\"," ",\"\","

testthis ",\",\"," ",\"\","

testthis "1,2,\"3,4 5 6 7\",8,9,10" "1,2,\"34 5 6 7\",8,9,10"

testthis "1,2,\"3,4,5 6 7\",8,9,10" "1,2,\"345 6 7\",8,9,10"

testthis "1,2,\"3,4,5,6 7\",8,9,10" "1,2,\"3456 7\",8,9,10"

testthis "1,2,\"3,4,5,6,7\",8,9,10" "1,2,\"34567\",8,9,10"

testthis "1,2,\"3,,4,,5,,6,,7\",8,9,10" "1,2,\"34567\",8,9,10"

testthis "1,2,\"3,,4,,5,,6,,7,8\",8,9,10" "1,2,\"34567,8\",8,9,10"

testthis "\"tricky one where the quoted string starts the line 3,,4,,5,,6,,7,8\",8,9,10" "\"tricky one where the quoted string starts the line 34567,8\",8,9,10

testthis "1,2,\"3,,4,,5,,6,,7,8 tricky one where the quoted string ends the line \"" "1,2,\"34567,8 tricky one where the quoted string ends the line \""

Firewall Rule Testing with BASH and TCPTraceRoute

2010-07-17T19:10:00.000+10:00

I wanted to block the use of any DNS server except those that I select (google, openDNS and my router).

I also wanted to make sure that these DNS servers work and that others do not.

So I made a BASH script to verify my firewall rules.

#!/bin/bash

# only particular DNS servicer are allowed to be contacted.

# this tests that this is so

LOCAL="the.IP.address.of.your.router"

ALLOW="8.8.8.8 8.8.4.4 208.67.220.220 208.67.222.222"

BLOCK="220.233.0.4 61.88.88.88 202.139.83.3 61.9.194.49 61.9.195.193 61.9.133.193 61.9.134.49 203.161.158.2"

echo "Testing DNS servers via UDP that are allowed to work..."

for d in $LOCAL $ALLOW; do

dig @$d somename +time=1 +tries=1 +notcp > /dev/null

[ $? -ne 0 ] && echo "FAIL: Failed to get response from $d via UDP." && exit 1

echo "PASS: $d responded via UDP."

done

echo "Testing DNS servers via TCP that are allowed to work..."

for d in $LOCAL $ALLOW; do

dig @$d somename +time=1 +tries=1 +tcp > /dev/null

[ $? -ne 0 ] && echo "FAIL: Failed to get response from $d via TCP." && exit 1

echo "PASS: $d responded via TCP."

done

echo "Testing DNS servers via UDP that are NOT allowed to work..."

for d in $BLOCK; do

dig @$d somename +time=1 +tries=1 +notcp > /dev/null

[ $? -eq 0 ] && echo "FAIL: Got response from blocked DNS server $d via UDP." && exit 1

echo "PASS: DNS server $d via UDP was correctly blocked."

done

echo "Testing DNS servers via TCP that are NOT allowed to work..."

for d in $BLOCK; do

dig @$d somename +time=1 +tries=1 +tcp > /dev/null

[ $? -eq 0 ] && echo "FAIL: Got response from blocked DNS server $d via TCP." && exit 1

echo "PASS: DNS server $d via TCP was correctly blocked."

done

echo ""

echo "Firewall PASSed."

Then I thought that I might be able to test other TCP blocking rules by setting the IP packet's time-to-live (TTL) to a small number and looking for ICMP time expired packets. To do this I needed to use tcptraceroute to get the core functionality. On the Mac I got this from fink.

If I get some response, the firewall is NOT blocking an outgoing port. If I get stars (* * *) then it probably is blocking the port.

#!/bin/bash

echo "Testing blocked outgoing port..."

# set TTL to 2 hops: host to ADSL router, ADSL router to ISP gateway

# if the ISP responds to TCP TTL timeouts then a blocked port should get '2 *'

# whereas an open outgoing port should get something more complicated like this:

# '2 37.1.233.220.static.exetel.com.au (220.233.1.37) 23.176 ms'

#sudo tcptraceroute -q 1 -w 1 -f 2 -m 2 www.google.com 79 | grep '2 *' && echo "blocked"

BLOCK="135 136 137 138 139 445 593 1863 110 9000 5190 23 1503 1720 53"

VICTIM="www.some.real.site.com"

# The victim should not get any packets if the firewall rules are right.

for p in $BLOCK; do

sudo tcptraceroute -q 1 -w 1 -f 2 -m 2 $VICTIM $p | grep '2 \*'

[ $? -ne 0 ] && echo "FAIL: Port $p is open for outgoing traffic." && exit 1

echo "PASS: Port $p is blocked for outgoing traffic."

echo ""

done

ALLOW="80 8080 443 25 21 119 22 123"

VICTIM="www.some.real.site.com"

# again, the victim should not get any packets since TTL is so small.

for p in $ALLOW; do

sudo tcptraceroute -q 1 -w 1 -f 2 -m 2 $VICTIM $p | grep '2 \*'

[ $? -eq 0 ] && echo "FAIL: Port $p is blocked for outgoing traffic." && exit 1

echo "PASS: Port $p is open for outgoing traffic."

echo ""

done

echo ""

echo "Firewall PASSed."

You will need to modify the script to suit your firewall rules.

A Post from the command line using GoogleCL

2010-06-29T21:40:00.002+10:00

One small step for Google. One giant leap for scripting.

Google Captured Data and Passwords in Australia - an Estimate

2010-06-20T18:25:00.001+10:00

Most would have heard by now that Google's StreetView cars have been taking pictures of our streets and at the same time collecting WiFi SSIDs. But they have also been collecting other data that has got them into trouble with Governments and privacy groups.

http://googleblog.blogspot.com/search/label/privacy

Google sponsored report

http://www.google.com/googleblogs/pdfs/friedberg_sourcecode_analysis_060910.pdf

In Australia it is no different. Senator Conroy has been very vocal about this.

http://www.abc.net.au/news/stories/2010/05/25/2908415.htm

I was chatting to a friend and a quick calculation seemed to indicate that the amount of data and passwords captured must be small.

So, with lots of assumptions I have looked at two cases: the 'best' case and the 'worse' case.

To do any estimation I needed numbers. Fortunately our Bureau of Statistics (ABS) provided what I needed.

Here are my assumptions:

1. From the ABS December 2009, 5.2M ADSL and Cable subscribers. http://abs.gov.au/ausstats/abs@.nsf/mf/8153.0/

2. 114,400 TB downloaded data per year (ABS)

3. Uploaded data is 5 - 20% of downloaded data.

4. Between 50 and 80% of all subscribers use encrypted WiFi.

5. WiFi range from an indoor household Access Point is +-100m to +-250m.

6. The StreetView car samples 5 channels per second (See report).

7. Data in overlapping channels can be received in channels 1, 6, 11.

8. StreetView car travels at 30 - 50 km/h between 8:00 and 16:00. The car need bright light to take photos and for safety (fatigue) reasons they would only do 8 hour shifts - probably with breaks every 2 hours or so.

9. Uploaded data is sent evenly throughout the day from domestic homes and that between 8:00 and 16:00 the upload data rates are average.

10. Households send 1 - 10 non-secured passwords per day.

11. Household use the internet between 16 and 20 hours per day.

WiFi Reception Performance

One important variable that I did not model was the error rate of received frames that is related to distance: the further away from a WiFi access point, the lower the change of receiving a frame without error. The assumption that you can receive all frames within 100m (worse case) or within 250m (best case) is, frankly, silly and unrealistic. This will mean that any result is going to establish an upper limit for both cases.

WiFi Channels

So basically I assume the street car samples 3 channels at the rate of 5 per second. I conservatively assume that data on all the other WiFi channels can read using just these 3 channels. I doubt this is correct so again it will establish an upper limit for both cases.

Reception Period

I calculate the time the car is in range of a WiFi base is between 14 and 60 seconds and can sample between 5 and 20 seconds worth of data. The car probably travels at a speed between 30 and 50 km/h and can collect WiFi frames from any given access point for one third of the time.

Data Collected

By assuming that households transmit data continuously, and knowing the average amount of data sent each second I can estimate the amount of data collected on average per WiFi access point.

Australians downloaded 114,000 TB of data in 2009. Most web browsing is downloading, especially when we are talking about reading bank accounts and email.

Assuming uploaded data is between 5 and 20% of the amount we download, I arrived at an upload data amount of between 5700 - 23,000 TB per year or 400 - 1400 bps.

This means that they can record between 250 and 3400 Bytes of data per WiFi access point (SSID).

Password Capture

I have assumed that passwords are sent in the clear (unencrypted) between 1 to 10 times per day. Most sites (such as banks) use HTTPS and email hosting services generally support encrypted SMTP/POP/IMAP to send account and password information. But some sites may still allow access to mail and other discussion sites using unencrypted passwords. It is these passwords that Google could have captured on unencrypted WiFi access points.

The probability of capturing a password is between 0.002% and 0.14% so this means that between 17 and 3600 passwords would be captured Australia-wide as the StreetView car drove by.

Resulting Estimate

Data collected: 250 - 3400 Bytes per WiFi access point
Passwords collected: 17 - 3600 Australia-wide

The difference between my best and worse case is just over 1 order of magnitude for bytes collected and just over 2 orders of magnitude for passwords collected. This reflects the level of uncertainty on my estimates.

I think the real values will be much lower for these reasons:

1. During the day is not the peak time for downloads from households. On weekdays, on average, many family members will be at school or at work so it stands to reason that less internet activity will take place.

2. Encrypted WiFi access points seems to be closer to 80% rather than 50%. People are more aware about security and ISPs have done a lot to encourage the security of wireless networks.

3. Few services use unencrypted passwords - I can not think of any except for POP based email. All banks use some for of encryption - to do otherwise would be incompetent. Unfortunately it may be the small businesses that allow staff to access their email via unencrypted POP that are letting their employees down. GMail and Yahoo only seem to allow encrypted authentication so your account name and password are safe.

4. WiFi range is probably not even 100m and hardly 250m and the ability to pickup a transmission from a laptop/mobile at these distances is low. The further away from a WiFi access point, the higher the probability of receiving an errored frame that would contain no reliable data.

5. Uploaded data is probably less that 5% of Downloaded data. A packet containing an account name and password is small compared to the resulting page that gets downloaded.

6. Even when a WiFi access point is unencrypted, typically traffic to sites that require privacy are encrypted. So the actual unencrypted data is publicly available web pages, images, video and javascript code.

My Guess

The above best and worse case set an upper limit. But the worse case is too optimistic regarding the amount of uploaded traffic, WiFi range and the number of unencrypted access points. So I would suggest the number of passwords collected to be around 17 - say 10 to 100 - Australia-wide and the amount of unencrypted personal data to be much less - say 10 to 100 bytes per household.