Gnomic Notes

Archiving on hard disk

2010-09-27T14:37:00.000-04:00

How I archive at home:

The difference between an archive and a backup

The most important difference is an archive is the master copy and a backup is not a master copy. If you lose the archive copy, you risk losing the content permanently. Because it is is the master copy, it is sensible to have a backup for the master copy.

Media choice

I used to archive to CD-Rs and DVD-Rs. They had a great GiB/$ ratio which was an important factor for me. They were also impervious to some damages: power surge, file-system corruption, fat-finger deletion, etc. But they were inconvenient to handle as you need many of them. It was not so bad 10 years ago when I had my first CD-R drive because my data size was small wrt. to CD-R size. But over the years, my data size grew and grew and I've gotten lazier feeding DVD-Rs into the drive.

Now I archive to harddisk. It has became economical to do so. As of today, the lowest ratio for harddrive [1] on newegg.com is 14.70GiB/$. You'd need at least 2 of them, which brings it to 7.35GiB/$. The cheapest DVD-R media goes for $18.00 for 100 single-layer discs[2], equivalent to 24.27GiB/$. DVD-Rs are still cheaper, but it takes a lot of time to write to them.

Burning one DVD-R takes 15 mins, writing the equivalent amount, 4.37GiB to a harddisk through USB2 (assume write speed is 20MiB/s) takes about 3.7 mins. Let's assume that we are going for time and so we do not bother validating the archive (re-reading), and also the data stored exactly occupies a DVD-R. The first assumption gives a huge advantage to the DVD-R because it eliminates a huge overhead in validation: the media has to be ejected and re-inserted. The second assumption also eliminates the overhead in changing DVD-R media. So, the time saved is 11.3 mins/4.37GiB = 2.59mins/GiB. If you have to archive 100GiB, copying it to harddrive is quicker by 100*(15-3.7 mins/4.37GiB) = 4.31 hours. In reality, the time saved, thus the time-cost difference, is even more pronounced as you don't have to wait in front of the computer while the data is being copied. I am usually present for less than 5 minutes total: initiate copying, initiate validation. The time difference that matters to you (you not being a slave to the DVD-R drive overlord) is 100*15mins/4.37GiB - 5mins = 5.63 hours/100GiB.

For 5.63hours/100GiB, I'd pay the cost difference of 100*(24.27-7.35)/(24.27*7.35) = $9.49/100GiB for harddrive.

[1] $95 for 1.5 TB HD is $95 for 1396.98 GiB = 14.70GiB/$.
[2] SL DVD+R capacity: 2295104 2KiB sectors = 4.37 GiB. $18 for 100 discs = 24.27GiB/$.

Backup of archive

According to some articles, harddrive is prone to losing its magnetism when stored for a long time. This supposedly affects recent, high-density harddisk because the bits are packed closely together. Other causes that can affect a harddrive: the motor lubrication may evaporate or chemically degrade so as to prevent the disk spinning correctly, a power surge may fried some drive electronics.

Since the archive copy is the master copy, it is prudent to back it up. The easiest way is to have a similarly-sized drive. Then you can just do a bit-by-bit copy from the master archive to the backup archive which is faster than filesystem-level copy if the drive is fully filled.

The backup media should not be of the same make and model. Harddrive is manufactured in batches, and the same defect usually is present in every drive in a batch. If you buy two drives of the same make and model at the same time, they may come from the same batch.

Partitioning archive disk

If the archive disk is big if you may want to partition it into smaller pieces. Partitioning can limit the extent of filesystem corruption as long as you operate on (mount) one partition at a time.

How big should a partition be is up to you. For my data, I'd partition a 500GB drive into 2 250GB slices, and a 1.5TB drive to 3 500GB slices.

Encryption

Encyption is optional. But it can be useful in Windows OS.

Windows eagerly mount all partitions on a disk. This defeats the intention of having multiple partitions. However, encyrpting each partition will let you mount only certain partitions. FreeOTFE or Truecrypt is my encryption software choice on Windows.

Operation & maintenance

After archiving to the master disk, I follow up with doing a bit-by-bit copy of the affected partition to backup disk. This shows another benefit of partitioning: if you have a gargantuan drive, you don't have to touch all.

After copying to the master disk, I took both disks offline. I use SATA drives for my archive disks and have 2 SATA docks. The disks are offline (unpowered, plugged off, in safe storage) most of the time. The master is online only when I need to read or write from it. The backup is online whenever I modify master. Whenever the backup is online, there is a brief window of risk where both drives may be destroyed (e.g.: a power surge).

The window can be eliminated. One way is to copy the master partition to online disk, take the master offline, bring the backup online, write the copy of master partition from online disk to the backup disk. This is too cumbersome for me and I have no need for this kind of data guarantee.

Around every New Year, I do badblocks -n on the whole disk for each disk. See my previous posting about badblocks -n.

Badblocks: non-destructive read-write mode

2010-09-27T12:26:00.000-04:00

With the -n flag ("non-destructive read-write mode"), badblocks does the following sequence in every iteration:

read data
write test pattern (test pattern is ~0 (0xffff in 32-bit int))
read test pattern
compare test pattern
write data

It will try to write back the original data when terminated with the following signals:

SIGHUP
SIGINT
SIGPIPE
SIGTERM
SIGUSR1
SIGUSR2

Killing it with SIGKILL is dangerous because you have, at best, 3/4 chance of corrupting your disk (the only safe state to SIGKILL is when it does read data).

Setting up Samsung ML-1740 in OpenVZ VE

2010-05-26T03:09:00.001-04:00

This is a quick walkthrough for setting up Samsung ML-1740 (or other Samsung laser printers) through USB port in Debian Linux OpenVZ VE . This adds on my previous guide and present a shorter walkthrough than the one in http://wiki.openvz.org/USB_Printing_in_VE

On HN:

Make sure the printer is on and connected to the usb port.

# aptitude install usbutils
# lsusb

Make sure the printer is listed in lsusb's output.
If this is the first USB printer connected to HN, then there should be /dev/usb/lp0.
Make sure /dev/usb/lp0 uid is "root" and gid is "lp" and the permission is at least 660 (u=rw,g=rw).

# vzctl set 555 --devnodes usb/lp0:rw --save

Now enter the VE:

# vzctl enter 555

Make sure there is /dev/usb/lp0 in VE (exported by HN). Make sure the permission is the same as in HN, owner user "root" and group "lp", and permission is at least 660.

# aptitude install cups splix
If you already have CUPS installed, restart it because the printer has to be powered on and plugged in before CUPS is started for it to see the printer.

# ${webbrowser} http://localhost:631

Go to "Administration", "Add printer", Enter name and click "Continue".
Next there's a device selection. Your printer should be listed there. My printer was listed as "Samsung ML-1740 USB #1 (Samsung ML-1740)".
If your printer is not there, the more likely causes are incorrect permission and/or CUPS started before the printer is up.

Select your Samsung model on the next screen and keep pressing "Continue".
You're done.

I'll never use JFS for root FS anymore

2010-05-26T01:55:00.001-04:00

I love JFS, but I'm not going to use it for root FS anymore.

One of the computers had JFS for its root FS. The computer crashed. The filesystem became dirty and refused to be mounted rw until it is fsck-ed. Unfortunately, the fsck program for JFS exists only as a userland tool (kernel cannot do it) and Debian's standard initramfs image does not have fsck.jfs. So the root fs was mounted ro and, as a result, several important services, including ssh, failed to start.

It was a good thing that the computer was downstair. I just went downstair and used a rescue CD to fsck.jfs the volume. I had 3 rescue CDs: trinity 3.3, knoppix 6.2 and system rescue disk 1.3.2. Only the last one has fsck.jfs.

From now on, I'll be using JFS only for data volumes.

iftop, vmstat, iostat, dstat, iotop, lsof, fuser

2009-07-30T01:38:00.003-04:00

Some utilities I find helpful:

iftop -- shows current open connections and their transfer rates
vmstat -- shows virtual memory status (found in procps package)
iostat -- shows current IO transfer rate by devices (found in sysstat package)
dstat -- combines vmstat, iostat and iostat-like info for network IO
iotop -- like top, but the focus is on IO transfer rate
lsof -- get info from currently open files
fuser -- does a subset of lsof: identify processes and users using certain files

Adding an md device to a LVM VG containing the root filesystem

2009-07-20T19:54:00.003-04:00

Debian:

When adding an md device to a LVM VG containing the root filesystem, like so:
vgextend vg_with_root /dev/md99

Don't forget to run the following:

/usr/share/mdadm/mkconf > /etc/mdadm/mdadm.conf
dpkg-reconfigure mdadm

The first one is to update the mdadm.conf, and the second one is to reconfigure the initrd image the new md device will also be assembled prior to activating the VG. Otherwise, activating the VG will fail because it cannot find a required PV (the new md device).

my wife, tetris ds mania.

2009-04-22T00:49:00.001-04:00

Need to record this for family posterity: my wife just got 569131 points, level 31st.

Keeping track of manually installed packages in Debian

2009-03-17T17:13:00.004-04:00

This is how I keep track of manually installed packages in Debian:

Everytime I need to install a package, I create a file with the same name as the package name.

Examples:

$ touch /usr/local/etc/packages/list/qmail
$ echo 2009-03-17 J requested ipsvd for his app > /usr/local/etc/packages/ipsvd

$ make -f /usr/local/etc/packages/Makefile
apt-get install "ipsvd"
touch /usr/local/etc/packages/list/.ipsvd
apt-get install "qmail"
touch /usr/local/etc/packages/list/.qmail

And if you do another make:

$ make -f /usr/local/etc/packages/Makefile
make: Nothing to be done for `install-packages'.

In practice, I use aliases so I don't have type long paths. Here's how to set it up. Just copy-and-paste the code below:

mkdir -p /usr/local/etc/packages/list
cd /usr/local/etc/packages
cat > Makefile <<EOF
packages=/usr/local/etc/packages/list/*

install-packages: \$(packages)

\$(packages): /usr/local/etc/packages/list/%: /usr/local/etc/packages/list/.%

/usr/local/etc/packages/list/.%:
`echo -e \\\t`apt-get install "\$*"
`echo -e \\\t`touch \$@
EOF

Video feed for President's address to Congress

2009-02-24T21:59:00.004-05:00

CNN, MSNB, and FOX are all showing the same feed. Are they not allowed to put their own crews there?

Mini-Tripod

2009-02-06T00:49:00.004-05:00

The mini-tripod I ordered just arrived. It has bend-able legs and its small size makes it a perfect travelling companion for my pocket camera. This should solve my problem with having to keep the camera steady for night shooting.

So, what night shooting is complete without a few pictures of the stars? Here I have ursa major, leo, puppis, and canis major. Oh, look, we're even graced by Saturn's appearance.

I love the tripod.

Calvin & Hobbes, $15/glass lemonade

2009-01-26T17:58:00.006-05:00

[Source: http://4.bp.blogspot.com/_RPTAaOI4RN8/SXy9awMC4YI/AAAAAAAAAdc/U2yipcT6NjE/s1600-h/calvinhobbes.jpeg]

Oil's market is International

2009-01-26T17:37:00.002-05:00

"The oil is drilled right here in Alaska, and yet we're paying $8 a gallon? Something is amiss here. The oil companies are making billions of dollars, and people here can't afford to eat," said Pat Samson, social services director for the Assn. of Village Council Presidents in Bethel, about 35 miles southwest of Tuluksak. [Source: In rural Alaska, villagers suffer in near silence]

The plight of Alaskan rural residents further evidenced the maxim that the market for oil is International. Having the oil being pumped nearby does not convey any special previlege to residents living in the area. It does not shield the residents from having to lock in prices in Summer, and they cannot prevent the oil from leaving their area.

In fact, why should the oil companies have to prioritise local residents? The residents are poor and their demands, compared to the international market, are immaterial. Setting up, no, even thinking about setting up to serve local residents would cost more than simply throwing the oil to the international market.

An effect of having senders be responsible for their mail storage

2008-12-02T17:42:00.004-05:00

Some people opine that having the email sender be responsible for its storage would reduce spam. That may be so, but I see doing so empowers DoS attackers with another means to carry their deeds.

In the proposed infrastructure change, a sender would put the message on his/her server and send a brief notification to the addressees. The addressees' clients then pulls the email from the location specified in the notification. In that world, a DoS attacker can work with a spammer to send bogus notification messages to a large number of clients. The messages direct the clients to pull an email from the victim's server. When the clients do so, they slashdot the victim's server.

I do not see a sure way to prevent clients from retrieving the email. A client that automatically heeds all notification messages is certainly not preventable. But a client that fully defers to human moderation may also participate in the slashdotting as well. After all, users are as much part of the problems as the spammers in current spam problem.

Blacklisting the victim's server is also a non-solution because that could be what the attacker wants or interferes with legitimate email retrievals. Blacklisting a mail URI is slightly better but still not a good solution because URIs are cheap.

If this infrastructure becomes common place, DoS attackers no longer have to construct a botnet because the whole Internet just becomes a super gargantuan botnet.

Bad valve stem seal -- how to tell

2008-12-02T11:08:00.003-05:00

http://www.fordforums.com/f655/5-0-uses-oil-rings-valve-stem-seals-how-tell-103279/#post1080284

Well, there are a couple of ways to tell. If you start the engine up
and it puffs out smoke, it's the valve seals because you are burning
the oil that has leaked past the seals when the engine was off. If you
slow down by downshifting and when you accelerate again, and it smokes,
it's valve seals. Why? Because when you deaccelerate by downshifting,
your throttle plates are closed and you have a lot of manifold vacuum.
That vaccum sucks in the oil past the valve seals. Then when you
accelerate, the increased air/fuel charge sucks it into the combustion
chamber and it is burned.
Unless the maintenance was completely ignored, I'd guess that it is
valve seals. They will deteriorate with age while a piston ring will
not.

You pushed me over and I; Keep falling, keep, keep falling the cliff.

2008-11-20T01:53:00.007-05:00

A sobering graph from my favourite site: http://calculatedrisk.blogspot.com/2008/11/comparing-stock-market-crashes.html

I am not sure that we are even half-way through this crisis. Keep falling, keep, keep falling the cliff..

One thought though: why 'Subprime Mortgage Crisis'? While subprimers made the initial appearance, it seemed people using their house as ATM had the larger role.

Adjusting my gas furnace

2008-11-20T00:26:00.006-05:00

Disclaimer: tinkering with gas is dangerous. Below are steps I took to adjust my gas furnace for my own documentation purpose. I decided to share it to public for educational purpose. If you blow up as a result of reading this article, you are on your own.

This is my gas furnace. Manufactured in March 1990. It is 18 years old now. Its efficiency rating is output_btu_per_hour / input_btu_per_hour = 56K BTU/70.5K BTU = 79%. Modern gas furnaces reportedly have 90% or better efficiency. But mine still works and winter in northern virginia is quite mild. I see no reason to replace it yet.

It's been cycling in short interval and I wanted to see if I could do something about that.

The first thing I checked was the flame colour. It should be blue. Orange flame indicates insufficient oxygen or some contaminations. If there is not enough oxygen, the flame will produce soot.

To adjust oxygen level, adjust the air stuter (picture below):

The furnace was a forced air model. The flame heats a heat exchanger which quite similar to a radiator. A blower blows cold air through the exchanger. This heats the air before it comes out of the exhaust register. Adjusting this gas furnace involves balancing the heat put out by the furnace with the cooling the blower provides.

In my case, although the spec says that my furnace could generate at 56K BTU/hour, the blower was not powerful enough to cool the exchanger. My house had 3 intake registers, one on each floor. The ductings was not large enough to reduce the air flow resistance to my blower's capability. If I openned up the blower panel, the blower managed to push enough air to cool the furnace at full power. But obviously I could not run it like that for daily usage.

Adjusting the power of my furnace was done by adjusting the gas flow. Here's my final setting at 3/4 of full flow:

To get at the setting I let the furnace operate as usual while watching the fan limit control switch. I set mine to 100F,140F,200F. I've seen some set at 90F,140F,190F, or 90F,110F,190F. The spec says that the max air outlet temperature is 170F. So, my goal was to get the heat exchanger temp steady at around 170F.

I let the furnace run as normal. I reduced the gas flow if the temperature goes up to the OFF setting (200F) or increased the flow if it held steady at less than 170F. I was gentle when adjusting the flow; a small nudge may be all that's needed.

After about 1/2 hour of tweaking, I managed to have it hold steady at around 170F. It kept at that temperature until the thermostat tells the furnace to shut down. Cycling problem solved. Cost to do that: 2 hours research, 1/2 hour of staring at the control switch.

I finally found the culprit

2008-11-14T03:55:00.010-05:00

One month ago, my wife's car had a dead cylinder. It was cylinder #3. It passed all the usual tests: spark plug sparking, compression test showed 170psi across all cylinders, fuel injector ticking, presence of fuel injector signal. But it just didn't want to start.

Peering down the cylinder hole, I noticed a pool of oil on top of the piston. A leaking valve stem seal could put too much oil into they cylinder and foiled the proper mixture for combustion. So I replaced them and cylinder #3 was still dead.

I then wanted to double check the compression test figures. It showed 170psi for all cylinders. That was a good figure, but it measured only the peak compression and did not show there was enough compression at the right time. So I bought Harbor Freight Tool's leak down cylinder tester for $30.

But they showed minimal leakage.

I was desperate by this time. I decided to take a look at the fuel injector. Two years ago I replaced fuel injector for cylinder #3 because it went bad. So I was very doubtful that the fuel injector could go bad again within just 2 years. But I was desperate.

Just for the record, it's possible to remove the fuel injector harness (fuel rail) without removing the intake plenum. You need to remove the bracket that secures the accelerator cable though. The bracket is secured by 1 bolt on the cylinder head and 2 studs on the intake plenum. Remove the bolt and twist open the studs with vice grips. Doing so will damage the studs. I replaced the studs with bolts. Good riddance since I hate dealing with studs in general.

That's when I found the culprit. Compare cylinder 1 and 3's fuel injectors:

No wonder cylinder #3 was dead. The bad o-ring must had been leaking fuel into the cylinder.

It seemed that the replacement pintle cap I used was too wide. When securing the fuel injector into the harness, the smaller opening on the harness pushed the cap against the o-ring.

Fortunately I had spare pintle caps. It was a good thing I didn't cheapen out two years ago by ordering more than I needed. I trimmed the rim of one and installed it.

That's all it took to bring the cylinder operational again.

Installing and removing recessed valve keepers

2008-11-14T03:20:00.013-05:00

Nissan Sentra GA16DE, as is the case with many other Japanese engines, have recessed valve. As such, a common valve spring compressor design like the one below won't work.

One tool that can work (or rather be made to work) with recessed valves is Leslie 36200.

To install the valve keepers, use it without the black attachment. Just position it on top of the valve keepers and push down. It has a strong magnet at the end so as you push down the valve spring retainer, the magnet will capture the keepers.

A valve spring retainer and valve keepers.

But as it was, its diameter was too big for the valve cylinder. So it needed to be filed. I didn't have a grinder so it took me 3 hours while watching Pink Panther to file it down. Oh good times.

The yellow zip tie represented my target diameter. It was set to a slightly smaller diameter than the valve cylinder's.

I pushed the zip tie as far down as possible and marked places that were still thick.

1.5" was deep enough for GA16DE.

I don't have a picture of using it to remove the valve keepers, but here's one of me using the installation attachment. Just assemble the valve spring, retainer and keeper, and push down with the tool. Note that I was posing for the picture because I didn't take some while I was doing the actual installation. In a real installation picture, you should see an air compressor hose going into the cylinder or a string of nylon rope to float the valves up.

If you do not have enough power to push down the tool with your hand (awkward position, short stature, etc.), you can try pushing down the tool with your body weight as pictured above. Use a piece of wood to spread the pressure and push down. This works for the intake valves too which are positioned further inside the engine and may not allow you to push down effectively.

Water pump old vs new

2008-11-14T03:04:00.004-05:00

From 1994 Sentra GA16DE 1.6L:

new vs old water pump. The old one had been on the car for 5 years and was leaking.

darcs recorded modifications to ghost files

2008-09-25T17:23:00.006-04:00

I was trying to branch out a one year old project, but got a strange error:

$ darcs get +DBMigration DBMigration.copy
Unapplicable patch:
Mon Jul 23 14:53:41 EDT 2007 Yohanes Santoso
* added sisc-1.16.6 external package
darcs: ./src/com/example/DBMigration/Frob1.class: openBinaryFile: does not exist (No such file or directory)

I checked the changelog and found that the second patch on the tree recorded modifications to files that were never added to the repository:

$ darcs changes -s | tail
Mon Jul 23 14:53:41 EDT 2007 Yohanes Santoso
* added sisc-1.16.6 external package

A ./external_package/
A ./external_package/sisc-1.16.6.zip
M ./src/com/example/DBMigration/Frob1.class
M ./src/com/example/DBMigration/Migrate.class
M ./src/com/example/DBMigration/TemplateRepository.class

Mon Jul 23 10:19:02 EDT 2007 Yohanes Santoso
* initial import

A ./src/
A ./src/com/
A ./src/com/example/
A ./src/com/example/DBMigration/
A ./src/com/example/DBMigration/Frob1.java
A ./src/com/example/DBMigration/Migrate.java
A ./src/com/example/DBMigration/TemplateRepository.java

The imported files were the java source files, but somehow that patch recorded modifications to the class files which were never imported.

I tried many ways to fix this so I can get my branch, but found modifying the patches repository directly was the only one that worked.

Patch files are stored in _darcs/patches directory. They are gzip-ed text files. Each represent a changeset (darcs record). I located all the files that contain references to these .class files, and edited them out. It worked!

Brokers meltdown

2008-09-19T09:53:00.002-04:00

Today started with yet another rally. S&P 500 is currently at 105% of yesterday's closing, which itself was also a 105% of the closing on the day before yesterday.

Crazy. Now I'm trying to dump stocks I've been buying on the cheap, but my brokers (tradeking and etrade) websites are absolutely crawling. The tradeking one even crashed briefly.

Come on, come back up quickly, I need to sell my stocks to this frenzied people.

Which of the following is the largest?

2008-09-04T12:55:00.001-04:00

Gish Gallop

2008-09-04T02:27:00.002-04:00

Hah! I've been looking for this exact phrase for sometime:

Gish Gallop

it is a debate technique that drowns the opponent in bullshits, lies and half-truths. The opponent is left having to explain and correct the claims lest he is assumed to accept them.

A much easier way to access Samba share on Linux guest on Windows host

2008-08-14T21:33:00.003-04:00

There is a much simpler solution to my earlier's one.

Steps:

Create a VirtualBox's HIF and attach it to the VM.
Set its IP address to, say, 192.168.2.1.
Start the guest OS.
From within the guest OS, give a static address to the HIF, say, 192.168.2.2.
From host OS, make sure you can ping 192.168.2.2
From the guest OS, make sure you can ping 192.168.2.1

This establishes a private network between the host and the guest. The guest is perceived as another node in the network hosting a Samba share.

VirtualBox: Accessing Samba share on Linux guest on Windows host

2008-08-14T01:05:00.009-04:00

Environment: VirtualBox 1.6.4 (latest as of 2008-08-14) running on Windows XP SP3. Guest is a Debian GNU/Linux.

The goal is to have the host Windows able to mount a Samba share provided by the guest as long as the Samba instance is running. To accomplish that, we are going to create a loopback adapter interface on the host, use VirtualBox's HIF on the guest, and bridge the two interfaces together.

Steps:

Add a Microsoft Loopback Adapter device (through ControlPanel->Add Hardware).
Create a VirtualBox's HIF and attach it to the VM.
Go to Control Panel->Network Connections.
Highlight the loopback device and the HIF, and right-click 'Create Bridge'.
Go to the newly created bridge's property and set its IP address to, say, 192.168.2.1. The guest OS will be able to contact the host OS with this address.
Start the guest OS.
From within the guest OS, give a static address to the HIF, say, 192.168.2.2.
From host OS, make sure you can ping 192.168.2.2
From the guest OS, make sure you can ping 192.168.2.1

At this point, from the host Windows, you should be able to do this:

net use e: \\192.168.2.2\the_share_name "password_for_share" /user:username_for_share /persistent:no

Update 2008-08-14: You don't have to create a bridge. There's a simpler method.