Stone.log

Strip all EXIF metadata from images

2013-02-07T11:14:00.000+01:00

As we take many pictures with phones and more-and-more intelligent cameras, there are more-and-more personal information in the photos. Before we put them on the internet it is a good idea to strip all information from them, to protect our privacy. You can do that with libimage-exiftool-perl's exiftool. Like:

exiftool -all= somefile.jpg

Online storage migration with mdadm

2012-12-05T14:52:00.002+01:00

Far far away... I stumbled upon an interesting problem in the previous weeks what I want to share with you.

So... let's start with some background details. Let's assume we have two custom build storages (Linux boxes with tons of hard disks, forty-eight, bound together with LVM and presented to the clients with iSCSI using SCST target), yes I know it's the poor man's storage but works nicely. The clients are Linux boxes using the iSCSI targets as RAW disks (one per machine) for KVM virtual machines. So... the problem is to migrate from the first storage (I will call it Chew from now) to the other storage (Jeti) with the minimum possible downtime on the KVM virtual machines.

The first solution can come into mind can be to dd the disks between Chew and Jeti while the guests are down. But given into account that we speak about 7TB storage this is a way too much outage. There are rsync like dd tools, and block device support patches to rsync but for larger disks (around 1-2TB) the last sync what we must do in down time is still a big time.

After a long week and very few sleep an insane idea come into my mind... let's use mdadm to sync disks! I already tested RAID 1 on network attached block devices in the past so this wasn't very new to me, but these are running virtual machines and valuable data. The would cut my head off if something bad happens! So the idea what first sounded insane and a bit risky but it was the solution with the acceptable downtime.

With one of my colleagues (Lavian) we started to polish the plan. The first idea was to get the two Logical Volumes to one machine then synchronize the with mdadm RAID 1. We created the same LVM structure on Jeti that Chew had, then grew every LV with 4M to have enough space for RAID metadata at the end of the disks. We used raid metadata version 0.90 because it is the one locates at the end of the disks. It was an important point because the other metadata formats where metadata at the beginning of the disks could mess our data. We attached the LV from Jeti to Chew with iSCSI so Chew had both the receiving and the sending side of the synchronisation, then we set up RAID 1 with mdadm and waited to sync. Note the client KVM machines still seeing the old sized disks not the larger ones because we didn't told them about the growing and the guest virtual machines happily runnig.

After the sync the downtime comes... We shut down the guests, logged out from the iSCSI, and also disabled on Chew, the later was necessary because:

we don't need them anymore
we used iSCSI target with RAM cache so we had to be sure that the RAM is flushed to the disk.

We also backed up our flushing with few "sync"'s and "sysctl -w vm.drop_caches=3"'s on Chew.

So at this point both Chew and Jeti had the same data and we have the outage only for several minutes, yessss, we did it boss! It was time to stop the RAID, zero the superblocks, on Chew logout from Jeti and reconfigure the clients to use the new storage. It is only a few minutes and then the guests can restarted.

This worked on few machines but then shit happened.... we started to see IO errors on some machines, so after a bit cursing and switching back to Chew on those machines we started to inspect the situation.

We wasn't able to find the error in the above described process. We suspect that some caches didn't get flushed the disk but wasn't be able to locate them. So very disappointedly went home that day. At this time I, as I written earlier, I was sleeping very little... so after that terrible day I slept almost 16 hours in a row, oh man that was gorgeous! When I woke up I came up with the polished and final solution.

The aim was to synchronise the two storages for the virtual machines... so why don't we synchronise them on the host side, at the very last place before the guests, and let the whole stack do the caching as it wants to do and don't bother with it!

So we attached Jeti's storage to the host machines, running the KVM guests, through iSCSI and did the synchron (of the two iSCSI targets) there with mdadm. Now we had the downtime at the beginning of the process because we had to detach the original (Chew) iSCSI storage and do the LV grow then reattach the storage. Then we build the RAID 1 devices and started the guest machines on that, the RAID sync finished after powering on the guests.

Great success! The second method worked flawlessly and we migrated all of our machines without future gotchas. Plus, because we left the machines on RAID devices we will be able to do further migrations without downtime. So as Antoine de Saint-Exupe said: "Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.", we get over the first crazy looking process and get a much nicer process and future default config: running on RAID 1 devices despite when we only have one storage for that, to be able to migrate without downtime at the future.

I hope that you have enjoyed this small story and if you know why the hell the first method not worked feel free to drop me a line!

Keep hacking! Cheers!

SCP Agent Forward-dal

2012-04-25T19:56:00.000+02:00

Felmerült a feladat, hogy SCP-zzek olyan gépre ahova csak SSH Agent Forward-dal lehet bejutni. (Igazábol egy SCB követelte meg a dolgot.)

A probléma az, hogy az SCP alapból nem továbbít SSH Agent-et. De szerencsére rátaláltam erre az írásra, ami megadta a megoldást.

A probléma gyökere az, hogy az SCP amikor meghívja az SSH-t akkor alaból kikapcsolja az Agent továbbítást, és ezt nem lehet felülbírálni sem a ~/.ssh/config szerkesztésével, sem a -A parancssori opcióval.

Szerencsére van az SCP-nek egy -S opciója, ahol meg lehet adni, hogy milyen SSH binárist futtasson. Így ha írunk egy wrappert, ami kiszedi a default Agent Forward tiltást az SCP által az SSH-nak átadott opciókból akkor nyert ügyünk van. Mentsük el a következő 3 soros kis script-et valami néven, például ssh-wrapper, adjunk rá végrehajtási jogot

#!/usr/bin/perl

exec '/usr/bin/ssh', map {$_ eq '-oForwardAgent no' ? '-oForwardAgent yes' : $_} @ARGV;

majd ezek után az SCP-t futtassuk a következő módon

$ scp -S ./ssh-wrapper some-file my-server:

végül dőljünk hátra elégedetten!:)

Make parallel script in bash

2012-04-03T16:03:00.004+02:00

I have a script what does thing as processing a file line-by-line. I modified this script to do parallel processing like follows:

#!/bin/bash

MAXJOBS=3

while read line; do
  ( echo "do thing with $line" ) &
  
  while [ "`jobs -p | wc -l`" -ge "$MAXJOBS" ]; do
    sleep 1;
  done

done < input.txt

wait

It starts the sub processes and if the count is as great as MAXJOBS then waits. At the end of the script the wait waits all the remaining processes.

UPDATE: If you need som for example grepping before on the input you should use:


...
done < <(cat input.txt|grep '#')

wait

instead of

cat input.txt|grep '#'| while read line; do

Files in directories

2011-12-07T17:05:00.003+01:00

I was playing with some rsync-ing nowdays and somehow there was more files than I was expecting. Tons of files... I was wondering how to find out where are those files in the huge directory structure.

My solution was this:

find -L . -maxdepth 1 -mindepth 1 -type d -exec /bin/bash -c 'echo -n "{} "; find -L {}|wc -l' \; | column -t

It prints the file count of the directoryies where you run.

With the help of this pice of .... hmmm... beauty, I was able to find thousands of files that neadn't to be there.:)

It works like follows:

stone@home:/usr/src/linux-headers-3.0.0-14$ find -L . -maxdepth 1 -mindepth 1 -type d -exec /bin/bash -c 'echo -n "{} "; find -L {}|wc -l' \; | column -t
./kernel         28
./arch           8032
./ubuntu         28
./Documentation  57
./virt           3
./drivers        1673
./tools          51
./ipc            2
./scripts        241
./crypto         6
./include        2670
./usr            3
./security       24
./sound          240
./net            194
./block          4
./firmware       2
./fs             221
./mm             4
./samples        19
./lib            21
./init           3
stone@home:/usr/src/linux-headers-3.0.0-14$

Goodby Unity, Wellcome Xfce (some tweaks to make me happy)

2011-11-28T16:04:00.003+01:00

I was fed up with the Unity and Gnome3 thing. I wanted to get my Gnome2 back and the old just works feeling. So I switched to xubuntu...

I was happy after configuring the panels to mimic the old Gnome2 but had to do some other tweaks to get the old feeling.

First was the annoying habbit that windows jumped between workspaces (for example when clicking a link in Thunderbird made my Chrome to jump to the same workspace) but after a bit Goggling I found the solution:

xfconf-query -c xfwm4 -p /general/activate_action -s none

to get the old scrollbars do:

apt-get remove overlay-scrollbar liboverlay-scrollbar*

Now I am completly happy!:)

Bitcoin -- money of the future

2011-06-02T18:10:00.003+02:00

Bitcoin is a P2P virtual money what is used as real money on the Internet for several years now. Some sais that this is the greatest innovation and the most dangerous thing since the born of the Internet. It is fully anonymous, decentralized and secure payment system. It doesn't needs a central authoritative to make it work and even governments can't control, tax or stop it. But for me the mathematics and the idea what makes it work is the greatest about it.

First of all it is backed up with RSA. So you have your private and public key (you have more actually to make it more private) and you can spend and receive money with them. Second it is backed up with a clever distributed time stamping solution to prevent double spendings.

Bitcoin as it is a chain of transaction blocks. This chain is maintained by the whole network and contains all transactions since the beginning (if it gets very long, there is ways to compress it). When you want to spend your money you concatenate the amount of money, the last blocks signature wih the recipient's public key and sign them with your private key then broadcast it on the network.

Clients do a proof-of-work process to include all new transactions in the chain. They form a block with the last block's signature, the new transactions and a special random part. This last random part is what is the prof-of-work. The client have to guess this part as to make the new block's hash (sha256) start with a fixed amount of zeros. Finding this kind of block is hard, the whole network can find just one in every ten minutes. When the machines become fast or the network grows the number of required zeros will grow.

So you have this chain of blocks with many zeros, does it good for you? Yes, because if your transaction is in the chain and is followed with several new blocks you can be sure that it is extreme hard to double spend that money what you got. Only an attacker who owns the majority of CPU power of the network can undo a transaction, and this scenario is very unlikely.

Every client works on his chain as it knows it and computes the proof of work. If a longer chain appears (some other client manages to do the work) it switches to it. Always the longest chain considered to be the real one. This is why the majority of CPU power rules the thing, because they can make the chain grow faster. An attacker's alternative chain will fall behind fast and gets ignored by the network and the first real spending remains the true.

How can someone get coins to spend? How Bitcoins are created? Since there is no central issuer Bitcoins materializes from thin air. There is a limited amount of Bitcoins can exists (21.000.000, but the smallest amount is 0.00000001 BTC, so inflation will take care that there will be enough for everyone) and till this amount is not created (the creation slows as time goes by, around 2100 will the last created) Bitcoins given to those who creates the proof-of-work. Those who does this coin creation seriously called the Miners. But Bitcoins can be exchanged from any ordinary money too (I think it can be a good investment as the price of Bitcoins rises).

As you can see the thing works. The details and the proofs for the method can be found in Satoshi Nakamoto's paper Bitcoin: A Peer-to-Peer Electronic Cash System. You can get the program from www.bitcoin.org and start using Bitcoins. You don't need to register, just run the program. It is available for Linux, OS-X and Windows. The Bitcoin users and accepting businesses number grows rapidly.

Rename Cyrus IMAP folders

2011-05-11T12:45:00.003+02:00

I had the task to restructure some shared IMAP folders on our IMAP server running Cyrus.

I come up with the following solution.

Rename the IMAP folder in cyradmin with the renm fromwhat towhat command

Adjust all users subscribed folder list by issuing the following command in the /var/lib/cyrus/user directory:
```
find . -iname '*.sub' -exec sed -i 's/fromwhat\x09/towhat\x09/' {} \;
```

This did the thing for me. Cheers!:)

Secure wireless gateway

2011-03-10T17:28:00.003+01:00

I'll go to abroad in the near future. At the hotel there will be Internet connection at the room, but only wired. I'll take my Android phone and my laptop, so I'll bring a small wireless router with me too. It is a Fonera 2g as hardware but runs OpenWrt.

As I'll have my own AP there why not secure my connection? I also have a private server somewhere at the USA, so I fired up an OpenVPN connection between my router and my server, and redirected all data through it. So now I'll have unsniffable connection in the hotel.

How I did it? It was easy after putting the VPN together (was also wasn't a big deal). Here is the recipe:

First I configured the OpenVPN normally but added the redirect-gateway at the end of the client config file, to make it redirect the default gateway through the VPN. After that I enabled it at boot time with

/etc/init.d/openvpn enable

I've edited /etc/firewall.user and added the next 3 lines to it:

iptables -I forwarding_rule -i tun+ -o br-lan -d 192.168.6.0/24 -j ACCEPT
iptables -I forwarding_rule -i br-lan -o tun+ -s 192.168.6.0/24 -j ACCEPT
iptables -t nat -I POSTROUTING -o tun0 -j MASQUERADE

where 192.168.6.0/24 is my wireless IP network.

And that's all. If now I boot the router it automatically connects to VPN and after redirects the gateway through it.

How to block ads in Opera?

2011-03-07T19:10:00.002+01:00

As far as I know (after 1 hours of goggling:)) I've found that the solution provided by http://www.fanboy.co.nz/adblock/opera/ is the best source for blocking ads in Opera. Just download http://www.fanboy.co.nz/adblock/opera/urlfilter.ini and put it on your Opera directory (described at the begining of the file) and voi la it blocks ads.

Adventures in native ZFS on Linux

2011-02-23T14:09:00.001+01:00

I started using ZFS on my Linux box around 1 year ago, after I've taken part on a Sun training where BalaBit (my workplace) sent me. ZFS astonished me. It combined the whole RAID, LVM and file system thing together and made it more pleasant with constant data integration check, and powerful caching. I felt in love.

But wait, stop, what the hack is ZFS? It is a file system and a build in logical volume manager by Sun Microsystems. With many interesting features like: data integrity, support for high storage capacities, snapshots and copy-on-write clones, continuous integrity checking and automatic repair, RAID-Z, deduplication, compression, multi level caching (with SSD if you like) and encryption.

That time Linux and ZFS wasn't a good company because of licensing issues (it still exists), but ZFS was available through FUSE (Filesystem In User Space). So I repartitioned my drive at home and started using it. That time only for some not so critical data of mine. I did some benchmarking on it too. It's speed wasn't something big because of FUSE but it was OK for me.

Time went by and a company namely Knowledge Quest Infotech started to write a port os ZFS based on Lawrence Livermore National Laboratory's work can be found at http://zfsonlinux.org/. And after several months it became ready for use (in january 2011). The licensing issues are still here so it is only available as module and maybe never will go to the native kernel. Building the modules or download from KQ Stor's (the brand what Knowledge Quest Infotech made for ZFS) is acceptable for me. (Oh sometimes I hate living on the edge... but this is the way I do in informatics.:))

I had several initial issues about the new native ZFS, for example that importing the old pool, what I made with ZFS-FUSE made all my files access mode to 000, but upgrading the file system version resolved this.

The most annoying issue I've found was the memory consumption of ZFS, first I suspected it as a memory leak, but it turned out to be a feature, but not on my desktop. ZFS itself takes very large amount of memory for ARC caching and if it needs by applications it releases them, the problem was that my PC was overloaded, Firefox, Thunderbird, several JAVA apps, etc... so my 3G RAM was almost full, but when ZFS started to allocate many memory and my desktop would need it my computer started to swap despite ZFS started to free memory (it was slower than kswapd). So many times this rendered my PC unusable for minutes. But after discussing it on the lists I learned how to limit ZFS's memory allocation so now everything is perfect. Don't blame ZFS, it's memory allocation is perfect for servers with more than 4G memory but racing with my desktop programs it wasn't a success story. But now it's good and I am happy with it. I've just bought another two 1T winchester to boost my storage space and with ZFS it was faster than I wrote this sentence!

It is still a new comer in Linux but actively developed so these first glitches will be surely resolved in the near future and it will become the most powerful file system in Linux.

It this post awakened your interesting please give it a try.:)

Android 2.2.1 on my Galaxy Spica

2010-12-30T12:39:00.001+01:00

Thanks for the guys at Samdroid I am using 2.2 on my phone for a week now and happy with it. Brain Fuck Scheduler is great!:) My phone is faster then ever!

Published with Blogger-droid v1.6.5

Slicehost is my new VPS provider

2010-12-28T16:09:00.001+01:00

I've switched to http://slicehost.com because I was a bit disapppinted with my old VPS provider.

Slicehost is an affordable and as far as I use a reliable VPS solution. It brings the newest Ubuntu distros with the possibility of using your own kernel.

The administration interface is great, it gives you console access. The servers aren't overloaded. It gives you cheap backup function too.

Give it a try if you looking for a VPS server!

Published with Blogger-droid v1.6.5

Installing VMware Server on Ubuntu 10.10

2010-12-21T15:04:00.004+01:00

Since VMware Server's developement doesn't follow Linux kernel changes, installing on recent distros isn't easy. Fortunately there are great ones who do patches and howto's to help.

I have folloverd the following post http://risesecurity.org/2010/04/02/vmware-server-2-0-2-update-patch-2/ and succesfully installed VMware Server 2.0.2 on my Ubuntu 10.10 box. Thanks!

Sudoku capture and solve on Android

2010-12-04T22:00:00.001+01:00

Tired solving Sudoku? Try Sudoku Capture on Android! Just take a photo on the problem and the program does the rest! Works great.

Published with Blogger-droid v1.6.5

Opera Mobile on Android

2010-12-04T21:42:00.001+01:00

I've just switched (again after a few months) to Opera on my mobile, as the default browser. I think Opera turbo and Opera link makes it the best and unbeatable mobile browser out there. You should try it if you havent already using it!

Published with Blogger-droid v1.6.5

I love VMware!

2010-11-18T17:43:00.003+01:00

It seems that I am on the road to become a huge fan of VMware. I am started to experience VShere and ESXi these days. I do it in WMware Workstation and it is great! I was a VirtualBox fan till this but new horizons opened for me. Zero click Windows installation, and so on...

I plan to become a professional on VMWare thinks, my first aim is to get a VCP (VMware Certified Professional). I hope that in 1 year I'll write a post about getting it.:)

Joe Satriani concert

2010-11-09T17:34:00.001+01:00

Yesterday was the Satriani concert here in Budapest. I was there. I can tell you that this was the best one I've ever seen.
The band was great, they enjoyed the show as well. The new tracks are great, but they played many old ones too. The extra was super with many great fast songs.
It was a great pleasure to be there. If you can join some future stop of the tour!

Published with Blogger-droid v1.6.5

Daylight saving time

2010-11-05T18:45:00.002+01:00

Since 1916 we change time two times a year. First it was invented to save money on lighting. But nowadays it causes more trouble than gain.

OK, on one side I like to see the sun at winter mornings when I wake up, it's better than waking up in dark and coming home in dark. But for example on calendar systems and many other places in this computerized world it causes trouble and eats up resources to adjust programs follow the many times changing and not world wide universal DST. In my company we had troubles in our calendar system because of DST...

And it you want the biggest reason against DST then it is my cat. Since we set the clock back 1 hour a week ago, he always wakes us up 1 hour before my wake up time, because he is hungry! An I don't like it!:)

I wrote my first exploit

2010-11-02T08:33:00.004+01:00

I had the following program for many years now:

main(int argc , char argv[])
{
char rest[100];
char first;
char *e;
e = getenv("NAME");
if(!e) exit(1);
if(strlen(e)>sizeof(rest)) exit(1);
first = e[0];
strcpy(rest, &e[1]);
printf("First char =%c\n",first);
printf("the rest =%s\n",rest);
}

I found the in the Data Fort Hack challenge.

I know the error in this code since the beginning but I wasn't able to exploit it. After reading the first part of the book Hacking: The Art Of Exploitation, I bought a few weeks ago, I succeeded to write an exploit this Saturday. My first ever working exploit!:)

It was great to see at the end that:

sh-3.2# whoami
root
sh-3.2#

androidwiki.hu lauched

2010-11-02T08:20:00.001+01:00

Today the Hungarian Android Wiki at http://androidwiki.hu is launched to the public.

Switching out stack randomization

2010-10-28T11:33:00.003+02:00

Since now I'm reading Hacking - The Art Of Exploitation and writing my first exploits I had to switch off stack randomization. The book presupposes this in the examples. Doing this is easy:

sudo sysctl -w kernel.randomize_va_space=0

Hacking: The Art of Exploitation

2010-10-22T07:57:00.001+02:00

Today I'll get my new book Jon Erickson's Hacking: The Art of Exploitation. I am looking forwart to it with excitement!

Published with Blogger-droid v1.6.3

Android Wiki

2010-10-22T07:46:00.001+02:00

Only a few days and the Hungarian Android Wiki will be launched at http://androidwiki.hu

I'am working hard to do the initial data, to get started, the rest is on You!

Published with Blogger-droid v1.6.3

Sonar app on Android

2010-10-20T17:56:00.001+02:00

I've found an interesting app on the Market. It is Sonar by DiCon. It can measure distances by generating small blips and listening to the reflections. Like a real sonar.

Published with Blogger-droid v1.6.3