As I was reading this week’s issue (55th) and clicking the links I stumbled upon couple of presentations that I thought I’ll mention. Some interesting facts and fiction from the presentations

• Interesting stats about Etsy
  • > 1 billion page views / month
  • $300+ million in sales in 2010
  • $41.1 million in December alone
  • 721 deploys in November
  • Engineering team grew 4x during past 15months
  • Great startup definition
• Work in trunk, branch in code
• Focusing on performance
• Logging and graphing almost everything
• Cool pictures and screenshots
• Perf-related data belongs in your server logs

Here are the presentations.

Configuring and running

There is a skype-bot project at github. It is written in Java and has support for simple functionality. You can clone it via:

git clone https://github.com/toomasr/skype-bot.git

The project is maven based and once you import the project into your favourite IDE that has maven support it should compile out of the box. Before we can actually run the project we need to make couple of adjustments. First make a copy of the project.properties file and make it personal.properties. Then change the username, password and pemfile attributes in the file. Make sure that the folder that has the pem file also has a der file that has the same name (except the extension of course).

Next lets build the project. It is sufficient to just issue a mvn install command. You will end up with a skype-bot-0.0.1-SNAPSHOT.jar in the target folder. Before we can run that we need to start the runtime. See Running the Runtime from the previous post.

Now we are ready to start the bot. Issue a java -jar target/skype-bot-0.0.1-SNAPSHOT.jar. Now expecting that the Runtime is still running and that you updated the credentials in the personal.properties file you should see your need friend come online and you can start a chat.

The bot has the commands that he reacts to defined in the org.zeroturnaround.commands package. But for starters you can tell him “bot: time” and it will tell you the time.

Web Server

The bot also launches a web server on port 2500 that has defined handler on the url http://0.0.0.0:2500/post. You can send a POST message there with the parameters message=? and group=?. This will instruct the bot to issue the message to the group. The group is looked up from the bot’s list of groups that matches the Conversation.PROPERTY.displayname.

So what can we do with this? Most of the todays services have webhooks. For example GitHub has Service Hooks, FogBugz have Web Hooks and others have similar services. This means that you can register a web hook and when an event happens the bot will tell you over Skype (in a group chat for example).

What next?

This is up to you, hopefully you have a friendly bot. Make him tell you important information that you care about and don’t let him waste your time. Check back to the original article if I’ve added anything to the series.

Join the Skype Developer Program

A lot of necessary documentation and also software is behind a login screen at Skype Developer Program. You will need to join the program to continue and log in. This will cost you 10$. I personally don’t like the monetary fee for documentation and a SDK but I’m happy this is not 100$.

Now the biggest challenge has been conquered and you can also open up Skype’s own Getting Started Guide. That guide will go into more details with some of the next steps, so if you get stuck here be sure to check their documentation also.

Create a new project at Skype

Lets create a new project at the developer page. Navigate to My Projects submenu.There you can choose between multiple projects to create.

Right now you need to choose SkypeKit for Desktop. You can name the project something along the lines of chat-bot-demo and fill the rest of the form how ever you like (or they require). Now we have a project in place and we can move on.

Download the SDK and the Runtime

Navigate to your just created project and lets get the SDK and Runtime. SDK is the library (com.skype.*) that you will use to talk to the runtime (headless Skype client). The SDK you can download right away but for the Runtime you need to put in a request and it can take up to several hours to get the download link (usually just 5-10 minutes).

Build a sid-java-wrapper.jar file

The SDK comes in source format only, before we can continue we need to get this into a single JAR file and also install it in your local maven repository. To build it we need ANT to be installed (I presume you do). Issue the following commands now:

toomasr@cigar:~/$ cd skypekit-sdk_sdk/interfaces/skype/java/api/
toomasr@cigar:~/skypekit-sdk_sdk/interfaces/skype/java/api$ ant
Buildfile: ~/skypekit-sdk_sdk/interfaces/skype/java/api/build.xml
 
clean:
   [delete] Deleting directory ~/skypekit-sdk_sdk/interfaces/skype/java/api/bin
 
compile:
    [mkdir] Created dir: ~/skypekit-sdk_sdk/interfaces/skype/java/api/bin
    [javac] Compiling 36 source files to ~/skypekit-sdk_sdk/interfaces/skype/java/api/bin
 
jar:
      [jar] Building jar: ~/skypekit-sdk_sdk/interfaces/skype/java/api/sid-java-wrapper.jar
 
clean-build:
 
main:
 
BUILD SUCCESSFUL
Total time: 3 seconds

We have the sid-java-wrapper.jar file now in that api folder. For easier project setup we will be using maven (and probably for some of you shoot you in the foot as an added bonus). We will need to install the JAR file into our local repository. I will use com.skype as the groupId and skype-sdk as the artifactId. Issue the following command to do the same.

toomasr@cigar:~/$ cd ~/skypekit-sdk_sdk/interfaces/skype/java/api/
mvn install:install-file -Dfile=sid-java-wrapper.jar -DgroupId=com.skype -DartifactId=skype-sdk -Dversion=1.0 -Dpackaging=jar

Run the runtime

Just to check if the runtime starts lets unpack and start the binary. If everything goes successfully you should see something similar. On a Linux machine I had to download couple of different binaries before I got a client that did not die (at least on EC2).

toomasr@cigar:~/downloadss/mac-x86-skypekit-novideo_3.5.1.1041_493734/bin/mac-x86$ ./mac-x86-skypekit-novideo 
SkypeRuntime Copyright (C) 2003-2011 Skype Technologies S.A.
SkypeRuntime Version: 3.4.1/mac-x86-skypekit-novideo_3.5.1.1041_493734
Proprietary and confidential, do not share this application.

Development keypair

Next we need a development keypair. It is a signed RSA private key that you need to request from the Skype developer website. See the screenshot.

Once you have the file downloaded lets convert it to a format that is used by the Skype SDK. This is the openssl command for this. Be smarter than me and don’t accidentally overwrite the downloaded PEM file because they let you download it only once. The keypair is valid for 60 days!

openssl pkcs8 -topk8 -in download-pair.pem -outform DER -out downloaded-pair.der -nocrypt

Create a Skype account for the bot

Go to skype.com and create an account for you bot. Also add this bot to your friend list and log in once with the bot and accept the request.

Lets sum it up

By now we should have the following in place

• Have 10USD less on you bank account
• Skype Developer account
• A project created at Skype Developer page
• Working Skype Runtime
• Skype SDK jar file
• Development Keypair in PEM and DER format
• New contact on your contact list and its username/password

If you are missing any of those or had problems getting them then check out the more in depth documentation at developer.skype.com or post a comment.

Where to go from now? Check out the next part, Getting it Running.

The Plan

Image by Simon Abrams

The Skype bot needs a bit more work than your average IRC bot because of the proprietary nature of Skype. There is much clicking and downloading of certificate stuff to get you started for example. Also I want to show you how to put the bot into action, so we will go beyond just the bot.

Getting started

In this Part I – Getting Started we will go through the steps to get you started with the Skype developer program and download all the necessary software for the bot. The registration process, necessary accounts etc. Once you complete these steps you can develop any other bot out there.

Hacking on the Java part

In this Part II – Getting It Running we will take a Java Skype bot and get it running. It requires a little more than just compile and run so it has its own chapter.

We will also look into how to use this bot to notify about important information to Skype chat rooms.

And maybe something more

If everything went well then you have your own bot running now. There are many ways to go forward and improve the bot and the services. Lets see if we have time to cover those.

Hopefully next time that I have a dumb crazy idea I go have a beer instead.

I chatted with the early arrived organizer and speakers about Ruby, Java deployments and British comedy sketches (Big Train, Little Britain and Alan Partridge).

The event had 2 presentations. One was about the Aframe, doing quite a niche stuff, they build a system for online video production (editing, transcoding, tagging). The presenter, John Cowie went over the technical hurdles that they have had to cross to get their product out the door. Also he had a positive story to tell that it is possible to shorten the gap between the ops and dev and thanks to that he is abe to take vacations without having interruptions as he used to (the London underground still told me to Mind the Gap).

The next presenter, John Leach talked about the Ceph project. A posix compliant distributed network storage and filesystem (think of HDFS, GFS) that happens to be OSS and is quickly maturing. We also got a fun history of the network storage stack. I was lucky enough to have read Apache Hadoop Goes Realtime at Facebook quite recently and was able to chime in couple of comments at later discussions.

After the presentations there were random chats and we visited a local pub. It was a great no pressure event where you can meet people with similar interest, either technical, comedy wise or board game wise (met a 9kyu Go player). The venue was really comfy and cosy.

PS. It was the second office I’ve seen so far that has a Go set, the first is ZeroTurnaround.

So in the previous part I gave you an intro of the game and how I stumbled upon the game and how it took over our lunches (now we have recovered and are back to random discussions). In this part we’ll look how to detect the furballs from Fling screenshots.

Abstract

Lets build a web service that accepts screenshots of Fling and sends you back a webpage link of the solution to the image or just bunch of attachments.

Solution

Lets divide and conquer.

• Image/Color recognition to detect the Furballs (covered in this piece)
• Algorithm to solve the puzzles (not covered, but code available)
• Algorithm to draw the 1 step screens (not covered, but code available)
• Email handling (not covered, but code available)

Detecting Furballs

When I started out with this one I had no idea how to do it. I sent an email to one of my friends who has done image recognition and asked him how to get started. He took a look at the screenshot and asked, why do I need to recognize anything, just detect the color and you can tell if it is an empty square or a furball! It sounded so obvious that I was somewhat ashamed for not thinking of this myself :)

Python and PIL

Go get the latest version of PIL installed to have the examples work. Also we’ll be working from the root of the fling solver project. Just clone my BitBucket repo. So lets get started and load up the image and verify that PIL is working. Or you can skip this and take a look at blog_demo_1.py.

shot = Image.open(os.path.join(os.path.dirname(os.path.abspath(sys.argv[0])),       
                               "etc/shots/sample001.png"))                          
shot_pix = shot.load()                                                              
shot.show()

Once you execute this an image should pop up, like the one on the right. Now lets isolate a square and dwell into it.

Because the grid is static it is a matter of figuring out the width and height and the offsets as where the squares start. You can use your top notch vision, pixel ruler or little bit of Python to get this right. Run the following and you will see how I’m using PIL to find the offsets (trial and error). You should see something like this.

x, y, w, h = 2, 70, 45, 45                                                          
# lets choose the leftmost square and visualise it                                  
draw = ImageDraw.Draw(shot)                                                         
draw.rectangle([x, y, x+w, y+h], outline=(255, 255, 255))                           
del draw                                                                            
shot.show()

Ok, excellent. We now know the coordinates. Lets now average the colors. We will iterate through every pixel and take an average of the R, G, B values. We’ll also display the image for debugging purposes.

# lets find the average color
avgR, avgG, avgB = 0,0,0
counter = 0
# load pixels of the image
shot_pix = shot.load()
 
for i in xrange(1, w):
    for j in xrange(1, h):
        counter = counter + 1
        avgR = avgR+shot_pix[x+i, y+j][0]
        avgG = avgG+shot_pix[x+i, y+j][1]
        avgB = avgB+shot_pix[x+i, y+j][2]
 
avgR = avgR/counter
avgB = avgB/counter
avgG = avgG/counter
 
avgC = (avgR,avgG,avgB)
 
draw = ImageDraw.Draw(shot)
draw.rectangle([x, y, x+w, y+h], outline=(255, 255, 255), fill=avgC)
del draw
shot.show()

Okay, looks about right. Now lets do this for the whole image. Here I will be reusing code from the flingsolver package. You can do the same.

fip = ImProcessor('etc/shots/sample002.png')
fip.debug = True
state = fip.process()
fip.show()

Voila, all squares are marked with their average color. Now detecting furballs and not furballs is quite easy. Just make a conditional based on some color. For example I used:

def isBall(...)
...
if avgR in range(60, 90) and avgG in range(90, 130) and avgB in range(36,55):        
	return False
else:                         
	return True

For the complete demo you can run blog_demo_1.py from the flingsolver repository. It should generate all the images shown in this post. Happy hacking your own images!

Next time when I have time I will continue with the solving Fling and writing the email handling.

Scaling

Throughout the years you’ve built multiple lists. Your data has grown. You have a big announce to make and you would like to send everyone a message. You make a campaign, duplicate this to the lists and hit send on all of them. You get couple of angry replies that they received multiple messages. Oh, okay, they were on multiple lists, I’m sure there is a solution for this.

The chimp says that use groups instead of lists. Have a master list with multiple groups and then you won’t have the problem of people receiving multiple emails of the campaign. Okay, lets give it a try. You create the list, you create the groups, you populate the groups. Some people are on a single group, some are on multiple groups. You send out a master campaign and you see that couple of people unsubscribed. Thats cool. Then you find out that they were unsubscribed from all the groups and unsubscription did not even let them choose a group. WTF?

You ask the chimp for help and they say that this is a feature. Choose one, a granular approach or bazooka but not both.

Blocking Any Account at MailChimp

Scenario: you are a happy chimp and sending out campaigns, everything is fun and cool until one day you are unable to log into your account. You get a message that your account has been blocked because you’re most probably a spammer. Wow, how did that happen?

You send an email to their support and find out that they found your address from stopforumspam.com and they block your account automatically. It does not matter how you ended up on that website ( in this case you have not even registered a forum account with this email address for ages ). You send them one more email and ask what provider do they recommend to use now that this bad luck hit you. They kindly reinstate your account. Weeehah!

So if you want to block somebody from MailChimp, get their email address to one of these stop spam sites. For quickest results see stopforumspam.com/add You can be sure that they can’t do anything for 48 hours.

Conclusions

Besides the two scenarios that I’ve mentioned I’ve had many more smaller issues with MailChimp. From not being able to call them, not being able to bulk delete people from large groups, not being able to remove email header images, not seeing email addresses of shared accessees etc.

The lists are dumb and cannot talk to each other without extensive scripts from infra wizards. Likewise, even if you are a dummy yourself, you cannot actually communicate with anyone at MC (they do not use telephone technology at all) except via email or Live Chat. Also, you frequently get signed out every 30 minutes whether you are in the middle of typing an email campaign or just sitting there wondering why you chose MC all those years ago…

Problem

You are running Mac OS X (maybe the problem is relevant on other platforms too) and even if you’ve started Tomcat and you can see the Java process in the Activity Monitor or with ps you can’t see the process with jps. You are not able to connect to the process with VisualVM nor JConsole. Bummer.

Solution

Tomcat scripts have a variable called CATALINA_TMPDIR and the java process is forked with

-Djava.io.tmpdir="$CATALINA_TMPDIR"

What happens is that jps and VisualVM/JConsole look for information about processes from the system TMP folder. Once you redefine this they cannot find their information. The solution is to eliminate this variable from the process.

Either execute the commandline that you can copy and paste from

ps aux | grep -i tomcat

and exclude the -Djava.io.tmpdir part or modify the script for your debug session. Thats it.

You’re welcome.

Introduction

My current Lenovo T60 laptop is at least 4 years old (nobody at my company knows anymore how old it is, as even support department doesn’t have any records about it) and is waiting for the replacement. It was running Windows XP pretty fine, if one doesn’t count some sporadic hangs and reboots. Around 1 month ago I started to experiment with different operating systems to see what’s happening in modern operating systems world.

At first I tried Ubuntu with it’s smart installer which installs the system somewhere in the existing NTFS file system which is pretty cool. except that laptop hung every other sleep…

Then I managed to install MacOS X on a separate hard drive, but I couldn’t find any hackintosh drivers for my particular wireless card (Intel PRO/Wireless 3945ABG).

Then I was back to Windows, which has Windows 7 as it’s most modern version. Well, it’s considered to be good practice to reinstall windows time after time so I decided to make a backup and do a clean install. For the first time in my more than 15 year experience with MS software I decided to use it’s own user state migration tool rather than manually copy all needed files and settings from ‘Documents and Settings’ folder. I don’t usually keep there much because of spaces in the folder names and all sorts of problems because of that, but still I got there some assorted important documents.

Fail F.A.S.T.

I happily selected ‘Files and Settings Transfer wizard’ (F.A.S.T.) from the Start menu, clicked next-next-next, let windows show me some progress bars and in ended up with a folder with a magic name USMT2.UNC and 2 files inside it: IMG00000001.DAT (with 1.2Gb size) and file called status (with a size of 16 bytes). Somehow I thought that exporting files from windows xp using it’s migration tool and then importing these settings in windows 7 sounds like a pretty normal usage pattern, but how wrong was I!

I copied all my backups to external hard drive, dropped all partitions and installed windows 7. (In the meantime I managed to fry that hard drive with everything on it, but that’s another story.) Now it was time to restore files from the backup!

So I went to Accessories menu to find something called ‘Easy Transfer’. Sounds promising… Except that it doesn’t know how to handle Windows XP FAST archive, that I had. Hmm. That’s weird. Searching on MS KB gave no pointers except that if I wanted to migrate files from an old computer I should have installed Windows 7 on a new machine, then created some magic disk for the old system, transferred files with it and then would import stuff back on Win7. How sick is that!? How the hell I’m supposed to to a clean upgrade if I must have 2 machines to transfer my stuff?

I googled for the magic folder name: USMT2.UNC. This time I found some command-line app called User State Migration Tool, which can be downloaded from MS, but unfortunately its loadstate.exe still couldn’t do anything with that poor folder. Another bit of information was that I should use exactly the same version of windows xp to restore files from such archive. (What’s the point of such transfer?) Anyhow, I installed Windows XP in a VirtualBox and was hoping to run restore procedure and then just copy files manually as I have done tens of times so far. Yeah… FAST said that I have insufficient space on my disk. How’s that – I had around 4Gigs of free space there and this stupid archive is only 1,2Gb?! So, even with the same Windows XP my ‘backup’ was useless.

Further search revealed a lot of complaining people who found themselves with such useless ‘backup’:

• http://www.bleepingcomputer.com/forums/topic1153.html
• microsoft’s own forums

However, this time I found a link to a tool called fastconv.

Here is a page with lengthy description:
http://windowsxp.mvps.org/fast.htm

Ok, let’s give it a try. At the first try fastconv log said to me: 
Error Source store is invalid. [ERROR=13 (Dh)]

Along with the fastconv there was one more utility called rmv2opq.exe and it was said that it

converts a removable media store into a normal compressed store

Whatever it means. Anyway, I ran it. The result was ‘astonishing’ – target folder contained the same .dat file and status file, but this time status file was 12, rather than 16 bytes! What a progress! However, this time, running fastconv.exe on these ‘new’ files gave more information – it started listing a lot of .dat files but crashed in the end. Fortunately, fastconv had a switch to keep temporary files. When I used that, it generated 2 folders: 0000/ and TRANS/.

Folder 0000/ contained 2255 files with names from 001.DAT to 8CF.DAT. When I looked inside these files I could see that these were extracted original files but with wrong names. TRANS/ folder had a binary file DB.DAT. However, when I looked inside I could see both .DAT file names mixed with original file names and all that in unicode, meaning that simple grepping wouldn’t help as unicode uses 2 bytes per symbol so all names had 0×00 between characters and everything was again mixed with binary data. Fiddling with the fastconv didn’t give any results as it kept crashing after generating temporary and database files.

Getting hands dirty

Ok, lets explore what we have. DB.DAT file’s header said: ‘m e m d b   d a t   f i l e   v 9   n o d b g‘. Couldn’t find anything reasonable for neither opening it nor format description to write my own parser.

Then I started looked for something to extract unicode text from binary files. And found an application called BinText from McAfee, that was doing exactly what I needed. From the DB.DAT file it generated me a listing like this: 

At some point I saw such pattern:

00000002CCB6   00000002CCB6      0   0000002E.DAT
00000002CD3A   00000002CD3A      0   arrow_l.cur
00000002CDB8   00000002CDB8      0   arrow_l.cur
00000002D002   00000002D002      0   arrow_m.cur
00000002D056   00000002D056      0   0000002F.DAT
00000002D0DA   00000002D0DA      0   arrow_m.cur
00000002D158   00000002D158      0   arrow_m.cur
00000002D3A2   00000002D3A2      0   arrow_r.cur
00000002D3F6   00000002D3F6      0   00000030.DAT
00000002D47A   00000002D47A      0   arrow_r.cur
00000002D4F8   00000002D4F8      0   arrow_r.cur

Which could be interpreted like this:

<original file name>
<corresponding .dat file>
<original file name in 8.3 format>
<original file name again in some other format>

Following was simple – clean up the log and leave only original file and .dat file:

$> grep -B 1 DAT trans.txt > files.log

Which resulted in a file that looked like this:

00000002F0A2   00000002F0A2      0   beam_r.cur
00000002F0F6   00000002F0F6      0   00000038.DAT
--
00000002F442   00000002F442      0   beam_rl.cur
00000002F496   00000002F496      0   00000039.DAT
--
00000002F7E2   00000002F7E2      0   beam_rm.cur
00000002F836   00000002F836      0   0000003A.DAT
--

Then I wrote a small program to parse this log and rename files (handling of duplicates was important as I couldn’t restore directory structure). Before final renaming I ran it several times with writing logs to spot all problems in the log file.

Ok, I lost the directory structure and some file names are still broken, but at least I got my files back and I can fix these small issues by hand.

Lessons learned? Don’t blindly trust ‘migration’ software and keep good ol’ files handy.

P.S. Post named after the song ‘I fought the law’ by Dead Kennedys.

toomasr@cigarillo-2:~$ java -server -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinal -version | grep -i heapsize
uintx ErgoHeapSizeLimit                    = 0                {product}
uintx InitialHeapSize                     := 65011712         {product}
uintx LargePageHeapSizeThreshold           = 134217728        {product}
uintx MaxHeapSize                         := 1040187392       {product}
java version "1.6.0_22"
Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)
toomasr@cigarillo-2:~$ java -client -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinal -version | grep -i heapsize
uintx ErgoHeapSizeLimit                    = 0                {product}
uintx InitialHeapSize                      = 0                {product}
uintx LargePageHeapSizeThreshold           = 134217728        {product}
uintx MaxHeapSize                          = 130862280        {product}
java version "1.6.0_22"
Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)

If there is an easier way, let me know. Will put this one on my tool belt at the moment.