Andrius Miasnikovas

Quick Glimpse at Go 1.5

Tue, 14 Jul 2015 00:00:00 +0000

I’ve been playing around with Golang for a few months now and I have to conclude that it is indeed a practical, well thought out language that is also fun to use. If you haven’t heard about or tried it yet, head over to tour.golang.org right now. There you can try the language right inside your browser, hassle free, no installation required. Though there’s something compelling about compiling things on your own machine. Actually, I have to admit that I was not that “into” Go when I first saw it and it took me about a year to come around and try it out again, but this time focusing more on the why, not the how. What really spiked my interest was watching these videos where developers from the Go team explain their thinking behind the decisions made and present their view of a modern, pragmatic programming language and tooling as a big part of it all. If you have some free time I encourage you to watch them. I do believe that there’s something to be gained here even if you choose not to learn Go. At least watch the talk by Rob Pike.

Some say that Go is a system programming language, though in reality it’s more of a general purpose one. But in the end there is no single programming language that will fit all of the criteria for all the people, that’s why there are so many of them. Though the way I see it - not many languages have a justifiable reason for their existance. When you dig into the history of some languages quite a few of the currently popular ones started out as some side-project, hack or a pre-processor for another language as a way to avoid using some unwanted other language. (The language names have been omitted to protect the innocent).

The thing I personally have been looking forward to this year is the release of the Go 1.5 which should be quite soon, I believe the 1 of August is the current due date. The cool thing is that beta is already out and can be downloaded today from the golang page. Among various improvements to the compiler and a lot of focus on Garbage Collection, my favorite feature is the new crazy simple way of cross compiling i.e. building statically linked, zero dependency binaries for platforms other than that on which you’re writing the code. Once you have your GOPATH environment variable setup, the build process is already as simple as doing a go build inside your project directory. But now you can specify the target Operating System and Architecture, so on a linux machine you can do the following

$ GOOS=windows GOARCH=amd64 go build .

this will produce a 64-bit Windows executable file. Similarly you can specify any other supported platform. And you can even create binaries for Mac OS X, now you don’t even need an Apple computer for that.

But wait, there’s more. What if I told you you can now develop Android and iOS applications with Go. Well actually I don’t think it’s meant as a replacement of the current development environments for mobile apps, but rather as a supplement for performance sensitive parts of the application (think NDK for Android). The iOS part is not yet ready at the time of writing, but you can go try the Android part right now. All you need to do is install a couple of packages and initialize the environment.

$ go get -u golang.org/x/mobile/cmd/gomobile
$ go get -u golang.org/x/mobile/cmd/gobind
$ gomobile init

Now you’re ready. There are a few sample apps included in the gomobile package. To try one out just do the following

$ cd $GOPATH/src/golang.org/x/mobile/example/sprite
$ gomobile build .

Voila, you should see a sprite.apk file that is the result of this build. You can now install it onto your Android phone or an emulator. If you’re going to try it on an emulator make sure that the emulator’s system image is for ARM architecture since that’s the target architecture for the library. Will need to investigate if there’s a way to build it for different architectures, because right now this app wouldn’t run on an Atom-based Android device. The quickest way to install the app (presuming you have Android SDK) is to run

$ adb install basic.apk

And to simplify things further you can run the application based on the package name which can be extracted from the .apk itself.

$ run-apk.sh basic.apk

Here’s the source for the run-apk.sh script:

#!/bin/bash
pkg=$(aapt dump badging $1|awk -F" " '/package/ {print $2}'|awk -F"'" '/name=/ {print $2}')
adb shell monkey -p $pkg -c android.intent.category.LAUNCHER 1

Admittedly this wasn’t a changelog type of overview of the new version of Go, but I just wanted to quickly mention a couple of cool new things you’ll be able to do with Go 1.5, which I’m excited about. Hopefully some of this sparked enough interest to motivate you to go and check out the language for yourself. Happy exploring :)

Quick Tip: Exporting PuTTY Sessions

Tue, 13 Jan 2015 00:00:00 +0000

Here’s a really quick tip. If you’re running Windows and are using PuTTY to manage your unix servers, chances are you have a lot of saved sessions i.e. information on how to connect to your server, what font type and size to use, etc. And if you ever tried copying them to another machine you noticed that PuTTY doesn’t have a user-friendly export/import sessions button. But there’s a way to perform a saved sessions export manually. PuTTY keeps all session related information in system registry, so what you can do is export that part of the registry tree to a file with the following command:

regedit /ea sessions.reg HKEY_CURRENT_USER\Software\SimonTatham\PuTTY

The line above will create a sessions.reg file which you can copy to a machine where you want to import your saved sessions and simply double click it. Windows will ask you if you’re sure you want to run it, since you’re actually modifying the system registry, so click Yes and voilà. You can run PuTTY and start using it the way you’re used to. One thing to note is that if you were using private/public key authentication (passwordless logins) for some servers then you need to copy over your private keys to the new machine as well. And remember to place them so that they can be found using the same path as on the old machine. If in doubt, just open sessions.reg with any text editor and browse trough it looking for lines like the following, then you’ll know what files you need to copy and where to put them.

"PublicKeyFile"="C:\\var\\keys\\key1.ppk"

Another use for this session export could be when you want to change something like a font on all your saved sessions, but when you have a lot of them, then it becomes a time consuming process. What you could do is export your sessions as shown before, then open the file in a text editor and simply do a mass replace for the setting that you want changed. For example you could change this line

"Font"="Courier New"

to this line

"Font"="Lucida Console"

Then just re-import your sessions by double clicking the file. It will overwrite your current session information. That’s it.

Quick Tip: Fun with JavaScript Console in Browsers

Wed, 07 Jan 2015 00:00:00 +0000

All modern browsers nowadays have integrated development tools among which there’s a JavaScript console where you can type in some JavaScript and test your ideas, debug your code or see output from your script’s previous events. But the console is capable of so much more. Well, actually IE’s console is somewhat more limited than others, but the rest of them are pretty cool. To get an idea of the possibilities visit the chrome developer tips and tricks page to learn more. One of the fun and useful things you can do in a console is logging highlighted text. This can be achieve using the C-style formatting notation %c in logging statements e.g.

console.log("%clook at me", "background:yellow");

You can shape the text any way you want using familiar CSS styling. It’s possible to change the font family, font size, add some text decorations like underline, strike-trough or apply a shadow effect, just remember to separate them using a semicolon.

console.log("%cfunky text", "background:yellow; font-size: 20px; color: blue; text-shadow: red 0px 0px 6px;")

The only thing I was not able to do was using a custom font i.e. one that is not provided by the OS. I tried using @font-face to specify the external URL of the font, tried using @import to import external CSS. I even tried embedding the whole font as base64 encoded string to check whether it’s the @font-face that is not being processed or the problem lies with retrieving external resources. Since it didn’t work I assume that it’s ignoring everything you might put before the curly braces. In fact if you enclose your styling with curly braces – it won’t work. So you can modify the style only of the text that you’re outputting, not the whole console. If you know that it’s actually possible to use custom fonts I would like to know how that’s done, so please leave a comment.

Besides the practical there’s also a fun side to this. Even though it seems you can’t use external fonts, but you can use fonts with pictures in them that are provided by your OS like Wingdings. And you can leave your own signature in the console on your page or just a message for the curious ones that will peek at the console when visiting your page.

console.log("%ck", "font-family: Wingdings; font-size: 24px")

The snippet above would print a nicely styled ampersand symbol on Windows. But unfortunatelly different OSes include different fonts and users viewing your page from a Linux machine or a Mac would most likely not be able to see your message as you intended it. But there’s another thing you can do – images. Yes, you can use images in the console. Well, actually you can set the background to display an image and then put some blank characters so that it gets rendered correctly.

console.log("%c  ", "background-image: url('https://cloud.githubusercontent.com/assets/6187567/2544544/434705aa-b602-11e3-9a1c-6868c69bd4b8.png'); background-repeat: no-repeat")

But typically console lines that are meant for logging are only about 13 pixels in height, so sany larger image would be cut off and if you would try it with a larger image that has a white background it would seem that it just doesn’t work. There’s actually a very simple way to get around this, you just need to set the font size high enough so that your whole image would be displayed. And if you don’t want to use the full size of your image, you can even specify the background size so that it scales correctly.

console.log("%c  ", "background-image: url('http://i.imgur.com/r9nq6d2.png'); background-repeat: no-repeat; background-size: 88px 128px; font-size: 128px")

Also note that if you try to add some text after the whitespace you left for the image, it will appear quite large. If you want to add a message on the same line but with smaller text you can do this by adding a third string parameter with your text. The styles you specified will not be applied to it and so you’ll still be able to print out a message of decent length if you need to.

console.log("%c  ", "background-image: url('http://i.imgur.com/r9nq6d2.png'); background-repeat: no-repeat; background-size: 88px 128px; font-size: 128px;", "ho ho ho")

Oh and by the way, the animated GIFs do work and are pretty cool way to surprise someone :) Have fun and check out Chrome developer docs for more information.

Quick Tip: Easy Time Lapse Videos with FFmpeg

Thu, 31 Jan 2013 00:00:00 +0000

In this tip I want to share with you a quick and easy way of creating your own time lapse videos using your webcam and a tool called FFmpeg. I just recently discovered this command line tool myself and can think of various cool uses for it e.g. connecting a webcam to a Raspberry Pi and leaving it somewhere to take snapshots on certain events, or stream video of some place you want to monitor.

Getting the tool

First thing is first – download FFmpeg. On the download page you’ll find various different ways of getting this tool. You can clone a git repository, download source code snapshots or get the pre-built binaries for your platform. I’m using a 64-bit version of Windows 7 so I go to the FFmpeg Windows builds page and get the 64-bit build with all the libraries statically linked. If you’re not sure what to download just get this file and it should work on most Windows machines. Also note that you’ll need a program called 7-Zip to extract the downloaded archive.

Preparation

Now that you downloaded and extracted FFmpeg open up a command promt and go to the binary directory of the tool e.g. ffmpeg-win64-static\\bin. Let’s check what devices are supported by executing the following command:

ffmpeg -list_devices true -f dshow -i dummy

You should get a response of something like this

...
[dshow @ 00000000020977e0] DirectShow video devices
[dshow @ 00000000020977e0] "Microsoft LifeCam Cinema"
[dshow @ 00000000020977e0] DirectShow audio devices
[dshow @ 00000000020977e0] "Jack Mic (IDT High Definition A"
[dshow @ 00000000020977e0] "Desktop Microphone (2- Cinema -"
[dshow @ 00000000020977e0] "Dock Mic (IDT High Definition A"
[dshow @ 00000000020977e0] "Headset Microphone (GN 2000 USB"
[dshow @ 00000000020977e0] "Rec. Playback (IDT High Definit"

We’re interested in the video devices, so take note of the one that comes up for you, in my case it’s “Microsoft LifeCam Cinema”. Now let’s see what resolutions are supported on this camera:

ffmpeg -f dshow -list_options true -i video="Microsoft LifeCam Cinema"

Remember to replace the video parameter value with your own video device name. You should get a list of supported resolutions and framerates for your camera. Choose a resolution that you want and let’s test how it will look:

ffplay -f dshow -video_size 1280x720 -i video="Microsoft LifeCam Cinema"

Notice that this time I used ffplay instead of ffmpeg because I wanted it to open a window with a live video feed of the camera to make sure everything looks good and to be able to set up the camera where you want. Creating time lapse videos involves leaving the camera in the same place for long periods of time so you need to be sure that you’re getting the view you expect.

Creating the time lapse video

Now that everything is in place you can start capturing images. You can play around with the setting, but my suggestion is to first try recording 1 frame per second and leave camera untouched for an hour or so. Point it at something that has movement, but don’t put the camera to close to the subject or the images might be blurry from time to time due to the auto focusing. Start capturing:

ffmpeg -t 100000 -f vfwcap -s 1280x720 -i -r 1 -f image2 camera%06d.jpg

the -t 100000 part means that it will try to capture that many images before the program exits, but if at any time you change your mind simply click q while the ffmpeg window is in focus. The frame rate is set with the parameter -r 1. Also you may or may not get some errors about buffer being full, but this does not impact your capturing session. Though if that bothers you just add parameter -v quiet right after ffmpeg. The images will be captured with file names of camera000001.jpg through camera999999.jpg

After your done capturing the frames you need to convert those images to a movie like this:

ffmpeg -f image2 -i camera%06d.jpg time-lapse.mp4

Voila, you have created a time lapse video!

Here’s a sample of my first attempt:

Quick Tip: Velocity Template Debugging

Wed, 25 Apr 2012 00:00:00 +0000

In this post I’ll try to share my recent experience with [Velocity]() template debugging. There are a few ways to debug the templates depending on where you think the problem lies. But at times you have modify templates written by others and that’s when you need to do some exploring like finding out what variables are actually passed to the template. This is not always easily found in code, sometimes it’s simpler to print this stuff out when you’re already in the template. Below is a macro that I used to print out request and session attributes and parameters.

#macro (installDebugWindow)
<script language="javascript">
function debugWindow() {
  vc = window.open("", true, "width=800,height=600,resizable,scrollbars=yes");
  vc.document.write("<html><head><title>Velocity Debug Console</title>");
  vc.document.write('<style type="text/css">textarea{width:100%;} tr:nth-child(odd){background:#ddd;} tr:nth-child(even){background:#fafafa;} td:nth-child(odd){font-family:monospace;} tr:first-child{background:#ccc;} tr:first-child td{font-family:Arial;vertical-align:middle;} td h3{margin:10px;} table{margin-bottom:30px;border:2px solid #000;}</style>');
  vc.document.write("</head><body>");
  vc.document.write("<h2>Velocity Debug Console</h2>");

  vc.document.write("<table width='100%'>");
  vc.document.write("<tr bgcolor=#cccccc><td colspan='2'><h3>Request Attributes:</h3></td></tr>");
  #foreach ($name in $request.getAttributeNames())
    #printRow($name $request.getAttribute($name).toString().
      replaceAll("\n", "\\n").replaceAll("\r", " "))
  #end
  vc.document.write("</table>");

  vc.document.write("<table width='100%'>");
  vc.document.write("<tr bgcolor=#cccccc><td colspan='2'>" +
  	"<h3>Request Parameters:</h3></td></tr>");
  #foreach ($name in ${request.getParameterNames()})
    #printRow($name $request.getParameter($name).toString().
    	replaceAll("\n", "\\n").replaceAll("\r", " "))
  #end
  vc.document.write("</table>");

  vc.document.write("<table width='100%'>");
  vc.document.write("<tr bgcolor=#cccccc><td colspan='2'>" +
  	"<h3>Session Attributes:</h3></td></tr>");
  #foreach ($name in ${session.getAttributeNames()})
    #printRow($name $session.getAttribute($name).toString().
    	replaceAll("\n", "\\n").replaceAll("\r", " "))
  #end
  vc.document.write("</table>");

  vc.document.write("</body></html>");
  vc.document.close();
}

function parseKeys(e) {
  e = e || window.event;
  if (e.ctrlKey && e.keyCode === 48) debugWindow();
}

if (window.addEventListener) {
  window.addEventListener('keydown', parseKeys, false);
} else if (window.attachEvent) {
  window.attachEvent('onkeydown', parseKeys);
} else {
  window['onkeydown'] = parseKeys;
}
#end

It creates a new window and prints this data separate from your actual page layout, just make sure that your browser doesn’t kill this pop-up window. I placed the macro in my VM_global_library.vm file to be able to access it from any Velocity template. Then you can put the call to this macro #installDebugWindow() at the end of the page. This macro is based on the one I found on the Apache Wiki. I adjusted some things to better suit my taste. In the original version whenever you open a template that contains a call to #installDebugWindow a new window will instantly pop up. This is not always the desired behaviour. In my suggested version to get this window with all the template variable information you have to hit the key combination Ctrl+0. This doesn’t clutter your view with debug information when you don’t need and you have access to it whenever you do. Hope someone finds this useful.

Experimenting with WebGL

Wed, 28 Mar 2012 00:00:00 +0000

I was experimenting with WebGL the other day and it’s pretty damn cool. Here’s a spinning 3d logo of the company I currently work for as an attempt at animating something in 3D. If you’re not in the loop, WebGL is a JavasSript API supported by modern browsers that allows interactive 3D graphics rendering in a browser’s window, specifically the canvas tag. The graphics rendering is GPU accelerated which means excellent performance (as long as you’re using it wisely). The API is designed by the Khronos Group and is based on OpenGL ES 2.0. If you ever used OpenGL then it will be easier for you to start, but similar to OpenGL ES the WebGL API doesn’t have the immediate mode which is widely used in OpenGL tutorials. If you want learn WebGL you’ll need to get better acquainted with the concept of shaders which are lower level programs executed on the GPU. Depending on the type of shader, they’re executed for each vertex of the object or each pixel to be drawn. Those things are fast as hell. If you want a good place to start learning this stuff see Learning WebGL blog. But note that they may not explain all the nitty-gritty details, so you’ll probably end up searching for more information elsewhere. Another good resource is the official WebGL cheat sheet or reference card, whatever you wanna call it. But before you go and start programming something for your page, note that WebGL is not supported by all the current browsers and some visitors might not see the graphics at all. If you can’t see the spinning logo demo yourself then you could try switching browsers. Like using the latest version of Chrome or Firefox (these work for me). But chances are that your particular hardware might not be supported even by those browsers. I’ll finish up by leaving you with a link to some really cool looking WebGL demos. Enjoy!

SSH tunneling simplified (I think)

Mon, 26 Mar 2012 00:00:00 +0000

It’s been quite a while since my last post so I’ll try to ease back into blogging. For now I’ll just share a little something I created for myself to help with the SSH tunnels that I need to create on a seemingly regular basis. Sometimes I forget or mix up the parameters that need to be passed to SSH client, especially when interconnecting these tunnels. In this simple tool you can enter the missing info to form a logical sentence in English language and when you click the generate button you get a command line for the SSH client. Pretty user friendly ;) And if you’re not sure what SSH tunnels are and why you need them, take a look at this post on SSH hacks.

A Story of Switching from Java to Developing for Mac OS X

Thu, 28 Jul 2011 00:00:00 +0000

A couple of months ago I’ve switched from Java development to developing for Mac OS X. Let me tell you, it’s a completely different world, but that was to be expected. The first thing is to get acquainted with the OS and it’s features. That actually didn’t take too long. I’m really pleased with the availability of a full-fledged UNIX console in the OS, so I’m able to leverage all of my linux knowledge. The eye candy and the elegant design of most of the applications really give you quite a difference experience of using the computer than it would be on another OS. And at the end of the day, I guess, that is the most important factor for most users. Feature-wise Mac OS X has all the benefits of *BSD systems, being a derivative of FreeBSD and all. Over the years I’ve been using different OSes and for a long time now, I’ve not seen many major issues in any of them. I would suggest that such things like stability are not really a factor for choosing an OS nowadays. All of the OSes struggle to be as stable as they can be and the result is reasonable. I’d say that neither Mac OS X nor linux nor any other OS is the silver bullet, but some are better than others. I had experienced a application crashes in every OS I worked with. No matter how good the OS is you still need your day-to-day software to be well written and stable. Sometimes it’s a matter of configuration and sometimes you’re just at the mercy of the application you’re using. When switching to a Mac you cannot overlook the sleek design of Mac OS X which I believe does lure a lot of users. I guess, I would agree, that this sort of design looks more profesional and makes a different impression on someone if say, you would be presenting something to your client vs. showing them something on Windows Vista.

Let’s talk about the development for Mac OS X. First of all you get familiar with the de facto IDE to be used – Xcode and the Objective C language. Not so long ago Apple released Xcode version 4 and it’s a complete redesign of the previous version. The new version is not exactly loved by all of the developers who got used to the previous version. Being new to this, I didn’t take up getting to know version 3 of Xcode, since everyone will have to switch to the newer version anyway. The development experience is quite different from that which I’m used to when I was developing using Java. And it’s not just the language itself. I kind of got used to all the features and the little things of the IntelliJ IDEA which I used before. Even in the Java world it’s an IDE that’s tough (if possible) to beat. I’m not affiliated with JetBrains in any way, but I feel really strongly about this and I recommend with complete confidence this IDE to anyone doing Java development. On the Xcode side of things you get an uncluttered window for writing code and a couple of side panes if you choose to show them. Nothing more, nothing less. The good thing is that this lets you concentrate more on the code at hand without any distractions. The not so good things are the refactoring options of which there are few and some of them like renaming the variables has bugs. Also I don’t particularly like the debugger which is of little help when you’re trying to inspect some of the variables and expression evaluation isn’t so great. It’s when you switch from one language to another and especially when switching environments, you begin to see how the things that you’re used to can be implemented completely differently. It’s always good to learn something new, but there are times, when you’re longing to have the features that you’re used to. And JetBrains come to the rescue once again with their new product – AppCode. It’s still in a pre-release stage, but I’m hoping it will evolve into a great product. I have faith in it because I’ve seen what they can do for Java development with IntelliJ IDEA. You can also take a look at this post about AppCode which also supports the idea that Xcode shouldn’t be the only choice for developing for Mac OS X and highlights the features of AppCode. These are my sentiments so far regarding the Mac OS X IDEs, I encourage you to try out AppCode and report any bugs to JetBrains to help them improve their product.

Introductory PhoneGap Tutorial for Android

Wed, 27 Jul 2011 00:00:00 +0000

I haven’t posted in quite a while. Been busy with all the job switching, getting used to Mac things and such. On another topic while cleaning up my hard drive I’m finding various neat stuff, that I forgot about. Will try and share some of it with the rest of you. These are mostly things that you either need or don’t care about at all. One of those things is a screencast that I’ve forgotten I made. It was created for a competition hosted by NetTuts+ and touches on some of the basics of PhoneGap platform. I didn’t actually win and now it’s a shame I remembered it only a few months later, but perhaps it could still be useful to someone.

Automatic Web Server Monitoring

Tue, 15 Feb 2011 00:00:00 +0000

Hi there! Here’s another quick suggestion for all of you out there who have to deal with a lot of web server environments and need to make sure that they’re all alive and responsive. Sure there are all sorts of tools for server monitoring and performance tracking, but I found that most of the time you just need to know if the server is up & running. And of course you should be notified if it’s not. Here’s my short Python script that does just that. You will need to tweak it a little to use in your environment. First of all I’ve set the content threshold to 2000 Bytes which means that the expected length of the page content returned is no less than 2000 Bytes. If it’s below this value, one could assume that something’s fishy going on. You might want to change this value to something else depending on what your web server returns. Another thing that you’ll want to change is the servers dictionary in the main() method. The logic behind this is that each of the web servers is assigned a responsible person that you would need to inform if something is wrong with the server. You should list all your servers that need to be monitored and the responsible person’s e-mails. The final change is on the line where you instantiate the HeartbeatMonitor class. The first parameter admins is a list of e-mails of the people who will receive the notification if any of the servers are down. You can leave this list empty if you want, but I usually put my e-mail in there since I like to be in the loop. The second parameter mailServer is the SMTP server which will be used to send notifications so make sure this one is set correctly. This parameter is actually optional and localhost will be used as the default mail server if you decide to skip it.

#!/usr/bin/env python

import smtplib
import urllib2

"""
This script helps monitoring multiple servers and notifies
via email when something is wrong.
"""

__version__ = "0.1"
__author__ = "Andrius Miasnikovas"
__license__ = "psf"

class HeartbeatMonitor(object):
    admins = []
    mailServer = None
    threshold = 2000

    def __init__(self, admins, mailServer='localhost'):
        super(HeartbeatMonitor, self).__init__()
        self.mailServer = smtplib.SMTP(mailServer)
        self.mailServer.set_debuglevel()
        self.admins = admins

    def sendEmail(self, recipient, content):
        sender = 'noreply@hmonitor.net'
        print '-- Sending email message: ' + content
        recipient = recipient.strip()
        if len(recipient) > and not self.admins.__contains__(recipient):
            self.admins.append(recipient)
        message = "From: %s\\r\\nTo: %s\\r\\nSubject: Warning: there's a problem with the server\\r\\n\\r\\n%s" \\
                  % (sender, ", ".join(self.admins), content)
        self.mailServer.sendmail(sender, self.admins, message)
        self.mailServer.quit()

    def check(self, servers):
        for (server, admin) in servers.items():
            try:
                h = urllib2.HTTPHandler(debuglevel=)
                opener = urllib2.build_opener(h)
                print server, "-",
                page = opener.open(server)
                data = page.read()
                if page.code != 200 or len(data) < self.threshold:
                    print "UNKNOWN"
                    self.sendEmail(admin, 'Server '+server+' is responding, but state undetermined.')
                else:
                    print "UP"
            except:
                print "DOWN"
                self.sendEmail(admin, 'Server '+server+' is not responding')

def main():
    servers = {
            'https://server1.company.com'         : 'person.one@company.com',
            'http://server2.company.com/web/app'  : 'person.two@company.com',
    }

    hMonitor = HeartbeatMonitor(['admin@company.com'], 'smtp.company.com')
    hMonitor.check(servers)

if __name__ == "__main__":
    main()

After you make your alterations go ahead and run the script a couple of times to make sure it’s working. Then if you’re running it on a Linux box you could automate the process by adding it to the cron scheduler list. Simply run the following command

~$ crontab -e

and add a line similar to this

*/30 * * * * /usr/local/bin/hmonitor.py

Where /usr/local/bin/hmonitor.py is the absolute path to this monitoring script and the part in front of it is the time when this script should be run. In this specific case the script will be run every 30 minutes, but you can change it to your liking. If you need more information on the crontab format which is used by the cron scheduler see the wiki page.

Search for Files on Remote FTP Server

Mon, 14 Feb 2011 00:00:00 +0000

I’ll keep this one short. Don’t know about you, but I’ve always been frustrated about how most FTP clients won’t let you search for files on a remote FTP server. I remember I used some client that had this functionality, but this was quite a while ago so I can’t even remember the name of it. If someone else knows any lightweight tools that let you do that, please leave a comment. Recently I needed to do some quick searches on the servers that I have only FTP access to. And you know what they say… “when you can’t find the right tool for the job – write it yourself” Actually I don’t know if anyone says that :) but I decided to do it and put together a Python script just for this task. I’m not sure if many others have run into a similar problem, but I put it on GitHub anyway. I called it FTP search for obvious reasons. You can find the script and a short documentation there. The only prerequisite is that you have Python 2.7.x installed. I can’t promise you that it will work with Python 3.x but you’re welcome to try. If you have any ideas on how to improve this little tool and wish to contribute then leave a comment bellow or contact me via e-mail that’s specified on GitHub’s project page.

Modifying Treepie To Display File Count

Tue, 25 Jan 2011 00:00:00 +0000

I’ve already talked a bit about managing large amounts of files in my post Disk efficiency when dealing with tons of small files so you can kind of tell that I’m a proponent of having as little files on disk as possible (but not less). In my everyday tasks I’m using this program called Everything by David Carpenter and I suggest you check it out if you don’t know it yet. The concept is simple, but the consequences are immense – the program indexes all the files and folders on all volumes connected to your computer i.e. your hard disk drives, flash drives, cd/dvd-roms, etc. And it tracks the changes so this index is always up-to-date allowing you very fast access to ANY FILE on your computer. The thing is that it indexes all the files on startup so the more files you have the longer it takes. To illustrate another reason why too many files aren’t great see the picture below.

Notice how the actual size of the files is about 806KB while because there are so many files (4422 to be exact) and the cluster size for my disk is 4KB the actual space occupied on the drive totals up to almost 18MB. We can do the math to better understand what happened:

cluster size: 4KB = 4096 Bytes

number of files: 4422

so even if each file contained only a single character i.e. file size of 1 Byte it would occupy the whole cluster and take up 4KB of disk space. Which means that the absolute minimum of disk space consumed by these files is 4422 x 4096 = 18112512 Bytes ~= 17.27 MB

And of course if the files are larger they will take up even more space. Hopefully I convinced you that you should remove any and all unnecessary files that you’re not using and don’t just let them hinder your drive’s performance. This is the part where we have to find a way to locate directories containing the most files so that we could decide whether we actually need them or not. In the past I’ve been using a program called TreePie to find which directories occupy the most space on disk. It looks something like this:

Unfortunately it displays the folders proportional only to the space occupied on disk. On the other hand the project is open source so I can modify it’s behavior and so I did. Here’s the patch file with my changes.

Index: AllDir.cpp
===================================================================
--- AllDir.cpp  (revision 16)
+++ AllDir.cpp  (working copy)
@@ -64,7 +64,7 @@
 {
    HANDLE hFirst;
    WIN32_FIND_DATA hFind;
-   INT64 nSize =;
+   INT64 nSize = -2;

    char sCompleteFile[MAX_PATH];
    char sListFile[MAX_PATH];
@@ -131,7 +131,7 @@
            if (hFind.nFileSizeHigh!=-858993460 ) //// ARRRGH!(system information folder)
            {
                //grandezza del file (aritmetica a 64 bit)
-               nSize += (INT64)hFind.nFileSizeHigh * ((INT64)MAXDWORD+1) + hFind.nFileSizeLow ;
+               nSize ++;
                //if (nSize > MAXDWORD){
                //  mb_log("ListAllDir: nSize sorpassa MAXDWORD");
                //}
Index: displayer.cpp
===================================================================
--- displayer.cpp   (revision 16)
+++ displayer.cpp   (working copy)
@@ -62,11 +62,7 @@
                glLoadIdentity();

                glRasterPos2f(5.0f , rect.bottom - 20.0f );
-               if (iv_nSize > 1073741824 )
-                   glPrint("%d GB  ",iv_nSize/1073741824);
-               else
-                   glPrint("%d MB  ",iv_nSize/1048576);
-               glPrint("(%d KB  )",iv_nSize/1024);
+               glPrint("%d files",iv_nSize);

                glRasterPos2f(rect.right/2.0f , rect.bottom - 20.0f );
                glPrint(sBuffer);

Pretty simple, huh? These changes are enough to get TreePie to start displaying the file count instead of the file size. Though I have to admit that I was unable to open and compile it with Microsoft’s Visual Studio Express. I’m still not sure if it was something with included project files or was it something with my machine’s configuration. I solved it by using an IDE that I have more experience with – CodeBlocks. I created a new project, copied over the files and soon I had my own modified version of TreePie. Which I’m happy to share with you all. Here’s an archive containing the modified Treepie version as a CodeBlocks project and includes a compiled version if you don’t have this IDE or don’t want to install it. I’m considering forking this project, cleaning up the code a little bit and maybe throwing in a few new features.

Online Tools That Make My Life Easier

Thu, 25 Nov 2010 00:00:00 +0000

Imagine a situation where you’re sitting in front of a colleague’s computer or one that is publicly used. By definition you’re missing all your favorite tools that you’re used to and you can’t install them even if you wanted to. Here’s a short list of online tools that help me a lot when I’m using a computer other than my own. These do not fully replace the desktop equivalents, but they’re good enough. Though it’s not a list of tools that replace desktop applications, but rather a list of online tools that complement them and some provide functionality that does not exist in desktop based applications.

HTML / Javascript

http://jsbin.com – this one allows you to write HTML and Javascript code directly in the browser and preview it instantly. Features a nice syntax highlighting.
http://zen-coding.ru/demo/ – this one is really nice. If you don’t have VIM handy or other editor that supports zen-coding, you can simply open up this page and start zen coding right inside the browser. And if you need some help with your zen here’s a cheat-sheet PDF and a nice post explaining what it is and how to take advantage of it.

Beautifiers

http://jsbeautifier.org – Javascript unpacker & beautifier. Does exactly what it says.
http://xmlbeautifier.com/default.aspx – simple XML beautifier, nothing fancy.

Regural Expressions

http://gskinner.com/RegExr/ – regular expression test page with possibility to save your expressions and a hint column if you forget something.

Data Processing

http://www.miraclesalad.com/webtools/md5.php – MD5 hash generator.
http://meyerweb.com/eric/tools/dencoder/ – URLencode and URLdecode any string using this page.
Base64 encoder and Base64 decoder.
http://www.stringfunction.com/ – various functions that can be applied to strings.

Network

http://networking.ringofsaturn.com/Tools/ – a set of different network related tools, that help you with your subnet calculations, tracerouting, domain resolution, probing and such.

Others

http://developer.yahoo.com/yql/console/ – that’s Yahoo’s YQL console which is way awesome. Being the page scraping enthusiast that I am, I really like how you can gather information from different sources in such a simple way, also note the REST API and the ability to save your queries.
http://hurl.it/ – not quite short how to explain it, but replaces tools like cURL and Wget for me. Basically does a request to specified ULR and returns the result.
http://hpricot.com/ – this one can be used for specific information extraction from any page a.k.a. Web scrapping.

Communication

http://webchat.freenode.net/ – a web interface for freenode IRC server. Really helpful in situations where you find yourself behind some firewall that restricts outbound access.
http://02.chat.mibbit.com/ – similarly to the above this is a web-based IRC client. The main difference it is not just for the freenode server.

This list is not meant to be exhaustive, but if you know some nice online tools and are willing to share them, leave a comment.

Private Key PFX to/from JKS Conversion Using OpenSSL and Jetty

Thu, 21 Oct 2010 00:00:00 +0000

Recently I’ve been watching quite a few screencasts which seem to be a really fun way to learn something. This made me want to create something of my own. So for this blog post I’m putting up my first attempts at creating screencasts while trying to explain something useful.

When dealing with PKI based application security, one usually encounters different systems on different platforms and making sure they can interact with each other can be quite a hassle. Testing these systems usually means you create you own self-signed private/public key pairs. And of course there are times when for testing purposes you need to convert your private key to another format because you generated it on a different platform or received it from someone else who didn’t ask you about your preferred private key storage format. There are quite a few storage formats devised by the public-key cryptography standards group. Most of the time I have to deal with keys in Java’s JKS format and PFX format used on Microsoft platforms so from time to time I have a need of converting one format into another. See the screencasts bellow of how I’m converting them both ways.

Convert private key in PFX format to JKS keystore

Note: after this conversion the alias in the Java keystore for the converted key is ‘1’. Since I’m using such conversions only for test purposes I haven’t looked into changing the alias to something more meaningful. But if you know a quick way of doing this without reimporting the key, please, post your method in the comments ;)

Convert private key in JKS keystore to PFX format

Here’s the source of the Java file I used in the screencast for private key an certificate extraction from JKS keystore. Note that I’m assuming here the usage of JDK 6 because System.console().readPassword() is used to read the password without echoing.

import java.io.*;
import java.security.*;
import java.security.cert.Certificate;

public class ExportKeyAndCert {

  private static class Base64 {
    static final byte[] encodeData;
    static final String charSet = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";

    static {
      encodeData = new byte[64];
      for (int i = ; i < 64; i++) {
        byte c = (byte) charSet.charAt(i);
        encodeData[i] = c;
      }
    }

    private Base64() {}

    public static String encode(byte[] src) {
      int length = src.length;
      byte[] dst = new byte[(length + 2) / 3 * 4 + length / 57];
      int x = , len = , old = , state = , dstIndex = ;

      for (int srcIndex = ; srcIndex < length; srcIndex++) {
        x = src[srcIndex];
        switch (++state) {
          case 1:
          dst[dstIndex++] = encodeData[(x >> 2) & 0x3f];
          break;
          case 2:
          dst[dstIndex++] = encodeData[((old << 4) & 0x30) | ((x >> 4) & 0xf)];
          break;
          case 3:
          dst[dstIndex++] = encodeData[((old << 2) & 0x3C) | ((x >> 6) & 0x3)];
          dst[dstIndex++] = encodeData[x & 0x3F];
          state = ;
          break;
        }
        old = x;
        if (++len >= 57) {
          dst[dstIndex++] = (byte) '\n';
          len = ;
        }
      }
      switch (state) {
        case 1:
        dst[dstIndex++] = encodeData[(old << 4) & 0x30];
        dst[dstIndex++] = (byte) '=';
        dst[dstIndex++] = (byte) '=';
        break;
        case 2:
        dst[dstIndex++] = encodeData[(old << 2) & 0x3c];
        dst[dstIndex++] = (byte) '=';
        break;
      }
      return new String(dst);
    }
  }

  static public void main(String[] args) {
    if (args.length < 2) {
      System.out.println("Usage: java ExportKeyAndCert keyStore keyAlias");
      System.exit();
    }
    try {
      String keyStore = args[0];
      String keyAlias = args[1];
      Console console = System.console();
      char[] storePass = console.readPassword("Enter keystore password: ");
      char[] keyPass = console.readPassword("Enter key password: ");

      KeyStore ks = KeyStore.getInstance("jks");
      ks.load(new FileInputStream(keyStore), storePass);

      Certificate cert = ks.getCertificate(keyAlias);
      String b64 = Base64.encode(cert.getEncoded());

      PrintWriter fout = new PrintWriter("public.cer");
      fout.println("-----BEGIN CERTIFICATE-----");
      fout.println(b64);
      fout.println("-----END CERTIFICATE-----");
      fout.close();

      Key key = ks.getKey(keyAlias, keyPass);
      b64 = Base64.encode(key.getEncoded());

      fout = new PrintWriter("private.key");
      fout.println("-----BEGIN PRIVATE KEY-----");
      fout.println(b64);
      fout.println("-----END PRIVATE KEY-----");
      fout.close();

    } catch (Exception e) {
      e.printStackTrace();
    }
  }
}

What’s New in Tomcat 7

Wed, 21 Jul 2010 00:00:00 +0000

Recently I watched this webinar about Tomcat 7 presented by Mark Thomas and would like to share my thoughts on the subject and what I’ve learned.

Servlet 3.0

The most prominent change is the support of Servlet 3.0 specification which supports such great features as

asynchronous servlets – not used by default
web-fragment.xml – you can specify parts of the configuration in your libraries which you plan to reuse

The effective deployment descriptor (web.xml) now is the sum total of all the fragment configurations contained in the application’s libraries. This means that you need to be more careful when using 3rd party libraries because they can contain some configuration portion that will prevent your application from working like you expected. But to my understanding you can fixate the effective web.xml used and prevent tomcat from scanning all the libraries for configuration fragments. Now deployment descriptors support programmatic configuration i.e. you can decide how to configure an application at startup by considering the environment the server is in and other variables.

Another interesting thing to note is that before the Servlet 3.0 specification implementation tomcat used two methods of session tracking: cookies and rewriting the URL with the JSESSIONID parameter. The latter in tomcat was mandatory meaning you could not easily switch it off. I actually tried some workarounds, but they produced some side-effects. If anyone knows a proper method to avoid ever seeing the JSESSIONID parameter in Tomcat 6 please let me know. In Tomcat 7 the URL rewriting method is no longer mandatory and a new session tracking method was added based on SSL session. It is more secure, but I believe it needs two-way SSL configured. Don’t quote me on this, because I might be wrong about the two-way SSL, will have to check it. While we’re at it I can mention that now there’s the ability to change the session cookie’s default name, domain and path.

<session-config>
   <tracking-mode>COOKIE</tracking-mode>
</session-config>

JSP 2.2

Not much to talk about other than saying that Tomcat 7 supports it. And maybe worth mentioning that you can now specify the defaults for the content type you’re returning and the buffer size. Also there’s a nice configuration option added called error-on-undeclared-namespace which quoting the documentation does the following:

The default behavior when a tag with unknown namespace is used in a JSP page (regular syntax) is to silently ignore it. If set to true, then an error must be raised during the translation time when an undeclared tag is used in a JSP page. Disabled (false) by default.

And here’s an example of how you would use this option

<jsp-config>
    <jsp-property-group>
        <url-pattern>*.jsp</url-pattern>
        <error-on-undeclared-namespace>true</error-on-undeclared-namespace>
    </jsp-property-group>
</jsp-config>

Also note that Tomcat 7 implements EL 2.2 and now is much stricter about the EL specifications, but according to Mark it’s unlikely to affect many users unless of course you’re doing some kind of hack with it.

Memory leak protection

This is a nice new feature in Tomcat 7 which has been backported to Tomcat 6 and if you’re using version 6.0.24 or later, you can enjoy the benefits. It’s implemented as a listener which tries to detect and fix the possible memory leaks whenever it can. These are the leak cases that it tries to prevent:

JDBC drivers – they should be de-registered upon application stop
ThreadLocals – if you create it when processing the request, you must set it to null before completing the response
Threads – if a webapp starts a thread (e.g. scheduler) it must stop it, Tomcat doesn’t do that for you because it might crash the JVM, but it does report the problem in the log files
ResouceBundle – Tomcat automatically fixes this one, but if you want some details turn on debug logging

Alias support

This is said to be a much requested feature which allows you to map a .war file or a directory with static content that you could for example share among other applications deployed on that server. The main advantage over using symlinks is that when using a symlink Tomcat follows it and undeploys the content along with the application. This is done by specifying the attribute alias in the tag that looks something like this alias="/aliasPath=docBase" and remember to use the absolute path when specifying the content location.

Even more changes

There are more changes like the ones in the manager application or the jdbc-pool library, but I won’t try to review them here. Other ones include embedding improvements which allow you to easily embed Tomcat using the class org.apache.catalina.startup.Tomcat and then do programmatic configuration of the server. This seems extremely useful for things like integration tests and in fact most of Tomcat 7 unit tests use embeded Tomcat. Tomcat 7 prevents session fixation attacks by default by changing session ID on authentication. Internal changes include switching from Valves to Filters. This change is ongoing and should eventually replace all Valves. And one thing you might notice when running the new Tomcat is that access log valve is enabled by default. This means that you’ll get another file in your logs directory which will log all access to Tomcat server. It’s formatted just like the Apache HTTP server’s access log and is useful in a number of situations.

I am using Tomcat 6 and 7 side by side for my current needs and can say that Tomcat 7 with all it’s nice features does contain a few bugs so don’t rush into using it for any environments that need stability. I’m still playing with the new Tomcat’s features and maybe will write another post regarding that.

UPDATE (2011-07-30)

Since this post is still relevant and gets a reasonable amount of traffic I decided to update it by putting a link to Mark Thomas’s recent presentation on What’s New in Tomcat 7, so that you would get the information straight from the source. The sound quality isn’t that great, but it’s worth watching/listening if you’re interested and have a half an hour to spare.