In my second post about Kayako, I would like to show what we managed to achieve. We posted our list of requirements for a help desk system in the previous post. I must admit that the setup took a lot of time we discovered a few bugs in Kayako along the way. Some of them were quite serious. The rest can be classified as defects as it looked like some functionality which was added on top of the existing system and implementation was not well thought through by Kayako.

Almost all errors were fixed by Kayako developers soon enough. However, we had one error which took us two weeks to fix. We exchanged emails with Kayako until they agreed to provide us with their open source code. From that point we tracked the bug within an hour. After setting up Kayako, as we have been making web applications for quite a long time now, our hands itched to make our own simple help desk system. If you are like us, you will understand, or at least will agree with us upon closer acquaintance with Kayako.

In the process of setting up Kayako I wrote as much as 126 posts in their forum. Almost all of them were answered by a wonderful man, a forum guru – Gary, who, incidentally, is not a Kayako employee. Gary had no avatar on the forum and I, being impressed by his experience and desire to help, offered him an avatar, which he gladly accepted.

We were able to achieve our goal. We have one support system that is used for everything we do.

1. Working with clients

Each client has his own Kayako portal.

Fig. 1 Client’s Kayako Help Desk portal

As you can see an unauthorized user can only access the login page. Moreover, the portal is closed with htaccess directly from Kayako (they have such an option). Users of such client’s portals are created by us in advance. The number of users is small and constant for each client. We created special mailboxes CLIENT_A@support.deepshiftlabs.com, served by GMail. Kayako will not accept mail from unregistered users in these portals. The number of such portals is not limited by the Kayako license. Limitations – all the portals will be on the same URL – http://support.deepshiftlabs.com. The only difference is in the parameter / index.php?/CLIENT_A. Ideally we would like to be able to create portals, as http://CLIENT_A.deepshiftlabs.com and suggest Kayako to find ways to license this scheme.

2. Integration with our web service

We have a web service – Nerrvana.

Fig. 2 Our web service help desk portal

Here and below, links from screen shots lead to real pages of our Kayako installation. For some reason Kayako developers decided that one portal user will never visit another one and they decided to keep a reference to a portal template in a cookie. This means that by visiting our portals it will seem that they have broken styles. If you would knew how it hurt during Kayako customisation, especially before we realized that the problem is not in our settings, but in the cookie you need to wipe every time before visiting a new Kayako portal, for example, from Firebug. Another way would be to use different browsers during the configuration phase.

Here you see a portal with all the options available – news, sign up, create a new ticket. In here we do not use GMail and handle incoming emails ourselves. We do not even use a spam filter and Kayako reads incoming emails using IMAP. The highlight of this portal is that it is fully integrated with Nerrvana. To begin with, we have implemented a way for Nerrvana users to create help desk tickets without leaving Nerrvana’s UI.

Fig. 3 Integration with Kayako in Nerrvana

Kayako’s API allows us to create a new ticket, and using API, add to the ticket all the necessary information about the problem that would help to resolve it faster.

But not everything is as it should be, and the problem was to synchronise user passwords. Why do we need this synchronisation? We want to provide excellent user experience. What could be more convenient than having a single password in Nerrvana and in Help Desk? To understand the problem we faced let’s consider a few scenarios.

The potential client (let’s call him “A”) emails the issue to contact@nerrvana.com. Kayako creates a ticket and a new account. Further, “A” registers in Nerrvana (we assume he uses the same email) and different password. Well, we can change it in Kayako through the API. But what if “A” will go to the Kayako portal and will change the password there?

Fig. 4 Kayako change password form

We cannot disable changing a password in Kayako because the portal is used not only by customers but also by potential customers who do not have Nerrvana account. In addition all portal visitors have access to “Forgot Password” in Kayako, which again allows you to change your password. That is we can synchronise the password changes with Kayako, but not vice versa.

Fig. 5 Kayako password recovery form

I described the problem in Kayako’s forum, hoping that we are not the first who ran into it and luckily, received the answer that satisfied us. It remained only to sign an NDA and get the code of two Kayako classes participating in resetting and changing a password. After that, we created a secret Nerrvana page, which we call after Kayako has finished the password changing operation. This page checks whether this user is also a Nerrvana user and if so – changes the password in Nerrvana.

As a result we have the following working scheme:

- User registers in Nerrvana and does not have Kayako account yet. Nerrvana creates an account as well as an account in Kayako via API, but without sending an e-mail from Kayako. Our email already contains instructions on accessing the Help Desk. Why bombard a new client with two e-mails? Kayako’s API has an option to supress registration email when an account is created via API.

- User registers in Nerrvana while having a previously created Kayako account. Nerrvana creates an account and changes his password in Kayako through the API to match his Nerrvana password.

- The potential client asks a question while not having a Kayako account. Kayako creates his account and sends him his user name and password by e-mail.

- The potential client asks a question while having a Kayako account. Nothing happens. Kayako simply creates a new ticket.

- The user changes the password in Nerrvana or resets forgotten one. We change the password via the API in Kayako too.

- Nerrvana’s user changes or resets password in Kayako. We use our secret script which we injected into Kayako’s code to change his password in Nerrvana.

3. Using Kayako to support software users buy, download and install

We created the first web application in the Startyco set – Answers, which you can buy, download and install. Again, we use the Kayako portal to support our customers.

Fig. 6 Kayako portal to support downloadable web application
(gradient in the header looks pretty ugly on a screenshot)

When you buy our application, we register the purchase in a special small (just 5 tables) database. We record the license granted, its type (trial or permanent and which domain it is issued for), client details. If the account does not exist yet in Kayako, we create it. Accounts can be created before purchase if a potential customer sent an email to us asking some questions. We use the Kayako database to authenticate clients on the site of our product – answers.starty.co, and tables in our own small database to determine purchases and new product upgrades available for download.

There are two possible scenarios. The first – a customer asked a few questions, Kayako created an account automatically and sent an email with account details. We answered his questions, and now, if a customer buys a product he will be able additionally login to answers.starty.co with his Kayako account. On purchase we simply add data we need to our small database I mentioned. The second – a customer buys the product or downloads a demo version. If account was not found in Kayako we create it through the API and at the same time record user information in our database.

When you create a new user via Kayako’s API, you can specify whether to send him an email with account details or not. A password is required for viewing and creating tickets from the web interface in Kayako. In this case, unlike Nerrvana, sending an email is appropriate. A user loads the product and at the same time receives an email from Kayako with access details to our help desk system. In Nerrvana we instruct Kayako via API not to send our own email to the newly registered user, because we want him to have only one email from Nerrvana with all info needed. So if you sell a downloadable product (not a web service) you can simply add your own tables into your database, register new customers directly in Kayako and add additional information in your own database.

Finally, in case if someone wanders into a default portal (the one with no parameters after URL), we have completely changed it by putting more buttons on it, redirecting to the portals of Nerrvana and Startyco. By the way, look at favicons of a client, Nerrvana, Startyco and default portals on screenshots above – we changed them too.

Fig.7 Default Help Desk URL has no portals but links to them

In my next post I try to give step by step setup instructions for the portals we use. Hopefully this will save you some time.

Let’s begin by creating a new virtual machine.

Fig. 1 Launch new virtual machine wizard

Fig. 2 Select Workstation 8.0

Fig. 3 Point to loaded netinstall image

Fig. 4 I install CentOS 6.2 64 bit

Fig. 5 Select location for VM files

Fig. 6 I have 4 cores on i7 so I give 2 of them to VM

Fig. 7 I have 16Gb RAM and give 2Gb to CentOS VM

Fig. 8 I use static IP’s and want to use bridged networking

Fig. 9 Accept defaults

Fig. 10 Accept defaults

Fig. 11 Accept defaults

Fig. 12 With minimal install VM file will be less than 1G

Fig. 13 Accept defaults

Fig. 14 VM setup is completed

Fig. 15 Press Tab here to plug kickstart in

Now we have to say a few words about kickstart. There are plenty of options for ‘ks’ parameter. I used web way. This is where my kickstart was but Vadym, who created it for me, may delete it later so I put in in here.

text
skipx
install
url --url http://mirror.cogentco.com/pub/linux/centos/6/os/x86_64/
repo --name=epel --baseurl=http://download.fedoraproject.org/pub/epel/6/x86_64/
repo --name=updates --baseurl=http://mirror.cogentco.com/pub/linux/centos/6/updates/x86_64/
lang en_US.UTF-8
keyboard us
rootpw 123456
firewall --disabled
authconfig --enableshadow --passalgo=sha512
selinux --disabled
timezone Etc/UTC
bootloader --location=mbr
zerombr
clearpart --all --initlabel
part /boot --fstype ext4 --fsoptions="noatime" --size=200
part pv.1 --size 1 --grow
volgroup vg0 --pesize=4096 pv.1
logvol / --fstype ext4 --fsoptions="noatime" --name=root --vgname=vg0 --size=8192
logvol swap --fstype swap --name=swap --vgname=vg0 --size 2048
logvol /var --fstype ext4 --fsoptions="noatime,nodev" --name=var --vgname=vg0 --size=4096
logvol /home --fstype ext4 --fsoptions="noatime,nodev" --name=home --vgname=vg0 --size=2048
 
services --enabled=sendmail,network,ntpd,ntpdate
 
reboot
 
%packages --nobase
epel-release
openssh-clients
openssh-server
yum
at
acpid
vixie-cron
cronie-noanacron
crontabs
logrotate
ntp
ntpdate
tmpwatch
rsync
sendmail
mailx
which
wget
-postfix
-prelink
-selinux-policy-targeted
%end

I do not want to go into true minimal install discussion as I trust Vadym Let’s say it is one of the minimal install kickstarts. You can get the most benefits from kickstarts if you are going to install many systems or move them around. We extensively use kickstarts in Nerrvana under Vadym’s guidance. Our core, FTPS servers, Selenium Hub and CentOS RC VMs are all kickstarted. We can move them to another hoster pretty easily and our kickstarts reflect all packages installed as well as our own RPM’s and customisations we need for each system.

In this case we are not going to reinstall or move a system – we just want to start it and not waste time clicking around selecting packages and creating file systems. Basically we want to save time. In future posts we may share different tips of our real kickstarts as well as the way we create Windows based Xen VMs with different browsers for Nerrvana. We simply want to share what we learned while building Nerrvana so others can benefit creating their own Selenum testing systems.

Back to our topic. Put your kickstart file to make it available via HTTP. You can simply start IIS and add this file to default IIS home directoty, for example.

Fig. 16 Add kickstart parameters and hit Enter

Fig. 17 Installation starts

Fig. 18 ‘Asknetwork’ param at work

Fig. 19 Enter network details

Fig. 20 Installation resumes

Fig. 21 Use Alt+F3 to see logs while installation goes on

Fig. 22 OS and packages from kickstart installed

Fig. 23 All done. Now install what you want.

Fig. 24 Takes 725Mb of disk space

Now you can login as root/123456, change password, set sshd, disable root login via ssh, create an account for youself, install MySQL, PHP etc.

In the design of the system’s site we used loadable fonts and of course we would not mind using them in the emails. However, according to Campaign Monitor, in recent years nothing has changed – external fonts cannot be used in emails. Ok, we will take this as a given and will only use CSS and standard fonts. Googling this topic brings a tip on HTML coding for emails from MailChimp where they suggest to use inline CSS instructions, i.e. include the styles directly in the html tags. I will illustrate, using GMail as an example.

We create a simple email that is sent to the newly registered users. Here’s how it looks in FireFox; pretty good.

Now I have sent it to my GMail account and let’s look how it renders.

Let’s play a game of “Find the 10 differences.” What do we see? Both size and font type are different. Why? When we create HTML page layout, we rely on the inheritance of the styles. Nested styles inherit their parents properties and for some reason Firefox calculated the style of the font size for the table cells in 16 pixels. In the case with mail services all is more complex. Forget about inheritance – external (GMail’s ones in this case) styles can override your inheritance at any time, as can be seen on the screenshots. Here is the piece of code we are interested in.

HTML in Firebug …

… and his current styles

Here the font size was calculated correctly, 22 pixels were inherited from parent’s div

In this example, as you understand, an HTML table is used to show a user name and password. Yes, do not throw spoiled vegetables at me. If you read the MailChimp article by following the link above, in paragraph 9 you will see this comment:

Inline CSS is safer, and plain-old FONT tags are safest (code like it’s 1996, remember?).

They were not kidding about the code in 90s style. It is true. Well, I digress, so we have a table located in the div’s, here it is:

<div style="font-size: 22px; padding: 20px 0 30px 0; font-family: verdana;">
<table>
<tbody>
<tr>
<td style="text-align: right; width: 50%;">Username:</td>
<td style="text-align: left;"><strong>Alibaba</strong></td>
</tr>
<tr>
<td style="text-align: right;">Password:</td>
<td style="text-align: left;"><strong>sim-sim</strong></td>
</tr>
</tbody>
</table>
.....</div>

What can we do here? That’s right, set your table styles to those that will override GMail ones. So we quickly copy-paste font styles into the table tags, like this – <table style=”font-size: 16px; font-family: verdana;”>. And here is the desired result:

As you see, it makes sense to apply styles to all elements surrounding content, leaving Google and others alike no chance to override them. Beautiful letters to you in all popular email services and clients!

Writing this post was quite easy as we have already written about a way we use project management system and its’ connection to the forum. I decided to add some stats:

Board started: 15/Jan/2009 2:16 am (no, we are not only launching boards at night)
Number of posts: 12705
Number of topics: 537
Number of users: 26
Number of attachments: 2677
Size of posted attachments: 513.56 Mb
Database size: 89.3 Mb
Posts per day: 12.61
Attachments per day: 2.66

These are for three developers who are engaged in work continuously and the system administrator. All other activity is from customers and two guys who have tried to work with us some time ago.

This is a breakdown of posts inside our team:
Igor – 6484
Dmitry – 2516
Alexander – 2843
Vadym (systems administrator) – 252

I decided it will be better to let you see it all first. Taking into account my addiction to screenshots with notes inside I made this screenshot (click to enlarge):

And finally, share some of the nuances.

It is important to give topics meaningful names and even more – when you reply change the subject to adjust to your message. You will agree that having one hundred replies on a topic with the name ‘What do you think about …?’ which all have subjects ‘Re: What do you think about …?’ do not make searching and working inside the board very pleasant. This is your team board – take care of it.

Sometimes you start discussing a task inside a topic but it quickly breaks into different subjects all mixed up. I advise to stop and create a new forum and split existing discussions into different topics. In phpBB you can use Merge topic or Move posts to do so. You will most likely do it anyway later. To give you an example we started working on our first web application in starty.co family of products and discussions started inside a topic as a product is pretty simple, we thought. It just seemed that a dozen more posts, and we’re done. When it became clear that we are not quite finished and more and more posts were created, I spent about two hours creating the right structure. I created a forum-category, created forums for each starty.co product and a forum to discuss common concepts related to all of them in it and then threw the messages to the newly created forums by subject. It is not a disaster, but quite boring work. If you detect this sooner you will have less of it.

When you write a message ‘one more thing’ often comes to mind and this thing often is not related to what you wrote. You write about architecture and this is about design or security. Do not be lazy – finish your current post and create a new post just with this thing in a right place. That is not to create islands of discussion A inside discussion B. When you will need to reread A you will not remember that something important about A is written in B.

Split all forum users into groups (us, clients, apprentices) and administer groups. The phpBB administrative interface is overwhelming for a small group of users, so the less time you spend there – the better for you.

In many forums it is handy to create Announcements or Stickies. For me, there’s no difference as I use them to place a topic at the top inside a forum. For example, in forums where we discuss articles in our blog I mark as announcements not yet published posts. It is convenient to have things you work on now in a separate section which is always at the top. The forum clients have access to uses an announcement topic as a place where updates are recorded. All posts in it are messages about upgrades on production and test systems, which version they run and what changes each new version introduces. But following our rules we do not explain each change or piece of functionality or a bug but just list them and create links to separate forums, clients also have access to, and can follow to read more details.

We try to discuss important things in the forum, and not on Skype and in the mail, because everyone understands that a couple of week’s later things will be forgotten. Seeking dialogues in different places is annoying and takes time. The key to the preceding sentence is the phrase ‘important things’. That is, we use Skype and e-mail of course too. Important points will still find its place in the forum in the form of: ‘We discussed on Skype so-and-so. We decided to do thus and thus.’ With emails it is much easier – Ctrl-c and Ctrl-v but only key information is copy/pasted.

Each of us has developed his unique style of work with the forum. For example, Vadym does not like a forum at all, so we communicate with him by mail, and then transfer important bits to the forum. Alexander & I go in the forum when we receive new messages in the mail, and Dmitry just opens forum home page time to time or uses search.php?search_id=unreadposts and clicks on the unread post icons.

And lastly, our forum is publicly available, but … it is closed with .htaccess and we switched off registration. That is you not only need to pass .htaccess but also have an account to see posts. Are we a bit paranoid about it? Maybe. For now it is not hard for me to switch on board registration, register a client account as if he had registered himself, switch off board registration, activate a new account and explain to a client how to use the forums, mail URL, .htaccess and account login details.

When it became clear, that dotProject’s forum does not suit us anymore, we decided not to fix and adjust it. We started using phpBB, despite of the fact that dotProject’s forum already contained lots of posts (to be precise 8247 posts by the time we swithed to phpBB). We had to visit it quite often at the beginning but with time more and more posts were created in phpBB and dotProject’s forum was visited less and less. Now very rarely. Our team is distributed – USA, Australia and Ukraine, so a forum for us is a must have. One may think that it is not needed if you are all in the same room and can just talk. It will be interesting to find out about your approach.

This is why we advise you not to look for an optimal project management system (with decent forum component) and use what fits best for handling projects, tasks, time tracking and communicate in a specialised forum.

In a list we announced here we have to write about PHP Under Control, I think Dmitry will be able to carve out some time and write about it next time.

We are quite conservative in the use of a version control system. Three years ago, Vadym had convinced us to start using SVN instead of CVS. At the same time Nerrvana came into our lives and we wanted to try some new systems that we thought would improve the process of its development. So we eventually adopted PHP Under Control (PUC) and Review Board (RB).

Today we will talk about RB. We use RB in the post commit mode. In this mode code is first committed to the SVN repository, and then later SVN RB hook* picks it up, adds to the RB which in turn sends letters to all the participants signed on changes in this SVN repository.

There is another mode – pre-commit. More information can be found in the RB documentation. This mode requires that the review request is created by hand. We thought it would be much easier to get it to do SVN. What if the errors will slip into the repository? They get there anyway But the requests are generated automatically, and errors in the code can be fixed in the next commit.

We have several repositories for different projects. RB makes it easy to customize who will get review requests. First, if there is no hook in the repository, there will be no review requests. Second, as shown in the screenshot below, within the administrative interface RB has a page that allows you to fine tune who gets notified with regular expressions.

Fig. 1 Fine tuning who gets what in admin UI

How does it work for us?

Let’s take a project were three of us are involved – me, Dmitry and Alexander. For example I did a commit. Immediately after a commit, a review request appears in RB with Open status. All three of us are working on this project will get a message about a new review request.

Fig. 2 Dashboard
(click to enlarge)

Using the link inside an email Dmitry and Alex can open diff, and see if everything is all right. We agreed that if all goes well, we review by setting the flag “Ship it!” You can also write a comment like “OMG – so much work, congrats on finishing” or “Have no idea what you did, life will prove if you are right. The first three lines – OK”.

Fig. 3 Review Request
(click to enlarge)

Let’s say, Alex looked at my commit in RB and is happy with everything. He marks the review as “Ship it” straight away. However Dmitry has some comments. He writes about them, and leaves the review as is (not marking it “Ship it”). All of us will get an email from RB with Dmitry’s comments. I am reading it, looking at the code and writing a response inside Review Board precisely around line 214 where Dmitry commented. For example – ‘Thank you, my terrible mistake – do not tell my mum please’.

Fig. 4 Ship it

Then I correct an error in the code and make a new commit with the commit message containing a link to this review. This creates yet another review request, but by looking at its commit message, containing a reference to RB review, it becomes clear to all that this commit fixes up a problem as a result of previous review. After that, Dmitry with a clear conscience, marks both reviews (one where problem was found and one where it was corrected) as the “Ship it”. It is quite possible there were no bugs in the code. In this case I write an explanation in the comments within the code directly in RB. Dmitry responds to it.

Fig. 5 Review menu

Alex can also join the discussion around that piece of code inside review request. In the end, Dmitry marks a review as “Ship it”, making it clear to me that it can be closed now. If I see two “Ship it” flags on my review request – this means both Dmitry and Alex are happy with a change, and close review request (Close-> Submitted). It should be noted that with each new commit or each new comment RB sends mail to all of us with links. Here is the message sent by RB to me with a message from Alex that he looked at my changes and marked it “Ship it”.

Fig. 6 Email notification

Once you leave a comment you’ll still get an email. An email about your own activity. We did not find how to configure the RB so that the author did not receive mail on their own activity and this is perhaps the only thing missing in RB. However, this does not affect our RB perception, which is very good overall.

In conclusion we are totally happy with Review Board and highly recommended it. If there are issues with the setup –  contact, we’ll help with what we can.

P.S. I almost forgot – here is RB of RB development team. You may want to look at it.

Fig. 7 Inline comments on RB site

* – about hooks
We took the hook for SVN. Vadym changed it by commenting out things we do not need and connected to SVN. You can read more about the hook for SVN here.

Here’s the code we use:

#!/bin/bash
REPOS="$1"
REV="$2"
cd ~apache
/usr/bin/python /var/svn/hooks/reviewboard-post-commit-hook.py "$REPOS" "$REV" &>/dev/null || exit 1

This is an actual file reviewboard-post-commit-hook.py we use:

#!/usr/bin/env python
#
# reviewboard-post-commit-hook
# This script should be invoked from the subversion post-commit hook like this:
#
# REPOS="$1"
# REV="$2"
# /usr/bin/python /some/path/reviewboard-post-commit-hook.py "$REPOS" "$REV" || exit 1
#
# Searches the commit message for text in the form of:
#   publish review - publishes a review request
#   draft review - creates a draft review request
#
# The space before 'review' may be ommitted.
#
# The log message is interpreted for review request parameters:
#    summary = up to first period+space, first new-line, or 250 chars
#    description = entire log message
#    existing review updated if log message includes 'update review:[0-9]+'
#    bugs added to review if log message includes commands as defined in
#      supported_ticket_cmds
#
# By default, the review request is created out of a diff between the current
# revision (M) and the previous revision (M-1).
#
# To create a diff that spans multiple revisions, include
# 'after revision:[0-9]+' in the log message.
#
# To limit the diff to changes in a certain path (e.g. a branch), include
# 'base path:"
"' in the log message.  The path must be relative to
# the root of the repository and be surrounded by single or double quotes.
#
# An example commit message is:
#
#    Changed blah and foo to do this or that.  Publish review ticket:1
#      update review:2 after revision:3 base path:'internal/trunk/style'.
#
# This would update the existing review 2 with a diff of changes to files under
# the style directory between this commit and revision 3.  It would place
# the entire log message in the review summary and description, and put
# bug id 1 in the bugs field.
#
# This script may only be run from outside a working copy.
#
 
#
# User configurable variables
#
 
# Path to post-review script
POSTREVIEW_PATH = "/usr/bin/"
# Username and password for Review Board user that will be connecting
# to create all review requests.  This user must have 'submit as'
# privileges, since it will submit requests in the name of svn committers.
USERNAME = 'XXXXX'
PASSWORD = 'XXXXXXXXXXX'
 
# If true, runs post-review in debug mode and outputs its diff
DEBUG = False
 
#
# end user configurable variables
#
 
import sys
import os
import subprocess
import re
import svn.fs
import svn.core
import svn.repos
 
# list of trac commands from trac-post-commit-hook.py.
# numbers following these commands will be added to the bugs
# field of the review request.
# supported_ticket_cmds = {'review':         '_cmdReview',
#                          'publishreview':  '_cmdReview',
#                          'publish review': '_cmdReview',
#                          'draftreview':    '_cmdReview',
#                          'draft review':   '_cmdReview'}
#
# ticket_prefix = '(?:#|(?:ticket|issue|bug)[: ]?)'
# ticket_reference = ticket_prefix + '[0-9]+'
# ticket_command = (r'(?P[A-Za-z]*).?'
#                   '(?P%s(?:(?:[, &]*|[ ]?and[ ]?)%s)*)' %
#                   (ticket_reference, ticket_reference))
 
def execute(command, env=None, ignore_errors=False):
    """
    Utility function to execute a command and return the output.
    Derived from Review Board's post-review script.
    """
    if env:
        env.update(os.environ)
    else:
        env = os.environ
 
    p = subprocess.Popen(command,
                         stdin = subprocess.PIPE,
                         stdout = subprocess.PIPE,
                         stderr = subprocess.STDOUT,
                         shell = False,
                         close_fds = sys.platform.startswith('win'),
                         universal_newlines = True,
                         env = env)
    data = p.stdout.read()
    rc = p.wait()
    if rc and not ignore_errors:
        sys.stderr.write('Failed to execute command: %s\n%s\n' % (command, data))
        sys.exit(1)
 
    return data
 
def main():
    if len(sys.argv) != 3:
        sys.stderr.write('Usage: %s  \n' % sys.argv[0])
        sys.exit(1)
 
    repos = sys.argv[1]
    reposname = os.path.basename(repos)
    rev = sys.argv[2]
 
    # verify that rev parameter is an int
    try:
        int(rev)
    except ValueError:
        sys.stderr.write("Parameter  must be an int, was given %s\n" % rev)
        sys.exit(1)
 
    # get the svn file system object
    fs_ptr = svn.repos.svn_repos_fs(svn.repos.svn_repos_open(
            svn.core.svn_path_canonicalize(repos)))
 
    # get the log message
    log = svn.fs.svn_fs_revision_prop(fs_ptr, int(rev),
                                    svn.core.SVN_PROP_REVISION_LOG)
 
    # error if log message is blank
    if len(log.strip()) < 1:
        sys.stderr.write("Log message is empty, no review request created\n")
        sys.exit(1)
 
    # get the author
    author = svn.fs.svn_fs_revision_prop(fs_ptr, int(rev),
                                       svn.core.SVN_PROP_REVISION_AUTHOR)
 
    # error if author is blank
    if len(author.strip()) < 1:
        sys.stderr.write("Author is blank, no review request created\n")
        sys.exit(1)
 
    # check whether to create a review, based on presence of word
    # 'review' with prefix
    # review = r'(?:publish|draft)(?: )?review'
    # if not re.search(review, log, re.M | re.I):
    #     print 'No review requested'
    #     sys.exit(0)
 
    # check for update to existing review
    # m = re.search(r'update(?: )?review:([0-9]+)', log, re.M | re.I)
    # if m:
    #     reviewid = '--review-request-id=' + m.group(1)
    # else:
    #     reviewid = ''
 
    # check whether to publish or leave review as draft
    # if re.search(r'draft(?: )?review', log, re.M | re.I):
    #     publish = ''
    # else:
    publish = '-p'
 
    # get previous revision number -- either 1 prior, or
    # user-specified number
    #m = re.search(r'after(?: )?revision:([0-9]+)', log, re.M | re.I)
    #if m:
    #    prevrev = m.group(1)
    #else:
    prevrev = int(rev) - 1
 
    # check for an explicitly-provided base path (must be contained
    # within quotes)
    #m = re.search(r'base ?path:([\'"][^\'"]+[\'"])', log, re.M | re.I)
    #if m:
    #    base_path = m.group(1)
    #else:
    #    base_path = ''
 
    # get bug numbers referenced in this log message
    # ticket_command_re = re.compile(ticket_command)
    # ticket_re = re.compile(ticket_prefix + '([0-9]+)')
 
    # ticket_ids = []
    # ticket_cmd_groups = ticket_command_re.findall(log)
    # for cmd, tkts in ticket_cmd_groups:
    #     funcname = supported_ticket_cmds.get(cmd.lower(), '')
    #     if funcname:
    #         for tkt_id in ticket_re.findall(tkts):
    #             ticket_ids.append(tkt_id)
 
    # if ticket_ids:
    #     bugs = '--bugs-closed=' + ','.join(ticket_ids)
    # else:
    #     bugs = ''
 
    # summary is log up to first period+space / first new line / first 250 chars
    # (whichever comes first)
    summary = '--summary=' + log[:250].splitlines().pop(0).split('. ').pop(0)
 
    # other parameters for postreview
    # repository_url  = '--repository-url=file://' + repos
    repository_url  = '--repository-url=http://XXXXXXXXXXX.com/svn/' + reposname
    password        = '--password=' + PASSWORD
    username        = '--username=' + USERNAME
    description     = "--description=%s[%s]: %s" % (reposname, rev, log)
    submitas        = '--submit-as=' + author
    revision        = '--revision-range=%s:%s' % (prevrev, rev)
 
    # common arguments
    args = [repository_url, username, password, publish,
            submitas, revision]
 
    # if not updating an existing review, add extra arguments
    #if len(reviewid) == 0:
    args += [summary, description]
 
    if DEBUG:
        args += ['-d', '--output-diff']
        print [os.path.join(POSTREVIEW_PATH, 'post-review')] + args
 
    # Run Review Board post-review script
    data = execute([os.path.join(POSTREVIEW_PATH, 'post-review')] + args,
                   env = {'LANG': 'en_US.UTF-8'})
 
    if DEBUG:
        print data
 
if __name__ == '__main__':
    main()

Part VII – our plans changed, we start implementing it.

In our last post we promised to tell you about the integration of our product (Nerrvana) with our forum (phpBB), blog (WP) and marketing site. We also were planning to show how we added code to collect analytics info into each component. Our plans changed since.
We decided to launch a beta version of our product without a forum, which takes the forum as well as Ideas and Answers out of our priorities list for now. We will add these parts during the beta testing period. But we need analytics anyway and it is time to start implementing it. Rephrasing ‘I sing what I see’ into ‘I code what I sing’ we start making it.

To heat up and tune we will complete a few tasks supporting our project.

1. Import GeoIP City database into PostgreSQL

We will need it to get visitors’ location. If you read our previous posts this is not something we terribly want from web analytics. However it can be handy to figure out what our clients business hours are. It is to understand that city info can be very wrong. According to this database I am in Perth which is not true for me. I am in Sydney but my ISP is in Perth. Country info is more reliable.

We take sources from GeoLite City and read ‘Loading the GeoIP City database into PostgreSQL’.

Create tables:

CREATE TABLE locations
 (
   id bigint NOT NULL,
   country character(2) NOT NULL,
   region character(2),
   city character varying(75),
   postal_code character varying(15),
   latitude numeric(6,4) NOT NULL,
   longitude numeric(7,4),
  metro_code integer,
  area_code integer,
  CONSTRAINT locations_pkey PRIMARY KEY (id)
);
 
CREATE TABLE blocks
(
  start_ip bigint NOT NULL,
  end_ip bigint NOT NULL,
  location_id bigint NOT NULL
);

Execute:

-- Data is in ISO-8859-15 format
SET client_encoding to "ISO-8859-15";

Put CSV files on C:\. Delete the first line containing copyright info and import them into database.

COPY locations (id,country,region,city,postal_code,latitude,longitude,metro_code,area_code)
FROM 'C:/GeoLiteCity-Location.csv' WITH CSV HEADER;
 
COPY blocks
FROM 'C:/GeoLiteCity-Blocks.csv' WITH CSV HEADER;

We use Unix slash / here anyway even though we are on Windows.

2. Enhance and improve log formats for Apache and ProFTPD

Before writing code we make a short stop and create own format for Apache logs based on standard “combined” type to make it more friendly for our home-grown web analytics. Here is a “combined” format:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

we create own type:

LogFormat "%h %{%F %T}t  \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \
visitor_id:%{visitor_id}n session_id:%{session_id}n user_id:%{user_id}n" nerrvana

Differences:

- dropped %l. Acording to docs this is the ‘Remote logname (from identd, if supplied). This will return a dash unless mod_ident is present and IdentityCheck is set On. Value always empty and useless for us.

- dropped %u. Also always empty.

- changed date/time format from standard [24/Jul/2011:08:49:07 +0000] to 2011-07-24 08:49:07

- added session_id:%{session_id}n user_id:%{user_id}n. We wrote about it before. These values will be empty in logs for now as we haven’t modified marketing site and Nerrvana UI pages yet. But this will not stop us – we have many other things to do before we will start looking into these values. For now we will add dummy values when we will parse logs to database tables.

Now about a few specific things we have to deal with.

First one is an FTP server (ProFTPD) which is part of our Nerrvana. It allows to upload Selenium tests and download results. Nerrvana can work with multiple FTP servers. We built it this way to make the system more flexible and potentially distributed. Each machine runs FTP server (currently we have only one) and also runs Apache with authorization overlooking ftp folders. It allows us to provide HTTPS access from UI to test results. Since FTP servers are running on different virtual or physical machines their Apache’s logs are located there too. When a user clicks a link to browse test results we loose him in Nerrvana logs. To solve this problem we use redirect pages which leave this info in Nerrvana logs and redirects the user to a page he wanted to see.

Second thing is also related to having a FTP server as a part of our system. Normal workflow we expect after registration – create a place to load files we call Space, connect to it with FTP client and load Selenium tests, create schedule and run tests. In our analytics we want to know how real workflow deviates from ideal one. It’s one thing if a client created his Space but never uploaded files, another – when he loaded and deleted files without making an attempt to launch tests. We need to know this to pro-actively help a particular client before he abandons our service or to see a wider problem many clients face. So we are going to parse the second type of log – xferlog.

It looks like this:

Tue May 31 16:08:17 2011 0 ::ffff:192.168.1.254 303 /var/lib/trillium/ftproot/demo433/sf_tests/_files/build/class/com/deepshiftlabs/sf_tests/SftestException.class b _ i r demo433 ftp 0 * c

From manual:

current-time transfer-time remote-host file-size filename transfer-type special-action-flag direction access-mode username service-name authentication-method authenticated-user-id completion-status

Everything is pretty straightforward. Client demo433 on 31st of May at 16:08:17 loaded (i) file SftestException.class. We do not create a security tracking system and this is why we will store bare minimum of info to keep client’s privacy. With FTP we only need to know the sort of activity (uploaded, downloaded, deleted). If files are constantly downloaded, deleted, uploaded – most likely files are getting edited in ftp client. We need to rush with enabling ssh access for client’s accounts. If we will store just file extensions we will be able to see preferences of a particular client on one side and have stats what languages are used by free/paid/heavy/light Nerrvana users.

Analytics? Yes. Web analytics? Don’t know – we craft analytics of a special kind not blindly following fundamentals of this art but strongly adhering to common sense, customer care and commitment to excellence.

On this optimistic note I will finish this post. Next post – log parsing and storage, crawler traffic recognition, exceptions in logs to ignore, extracting key phrases from search engine referrers. Did I miss something? E-mail – help me out.

This post lists our requirements for a support system and will be followed up (in our next post) by configuration instructions implementing them in a system of our choice – Kayako.
You may ask us ‘Why Kayako?’ We tried some other systems – open source OTRS, OSTicket and commercial HelpSpot. Below you can find some reasons why these systems were rejected by us.

HelpSpot
- no prebuilt list of customers
- sharing is global and is not per portal at this time. So we can not turn on sharing (seeing all tickets for a company) for a client and turn it off for Nerrvana.
OTRS
- did not like UI
- too complex
- no portals
OSTicket
- no sharing. Each user can see only their own tickets
- no time tracking
- no portals

What are our requirements?

1. We have clients (companies) we design web applications for and we need a support system. Using emails for support is not a viable option for us anymore. We deal with helpdesk support staff in these companies when we talk about support. Do they raise a ticket using their own support system in principle? How well their support department is managed? Do they require to create a ticket or search their internal system before escalating something to us? We do not know. But nobody can stop us from optimising the time we spent on support.

We more often have situations when Brad asks us exactly the same question we answered for Joy a week ago. Or even when Joy asks us about particular problem X, which Brad already raised and she has no clue that some replies already have been sent to Brad. We are doing our best to pick up common questions and move them to ‘How To’, a forum clients can access but despite of it lots of time is spent to find all the emails we sent to Brad and send it to Joy. It is often not enough to find the last email as it might not contain the whole story and if you do so you will have more questions from Joy anyway. Support system on our side with shared access to tickets for client’s staff allows us to find a ticket and forward it or point to a ‘How To’ article and close it.

In some projects an active development phase is completed and support work kicks in as a primary activity. We fix bugs, complete some change requests and answer questions. When you actively develop and support using email it is too hard to figure out how much time goes to support. Very important to know, it is (in yoda voice).

I do not think I will surprise you by telling you a fact that development speed is directly related to how much goes to support. At least in a small team like ours. If we code fast there is no time to maintain documentation up-to-date which causes more questions and more time for support. In this situation we need to figure out ourselves how much support the system we designed requires and what are the best terms for us to negotiate for a future support & maintenance agreement. It could be a fixed amount per month, or by hours actually spent or a combination of them. X amount a month for 20 hours of support with Y amount per hour for anything above 20 hours. So we continue to use dotProject as an internal project management system, phpBB as a place to discuss projects from dotProject as well as a place to discuss projects with clients and replace support by email with Kayako.

2. We created a system (Nerrvana) we are going to launch this year and we would like to have a decent support system accompanying it. Our clients do not even need to interact with the support system UI, but we will. It will allow us as developers to be organised and solve clients’ problems quickly.

Now let’s go from background info into details.

With point 1 (clients’ work) – all is simple and clear. We create in Kayako companies-clients and pre-create all staff members from their helpdesk departments. If changes will need to be made i.e. new person started – head of support will request us using Kayako as well. Kayako has an option to make a company shared which means all users can see and update all company tickets. They log in to custom portal for example http://support.deepshiftlabs.com/index.php?/client_1_company_name and see and update all tickets. It is by company, not a global setting. This is exactly what we want for our clients. Joy will see what we responded to Brad if she works on his ticket. One Kayako install allows you to have many portals and email queues associated with them. You create email, associate it with a queue and create a portal. If you have two clients you will have:

Email client1@support.deepshiftlabs.com and portal http://support.deepshiftlabs.com/index.php?/client1
Email client2@support.deepshiftlabs.com and portal http://support.deepshiftlabs.com/index.php?/client2

Tickets can be created from inside the portal or by simply emailing it to an associated email. New tickets created either way generate an auto-response containing a ticket ID allocated. By keeping it in subject you will update the ticket by simply replying to it.

Let’s look at requirements for point 2 (own product). Our portal will be on http://support.deepshiftlabs.com/index.php?/nerrvana. We read ‘Why your company should have a single email address’ and think it makes sense first of all for our clients and decided to go this way. Therefore all places where we will provide email including support will use contact@nerrvana.com. Each and every email will go to Kayako queue for processing. We are going to send only one auto-generated email (may be only while we are sleeping and can’t reply quickly) as it is important, from our point of view, for a sender to know that their email was delivered. Besides it will be from contact@nerrvana.com and it won’t be ‘please do not reply’ but a ‘please feel free to reply’ email. If Kayako does not have sender’s email in database it will create a new account automatically. So an important difference in Kayako configuration for our product will be an instruction to accept all incoming mail for contact@nerrvana.com. In point 1 (for our contract work with clients) we accept requests only from pre-created by us emails.

The second major difference in configuring Kayako for own product will be enabling and using Kayako API. By using it we can allow our clients to raise support requests from inside our product UI. We can check if Kayako account already exists (we do not create accounts in Kayako when client registers), create it first if necessary and finally create a ticket. Kayako uses MySQL and we store account info in PostgreSQL. After ticket was created via API we can add a private note to it at the same time also via API. It will contain UI page reference where request was created and all supporting info. In our case this will be a test run execution where we can add necessary SQL statements and links to logs to speed up resolution for one of us who will work on this case.

We are also planning to create client’s profile page with information about registration date, account type, short history of activity, business hours and make it accessible from inside Kayako. This will allow us to see if a request came from a new user who can’t launch their first test (our system allows to launch Selenium tests on different platforms) or from a longtime client who might know our system as good as we do (well, almost). To summarize – all such handy info can be added to ticket via API from inside our application.

Finally a few notes about mail server configuration for Kayako. Kayako allows to use GMail account with enabled IMAP but we decided to use our own mail server (sendmail). This decision is based on our belief that Gmail uses multilevel Spam filtering approach and even if we will disable the Spam folder in GMail there is a chance that we won’t see some mail as it might be filtered out by low-level filters. For now we are not even planning to add Spam filter to it (normally SpamAssassin). We do want to have full control around what we receive especially since we use a single email for everything. Even if we will be forced to turn on the Spam filter on sendmail someone will glance over it before deleting emails. For the record, Kayako is also able to detect Spam and has a special folder inside UI.

Cheers for now. Kayako is installed and our next post will be about making it all work in Kayako (expect lots of screen shots).

After reading Jason Cohen’s post “Naming your startup: Settle down, it’s cool!”, I recalled previously read “How To Pick A Company Name: Tips From The Trenches” and “17 Mutable Suggestions For Naming A Startup”, and even a manual from Igor International. I still remember how we came up with the name. All discussions have been recorded in our forum. Besides, I’ve never seen stories about inventing a name. And so was born the idea of this post.

Here is the (first?) naming chronicle. To ensure that you understand our conversations, I must say that English, for many of us, is the second, and for some – the third language. We were looking, of course, for an English name, so if some candidates may sound funny or have weird association in English, you’ll know that in Russian, they sound just great.

Now a few words about the subject.

Deep Shift Labs – the name of our team, and therefore the product name consonant with the words ‘deep’ or ’shift’ would have been good from our point of view.
Selenium – our product uses Selenium, and therefore the complete or partial (root or suffix of the word) presence of this word would also be not bad.
Test-ing – our service performs automated functional tests, and therefore such a word can be present in the name.
Cloud, Elastic, Virtual – the system creates a virtual environment consisting of 10 or 100, or … virtual machines.
Bug, Error – it is clear that we are looking for – bugs, errors.

We had such, I would say, semantic environment before we started. Perhaps it is wise to start formulating it before you will start. We have not been writting it explicitly, but as you’ll see all the discussion swirled mostly around these words.

By the time this discussion around a name started in January 2010 we were already working on our system for a year and a half, and therefore we were searching for a name for the already running system. The system is created in our spare from contract work, time. From the very beginning we stuck with a consonant to Selenium name – Trillium. The word also contains the letter ‘T’ from Testing, but, unfortunately, there are^* too many people already using it. Besides, it is very close to a popular chat client Trillian by Cerulean Studios. We started talking about a new name, as we were sure that we can come up with a better name with a free domain. With every moment of inspiration, we were generating and discussing names over and over again. Some of these dialogues are listed below with a small author’s corrections

The discussion participants:
Igor
Dmitriy
Alex
Oleg – advises us when he is interested in our questions and they intersect with his knowledge and experience. We consider him as an adviser and therefore ask questions … not so often

Mon Jan 11, 2010 11:14 am – Igor

Trillium – current name

ShiftOS or ShiftOs or Shiftos
ShiftOZ or ShiftOz or Shiftoz – Australia – Oz
Shiftoid – imaginary
Selenoid – imaginary
Selenius – imaginary

Selen – imaginary
Selenite – “contains no significant selenium; the similarity of names comes from both substances being named from the Ancient Greek word for the Moon.”
Selenide – “chemical compound in which selenium serves as an anion”
Selenosis – Selenium poisoning
Selenious – containing Selenium

Gridder – a player

Found and immediately discarded (a word ‘test’, turned out to be quite dangerous):
Testaceous – having a hard shell or shell-like outer covering
Testa – the protective outer covering of a seed.
Testee – one who takes or has taken a test.
Testoid – any substance, as testosterone or androsterone, that promotes male characteristics.
Testis – an organ which produces sperm.

Tue Jan 19, 2010 9:08 pm – Dmitriy

In a parallel discussion topic we were talking about character/logo, where pretty soon we agreed to use a chameleon. It is catching bugs (even at a great distance, like our system), changes colour and looks in all directions. Therefore, further you will come across words that intersect with the word ‘chameleon’ (author’s note).

Chameleon is a cool personage too. You have everything – nice tail, and unique tongue, eats bugs, and the changes colours.

Mon Jan 18, 2010 2:19 pm – Igor

Two sounding like chameleon + selenium names:

Seleon – imaginary
Selion

Wed Jan 20, 2010 11:45 am – Igor

Dmitriy: “Chameleon is a cool personage too. You have everything – nice tail, and unique tongue, eats bugs, and the changes colours.”

Additionally a chameleon is a much simpler character to create funny situations around. Another perfect match is an ability to ’stretch’ his tongue – we also can stretch and create more VMs as well as to ‘change colour’ – we can test Windows IE8 and Firefox on Linux. These are very nice parallels.

Today on the morning train I was thinking that we need to sort of align our character/logo concept with the product name and also invent a motto.

What is the essence of our system? We are a “Selenium cloud”. Cloud means an elastic, stretchable environment which can be used on demand (when needed). Amazon calls its service Elastic Computing Cloud, Oleg’s business is called Parascale^** which uses ’scale’ or ’scalable’. I also used word ‘valium’ thinking of combining it with ’selenium’ and motto ‘Stress free Selenium testing’ or something along these lines but after passing several stations (remember I was on a train), I realised that being in fact stress-free it does not reflect the root purpose of our service, which is from my point of view – scalability Selenium way.

After I realised this I started with words which are:
Chameleon
Selenium
Elastic
Valium
Cloud

combining them into names:

–Valium&Selenium
Valenium (+)
Selanium (+)

Selium – should not be here but ….
–Scale&Selenium
Scalium
Scanium

–Cloud&Selenium
Closelin (-)
Closeus (-)
Seclonium (-)
Closenium (-)
Clodium
–Elastic&Selenium
Seltic
Elanium (+)

The ones I liked, I marked with a plus and otherwise.
Motto in all these names could be:
- stress free Selenium cloud
- elastic Selenium cloud
- Selenium on a cloud

We can polish motto further. I am expecting this motto in big letters to be on top of the home page on the right side from logo and service name.

Wed Jan 27, 2010 2:04 am – Alex

Dmitriy: “Chameleon is a cool personage too. You have everything – nice tail, and unique tongue, eats bugs, and the changes colours.”

Agree. Well, congratulations, on settling with a personage.

Igor:
“–Valim+Selenium
Valenium
Selanium
Selium – – should not be here but ….”

I associate Selanium and Elanium with drugs anyway, similar to ‘relanium’. Selanium sounds better. Can I offer a couple of options? No? Well, I will anyway – Secleon (Selenium cloud chameleon), Scasecl (Scalable selenium cloud).

Wed Jan 27, 2010 11:16 am – Igor

Alex: “I associate Selanium and Elanium with drugs anyway, similar to ‘relanium’.”

Agree. Was carried away a little.

Alex: “Well, I will anyway – Secleon (Selenium cloud chameleon), Scasecl (Scalable selenium cloud).”

Your names account for everything but are hard to pronounce. Perhaps it makes sense to abandon attempts to include many things into a name as we can end up with something similar to A.S.S. Concerts, and just call it, for example – Debugz. Sounds as debug and bugs and destroy bugs and move essence of it – Selenium cloud – into a slogan.

Wed Jan 27, 2010 9:22 pm – Oleg
(we sent slogans and asked for an opinion – author’s note)

“selenium in the cloud” – sounds as a good description
“stress-free selenium service in the cloud” – good slogan to me

Sun Jan 31, 2010 11:33 am – Igor
Debugz – domain name taken.

Read here:
http://en.wikipedia.org/wiki/Bazooka – domain name is obviously taken but I write anyway may be it will enlightens you with some deviations of it
http://en.wikipedia.org/wiki/Chameleon – “and rapidly extrudable tongues” thinking of ‘eXtrudius’ or ‘Extrudius’.

Since it is Selenium on demand – may be ‘Demandus‘ or ‘Demandium’ or ‘Demanium’ with а Selen-ium suffix?

Sun Jan 31, 2010 11:35 am – Igor

Moving to Esperanto.

Mon Feb 01, 2010 10:29 pm – Igor

From Esperanto:
eskali or eSkali – translated as ‘scale’. It can be written in eBay style – small letter then capital). Registered trade mark is not found.
insekto or inSekto – translated as ‘bug’. Word contains ‘Se’ which is Selenium from periodic table. Registered trade mark is not found.

Other words from Esperanto:
demand – debito
kill – mortigi
cloud – nebuligi, nubo

Other non Esperanto names – trademarked and domains are taken:
Bugster
Bugbuster
Buzzuka
Buginator
Buggle

Harry Potter theme:
bugdemort.com – available but I am afraid of J. K. Rowling legal action
debugtor.com – was it dementor in a book? I like it. Do you?

debugius.com – available
debugos.com –it is better than shiftos.com we were thinking of. Domain is available. selentos.com

Tue Feb 02, 2010 1:39 am – Alex

Discussed (with Dmitriy – author’s note) and think that ‘eXtrudius’ or ‘Extrudius’ sound good.

Igor: “Demandus or Demandium or Demanium”

These ones smell demonology which is scary

Tue Feb 02, 2010 1:52 am – Alex

We had a discussion tonight. Initial considerations – the trademark and domain name must be at least similar, ideally be the same. Of course this is not necessary condition, but it is highly desirable.

Igor:
“debugtor.com
debugius.com
debugos.com”

The last two names seem to be ok (debugius.com, debugos.com), as previously discussed names ‘eXtrudius’ or ‘Extrudius’. But none of them is an undisputed leader in each has some flaw to us. To make it easier to choose we also have prepared some names. All domains are free:

clinnium.com (1)
bugspell.com (3)
bugzcore.com (1)
buzzless.com (1)
deepium.com, deelium.com (3)
deesbug.com (2)
deesc.com (2)
fireum.com (2)
s-clo.com (1)
sellenia.com (1)
selenager.com (1)
seekabug.com (1)

In brackets is the number of votes for each name, the favorites are easy to see Let’s now based on the resulting list (three votes) will hold a final vote by selecting the three candidates and get the winner.

Tue Feb 02, 2010 9:53 am – Oleg

Except for “selentos” and “escali”, everything else sounds bad to me.
You seem very focused on the -os and -us endings. Those aren’t terribly appealing.

Consider that many very popular web startups had weird names that had nothing to do with what they actually produced. If you want something that specifically focuses on what you do, what is it you think your business does? Selenium on demand testing. Selenium is derived from greek “selene” — moon. Is “moonwalk” taken?

Igor: “moonwalk.com is taken.”

Tue Feb 02, 2010 12:50 pm – Igor

Moon theme – Mooongle.com
We can draw a chameleon behind a name and make two o’s his eyes. Pronounced as moongle. moongle.com is taken so I used triple ‘o’.

Also Moon in Latin is Lynaris and Lynaris.com is available but I like Mooongle more – sounds better and even like “you know who”.

Tue Feb 02, 2010 2:31 pm – Igor

We turn to the names of the Aborigines.

Tue Feb 02, 2010 8:00 pm – Dmitriy

Oleg: “… many very popular web startups had weird names that had nothing to do with what they actually produced”

We discussed the idea of using some Russian nice-sounding word, such as ‘oblako’ (cloud).

Igor: “We turn to the names of the Aborigines “

Many nice words I have already heard (Dmitriy knows them as our clients branches are functioning in these towns – author’s note). For instance arana – moon. By the way, I like the words aranda and armidale. Toowoomba also sounds mysterious.

Igor: “I like Mooongle more.”

The only thing I don’t like about Mooongle – association with Google. Or it is a good association?

Tue Feb 02, 2010 8:46 pm – Igor

Another brainstorming session on a train to work.

Domain names and mottos:
nerrvana.com – finding errors in a pleasant way
erranium.com – finding errors with stress free Selenium cloud
erronium.com – same as above
moonshu.com – moon and shoot – just and easy to say word related a bit
nirvanium.com – free from suffering Selenium testing – this one is cool
moonerr.com – find your web application errors with Selenium cloud (my kid said it sounds like mooning and we do not want it right?)

I like ‘moonshu’.

Dmitriy – I agree – trying to sound like Google is NOT good.

All domains are available. Let’s now not only offer options but state if domain is available or not. Many aborigines’ names (I posted earlier a link) appear to be taken.

Tue Feb 02, 2010 8:59 pm – Dmitriy

Igor: “I like ‘moonshu’.”

I think it is too Asian. I like nerrvana more.

Igor: “finding errors in a pleasant way”

Nice one

Tue Feb 02, 2010 10:04 pm – Alex

Dmitriy: “The only thing I don’t like about Mooongle – association with Google. Or it is a good association?”

In principle, any associations are good if name is easy to remember. The only bad thing it is hard to pronounce ‘ngl’ in Russian. Name is not bad apart from it.

Tue Feb 02, 2010 10:13 pm – Alex

Igor:
“nerrvana.com – finding errors in a pleasant way
nirvanium.com – free from suffering Selenium testing – this one is cool”

These two names are better than others to me. And if to choose from them I would take a second one. I do not like double letters, people will make mistakes and come to a different domain. If we would have in the same business with http://nervana.com/, it would be a clever move. By the way, they acted short sighted by not registering this domain.

Wed Feb 03, 2010 10:39 am – Oleg

Igor:
“nerrvana.com
erranium.com
erronium.com
moonshu.com
nirvanium.com”

Honestly, this -um or -us ending doesn’t sit well with me, but it is just me. My preference, now that you pointed it to me, is to go with an aboriginal name.

Nirvanium sounds least offensive, but it is too close to Nirvanix. That’s grounds for customer confusion and a possibility Nirvanix getting upset and suing you — not that they’d have a case, but can you afford fighting it?

But it’s your company, not mine

Wed Feb 03, 2010 12:28 pm – Igor

Some companies have such nice for my heart names – Igor International while offering guidance on finding a name.

Turning to the Maori dictionary now

demand – tono. Any ideas around ‘tono’?
‘Test’ and ‘try’ (source)
birrku.com
garrmun.com

In many languages a word ‘moon’ starts with ‘lyn…’
lynerr.com
lynderr.com

‘Minder’ is a good word and variations from it for your consideration:

moonminder.com – taken
deminderr.com – demand, error, minder
seederr.com – sees errors instead of you
moterr.com – motor + error
tuchkus.com – just another weird name in English but cloud in Russian
lynabite.com – moon + bite
figerr.com – show ‘dulya’ to errors in your application
donterr.com – do not allow errors to sleep into your application
mustdier.com – no comments
mustdierr.com – place where errors die

Looks like we need a break. Will print aboriginal dictionaries today and have a look once again.

Wed Feb 03, 2010 2:02 pm – Igor

testyfire.com – what about this one ‘fire tests with testyfire’?

testifire.com is already taken but testyfire is even better as it looks like tasty/test=testy (well I remember dangers of starting with test… – ok, ok)

moonfier.com
bugfrier.com
bugfier.com
vemoon.com – virtual Selenium cloud
bugmuter.com
antibuz.com
antibuzzer.com
bugjiuce.com
moongun.com
moonoh.com
testoh.com
moonte.com
buginar.com
bugoh.com
shotomo.com – this one like Japanese sushi bar ’shot’ ‘moon’
debugie.com – has debug and bugie and also like bungie
moongie.com

arilpa.com – aboriginal ‘moon’
gigalee.com – aboriginal ‘moon’

goondooloo.com – Southern Cross but typing it will be a nightmare. Will anybody need to type it?
goolabah.com – box tree – just sounds nice

nananga.com – ‘moon’
woolahra.com – ‘moon’
tanaranga.com – ‘moon’

Wed Feb 03, 2010 6:08 pm – Igor

While you were sleeping Oleg and I exchanged quite a lot of emails and summary of preferences we have:

Debugie
Debuggie
Gigalee
Tanaranga

It does not mean you have to agree or select from these ones.

Wed Feb 03, 2010 9:04 pm – Alex

Igor: “Debugie”

I associate it with debug + IE

Wed Feb 03, 2010 9:33 pm – Igor

Alex: “I associate it with debug + IE”

Популярный суффикс – ничего особенного
Самое извесное название пришедшее в голову http://www.bungie.com(net).

Quite popular suffix -ie nothing microsoft-ish in it. Most famous will be probably http://www.bungie.com(net).

Wed Feb 03, 2010 10:52 pm – Igor

Travelling north from Sydney along the coast by map:
mallabulla.com
bootibooti.com
mooneba.com
kremnos.com
deebung.com – minor variation from real ‘geebung’
deebuh.com – imaginary but looked at the names on a map
deeplie.com – imaginary but looked at the names on a map
neerdie.com – real one

Wed Feb 03, 2010 11:31 pm – Alex

Igor: “deebuh.com”

It’s funny. We can consider it as deep + BUH! The bigger they are the harder they fall.

Igor: “deeplie.com”

Very easy and resounding, I like it. Somehow, I immediately wanted to check domain figli.com. It is registered until 11th Feb 2010. We may be lucky and it will expire and become available.

Thu Feb 04, 2010 12:55 am – Dmitriy

Igor:
“deebung.com
deebuh.com
deeplie.com
neerdie.com”

It’s pretty nice – but how to choose one?
I feel we will finally write names down on pieces of paper and pull out of it.

Wed Feb 03, 2010 13:25 pm – Oleg

The word ‘lie’ has implications. Deeplie sounds more like a conspiracy theory website

Thu Feb 04, 2010 8:55 pm – Dmitriy

Igor:
“deeplie.com
figlee.com”

I also like both. We could use ‘deeplee’ to disassociate ‘deeplie’ from a lie, but it is taken.

Thu Feb 04, 2010 9:06 pm – Alex

Igor:
“deeplie.com
figlee.com”

Yes, both names are good, hard to choose.

Fri Feb 05, 2010 9:42 pm – Alex

Igor: “Google calls things this way – PubSubHubbub“

Things allowed the gods, are not available for the mortals

Tue Feb 09, 2010 10:19 pm – Igor

Alex: “So, let’s use Deeplie.com? We have already started using it in our forum ”

I do not know about you, but I will just do other things and look at all the names again after the break.

Mon Feb 15, 2010 2:48 pm – Igor

(10 days past – author’s note)

Appear to be available:

prompty.com – from dictionary “prompt – ready in action; quick to act as occasion demands.”

аs well as:
promptie.com
promtie.com

Mon Feb 15, 2010 4:48 pm – Oleg

deeplie – can be associated with ‘deep lie’
feeglee – can be associated with ‘fig leaf’

Both are not very good.

prompty.com is neutral

I looked through old names. Do you like gigalee.com – it is aboriginal and it is moon(selen) – so nobody can say we tried to get as many ‘g’ as google has

Mon Feb 15, 2010 9:35 pm – Alex

Oleg: “prompty.com is neutral”

Sounds a bit ‘mumbly’

Igor: “velenium.com or vellenium.com – as Virtual sELENIUM”

Velenium sounds nice.

Igor: “virshify.com – using virtlib command ‘virsh’ used to manipulate virtual machines.”

All consonant sounds are muffled and sizzling. And we still remember name nerrvana.com (this was Alex who turned us back to it – author’s note) and we still like it.

Mon Feb 15, 2010 10:21 pm – Igor

I remember nirvanium.com
nerrvana? Which post?

Tue Feb 16, 2010 9:39 am – Igor

Morning recap:
nerrvana
nirvanium
velenium

Tue Feb 16, 2010 8:10 pm – Igor

Morning workouts with ‘err’

eterrnity.com

Is ‘eterrnity’ a bad association? I do not know how to think of eternity as ‘timeless’ as I do not want it to be associated with never ending process

Motto could be: in eternal (never ending) search of errors (not never ending system work)

counterr.com
enderr.com
eraserr.com
executerr.com
explorerr.com
vierr.com – makes errors visible
shifterr.com – Shift Labs approach for searching errors

fryerr.com – deep frying bugs since 2010
runerr.com
rulerr.com – a new way to debug errors with Selenium

Anything touched?

Tue Feb 16, 2010 9:00 pm – Dmitriy

Igor: “Anything touched?”

Yep, use of ‘err’ in the end – perfect!

I like:
enderr.com
eraserr.com
shifterr.com

First two are similar to killerr (btw domain is available!).

shifterr – good.
Shifter is also “an adjustable wrench/spanner” but it is ok.

Wed Feb 17, 2010 4:40 pm – Igor

Err at the beginning:
erraser.com
erraserr.com

Unfortunately zerro.com is taken.

Wed Feb 17, 2010 10:40 pm – Alex

Igor: “Anything touched?”

Not bad – enderr.com

Thu Feb 18, 2010 9:31 am – Igor

Alex: “Not bad – enderr.com”

Your and Oleg’s point of views matched this time, and he likes our names very rarely.
New names are born every day now. It has probably became a habit.

eclerr.com
testerr.com
enderr – easy to say and a good combination of words ‘end errors’

Thu Apr 08, 2010 11:14 pm – Alex

Igor: “eclerr.com”

Éclair – not our audience. Beer-air, chips-air fit perfectly

Igor: “enderr.com – good name but hard on pronunciation a bit (compare with velenium)”

Completely agree especially for people who speak with a lisp.

Igor: “nerrvana.com – I would like to check trademark”

Hmm, interesting name.

Igor: “velenium.com – easy to pronounce and associated with Virtual Selenium”

Also not bad.

Dmitriy: “We also had: deepium.com, deelium.com, deesbug.com, bugspell.com”

Comparing with nerrvana.com and velenium.com they definitely loose.

This concludes our naming chronicles and discussion switched to the logo, and this, as they say, is another story. Somehow without even voting, we have registered nerrvana.com and started using a new name – Nerrvana.

Of course, at launch each virtual machine is identical to the template including the inherited set of the network settings. All virtual machines run on the same subnet, and therefore they should not use the same IP obtained from the template image to prevent conflicts. That is, each machine should get its own IP. It would seem that the solution is simple – use a DHCP server and dynamic IPs.

However, there are other options which I will discuss in this post.

1. Why do we need virtual machines?
2. The problem of duplicate IP addresses when running multiple VM clones
3. Encoding IP into MAC address
4. Allocating IP by MAC: DHCP
5. Allocating IP by MAC: mac2ip script
a) Linux
b) Windows
6. Final notes

1. Why do we need virtual machines?

Our project, Nerrvana, executes functional tests for web applications in different browsers. Tests are run with a special framework for functional testing – Selenium, which can simulate user actions inside a browser (clicks on elements, mouse movements, data input, text reading), make screenshots of web pages and some other stuff.

A web application test is a sequence of actions emulating user behavior, and comparison of their outcomes with expected ones. For example, a simple test – a login. You need to open the login page, enter your login and password, press enter, and make sure we’re logged in – say, seeing the standard welcome message which includes username. Everyone knows that browsers can quite differently display and work with the same page, and so it makes sense to perform the same tests in the most popular browsers. As I already mentioned, our system does just that – it launches Selenium grid in a virtual environment and executes tests.

Tests in the selected browsers and operation systems are run simultaneously and completely independently from each other. Execution of a test suite in one environment was given a funny name by us – speck. If I want, for example, to test login functionality inside IE8 and FF3.6, our system will create two independent specks and will launch tests using selected browsers. Not going into details too much, I should say that for every speck we create at least two virtual machines. One is a “hub”, will execute tests. It can launch tests in Java or PHP we support for now. The second machine is a “tester”, running Selenium RC, OS and the browser of your choice. Selenium RC is an interaction point between tests executed on the “hub” and a browser.

2. The problem of duplicate IP addresses when running multiple VM clones

We have a template image for each type of a virtual machine. When the core of the system requires another machine, this image is copied, and runs. The template image itself remains unchanged. Thus, many copies of essentially the same machine can be up and running at the same time.

And here a problem arises.
Naturally, at the moment of launch each virtual machine is identical to the template. All settings are inside an image, which can not be altered. This obviously includes a network setup. Since all the virtual machines run on the same subnet, it is clear that they must not use same IP, obtained from a template machine to prevent conflicts. So, each machine must dynamically obtain its own IP. It would seem that the solution is simple – to use a DHCP server, which will give each machine a unique address.

However, not all that easy. The fact is our core actively interacts with virtual machines. It plays an active role keeping virtual machines as slaves and doing a lot of work with them: makes sure all virtual machines (VMs) in a grid have successfully started, starts Selenium RC on all testers, loads and executes code on the hub, monitors tests execution, and then gets back results (screenshots, logs, etc.) before destroying VMs. System core, anyway, need to know VMs IP addresses which were just launched on its’ request.

We had two solutions in mind:
1) Once the virtual machine is booted, it gets a random address from DHCP, and then somehow registers itself with the core by saying: “I am – a machine of a this type. My IP, received from the DHCP, is this”.
2) Somehow, the core tells VM what address it should use. Here the core knows VMs’ IP before it boots.

The first approach seemed to us quite difficult to implement for several reasons. It required some shift of responsibilities from the core to VMs (we wanted to keep them as slaves), problems with matching requested and registered VMs as well as some inconveniences in changing core components.
The second option was more appealing to us, because the core has full control over the VM from the point of its creation without an uncertainty period while waiting for the VM to boot and register itself from option one.

3. Encoding IP into MAC address

How could we pass an IP to a starting VM? We have found the following way: when you start a virtual machine, you can specify a MAC address of the virtual network adapter. A virtual machine itself can read it any time. Our ‘invention’ was to use MAC address as a carrier of information about the IP (could be anything you want).

How we encode IP into MAC address? We use Xen for virtualization, and its virtual NICs must have MACs assigned which look like this: 00:16:3E:XX:XX:XX, where 00:16:3E – network adapter manufacturers’ code. We can use last three bytes at our discretion (remembering, of course, the fact that the MAC must be unique). Suppose that our system will be set on the subnet 10.0.0.0/8. A logical decision in this case will be to use for the last three bytes of MAC corresponding to the last three bytes of IP addresses that we want to encode. For example, to start VM with IP 10.1.1.3, we start VM with MAC 00:16:3E:01:01:03.

Now we only need to teach the VM to get an IP from its MAC. Again, this can be done in two ways.

4. Allocating IP by MAC: DHCP

The first way is pretty obvious. We will run a DHCP server, which will be configured to give IP by MAC-address in accordance with the described above method. Template image must be configured with DHCP enabled.

Work scenario will look like this:
1) The core needs to start a VM.
2) The core looks at IP-addresses pool, chooses the first unoccupied IP (for example, 10.4.0.15), and converts it to a MAC (00:16:3E:04:00:0F)
4) The core makes the template image copy and prepares a VM configuration file (which contains MAC address)
5) The core launches the prepared VM
6) OS inside launched VM requests IP address from DHCP server. DHCP server checks its MAC to IP table and responds with IP 10.4.0.15.
7) The core is pinging 10.4.0.15, and upon receiving a response, continues to work with a virtual machine (in our case connecting via ssh)

Pluses and minuses:
- requires a DHCP server
- if we move to a different subnet or get a different pool of IP-addresses, we will have to reconfigure not only the core but also a DHCP server
+ uses a standard approach to dynamic IP allocation

5. Allocating IP by MAC: mac2ip script

The second approach is less obvious and requires some extra work. It lies in a need to create an extra script which will be injected into OS start-up process. It will get a MAC address, convert it to an IP, and assign it to a network card. Work scenario will be the same with the exception of point 6 which will read:

6) OS’s start up script inside launched VM will read MAC, convert it to an IP address and assign it to VM’s network interface.

Pluses and minuses:
+ no need for DHCP server and MAC to IP conversion table maintenance inside it
- each type of OS will require own implementation of mac2ip script

We implemented the mac2ip approach. We are working with virtual machines running CentOS 5.6 and Windows XP PRO SP3, and so we had two scripts mac2ip – for each of the systems.
Let’s look at both scripts.

a) Linux

#!/bin/bash
 
# first byte will always be equal to 10
IP1=10
 
IFCFG=/etc/sysconfig/network-scripts/ifcfg-eth0
NETWORK=/etc/sysconfig/network
 
case "$1" in
*start)
;;
*)
exit
;;
esac
 
# get MAC and validate it
MAC=$(ifconfig eth0|grep HWaddr|awk '{print $NF}'|grep ^00:16:3E)
if [[ -z "$MAC" ]] ; then
echo "Can't determine MAC address" >&2
exit 1
fi
 
# convert MAC to IP
set -- $(echo $MAC|awk -F: '{print $4,$5,$6}')
IPADDR=${IP1}.$((0x$1)).$((0x$2)).$((0x$3))
 
# change network inreface
sed -i -e "s/^IPADDR.*/IPADDR=$IPADDR/" $IFCFG
sed -i -e "/^HWADDR/d" $IFCFG
sed -i -e "s/^HOSTNAME.*/HOSTNAME=localhost/" $NETWORK

Now force this script to run before the network starts.

b) Windows

As you expect the same work on Windows XP is done in more arcane ways. Perhaps, over time, we will find a better way to convert MAC to IP.

The first problem I encountered is that it is relatively hard to change the interface BEFORE it will be enabled. Thus, the template VM must have default networking turned off by default as otherwise two or more simultaneously launched VMs running Windows will conflict with their identical IPs (it is static, because we are not using DHCP).

Ok, this is not the problem – we can turn off network interface. However, the utility getmac, which we will be using to get MAC address can not return it for the turned off interface! To work around it we will assign a random interface IP first, turn interface on, get MAC, and finally assign a correct IP.

We used devcon utility to manipulate devices.
Here’s how it looks:

@echo off
SET MAC=
SET IP=
SET MASK=255.255.255.0
SET GATEWAY=
 
rem get random IP. This IP will be used for a few seconds,
rem and this is why it is good enough to random generate two last bytes.
rem 100 - shift, to assure that we will not use any used IP.
set /A TEMP_THIRD_BYTE=100+%RANDOM:~0,2%
set /A TEMP_FOURTH_BYTE=100+%RANDOM:~0,2%
set TEMP_IP="192.168.%TEMP_THIRD_BYTE%.%TEMP_FOURTH_BYTE%"
set TEMP_GATEWAY="192.168.%TEMP_THIRD_BYTE%.1"
 
rem set network interface params
netsh interface ip set address local static %TEMP_IP% %MASK% %TEMP_GATEWAY% 1
 
rem turn on interface. We use this to do it -
C:\devcon\i386\devcon enable *VEN_10E*
 
rem get three last bytes from MAC
FOR /F "Tokens=4-6 Delims=- " %%d in ('getmac^|find "Device\Tcpip_"') do (
set /a dec_d=0x%%d
set /a dec_e=0x%%e
set /a dec_f=0x%%f
)
 
rem set real IP and gateway
SET IP=10.%dec_d%.%dec_e%.%dec_f%
SET GATEWAY=10.%dec_d%.0.1
 
rem change interface params to real ones
netsh interface ip set address local static %IP% %MASK% %GATEWAY% 1
netsh interface ip set dns local static %GATEWAY%

Add this script into start-up sequence – for example, this way:

reg ADD "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run" /v "mac2ip" /t REG_SZ /d "c:\init\mac2ip.bat"

After these scripts completion a VM will receive an IP address specified by the core and will be ready to work. The core will know about it from ping replies from an assigned IP.
Perhaps, this diagram will add some clarity (click for a full-size picture in a new window):

6. Final notes

Our approach with mac2ip script is rather slow on Win XP. The current implementation does the job in 15-20 seconds. You understand our feelings if we will say that same 15-20 seconds required from starting OS to a moment mac2ip starts. Script doubles time to launch a VM.

However, we aren’t in a rush to start using DHCP method (DHCP with binding IP to MAC), because:

- firstly, we know for sure that Linux will boot via DHCP slower than working version with a static address. Will the Windows boot faster with DHCP comparing with mac2ip script? Hard to say. We need to check it out.

- secondly, VMs are not shut down properly in our system. We do ‘virsh destroy’ for the VM which is similar to disconnecting power for a physical host. This saves us tons of time as we loaded all the data we needed from VMs and do not care about their integrity. We do not reuse VMs and they will be immediately deleted after use. So, DHCP leased VM address will not be released immediately. It will happen after default-lease-time according to DHCP etiquette. This means that we can not immediately start the VM with the same MAC (and same IP). Setting default-lease-time to a small value (seconds) is not a good idea. A more realistic option – to look at and use OMAPI/omshell to manipulate DHCP and remove unwanted entries straight after ‘virsh destroy’ing a VM.

- in third – any new approach will bring its own hidden pitfalls.

So the slow work of the Windows script in the current version is not a good enough reason to switch to DHCP.

What’s in my name? It’s soulless,
It shall expire, like the dismal roar
Of waves that hit the distant shore, –
Like nighttime noises in the forest!

Alexander Pushkin

In this post we’ll talk about dotProject – a web-based open source project management system. We have used it from 2004 to the present day (seven years). You may consider this post as a mini-guide on the use of dotProject, based on the experience of a team of four developers who build web applications for businesses. This post is quite lengthy as we wanted to share our experiences and avoid creating just another impractical review.

It is hard to remember now, amongst which systems I had to choose. I remember that I found dotProject and eGroupWare. Today, according to Wikipedia, there are tons of such systems. In 2005 the choice was much smaller and the level of their maturity much lower. I have installed dotProject and eGroupWare, and I liked the dotProject – it had all the features that were needed, and even more. If you’ve read some of my previous posts, you know that I am not a supporter of services when you have to pay regardless of whether or not you are using the service, where the data can not be easily migrated to another provider. In general, I like lock-in free services and respect people who create them. It is not easy. I want to be able to leave without loosing data or having a headache because of data conversion. In this case we are talking about the data stored in the database and the files that are attached to projects, tasks, etc. Otherwise, I feel insecure and dependent on the service provider. If we talk about project management systems, then this freedom is only given to you by an open source system installed on your server, developed (preferably) on the language you are familiar with. This gives you almost unlimited possibilities – you can dig up and tweak anything. Rule is simple. If you do not want to tweak – you can live without it. That’s why I chose dotProject.

Much later I decided to look at the two most frequently mentioned project management systems: Basecamp – with an emphasis in simplicity, and FogBugz – with a bias in bug-tracking, and I must tell you that even if they were free and downloadable (not SaaS) systems, then I would still chose dotProject. I did not see anything terribly missing for my work in them. FogBugz differentiates itself with evidence-based scheduling. It can not be used in our work – often project undergo a tangible change during development process, so I do not need to know the probability of a new version release by a certain date. Basecamp – a good system, with minimal functionality, but there is no forum. Posts (messages) in the project can be grouped using categories. Categories looks like a tags. For example, ‘design’. If you click ‘design’ you will see all top level posts in this category with their bodies. We find it quite unusual. Conventional forum will show only subjects which are easier to absorb (imagine you have 30 posts in ‘design’ category). You can not browse all messages in all projects. You need to switch between projects which we found inconvenient. But this is just us – both products are very successful which tells us we all have different projects to manage with different views on systems to use.

What would I like to tell about dotProject – system we use?

First – we are using an old version 2.0.4 (current one 2.1.15), and only basic dotProject modules. No add-ons. And later in the post you will see that for us (and perhaps for you), even half of the standard modules as well as half of the form fields attached to projects and tasks are not needed. How are we dealing with it? We had two choices – modify dotProject or simply do not pay attention to it which we selected. We have done one or two upgrades, and then just fixed the bugs and added a few things we needed. Unfortunately, the dotProject team did not want to use our code, we offered more than once. The system then met our requirements, and only in 2008 we started using phpBB for discussions in parallel with dotProject forum and later abandoned dotProjects’ forum completely. Why? On one hand, the increased complexity of projects dragged amount of posts in some forums to a thousand and more. Not an issue for phpBB, but not for dotProjects’ forum. We had to spend time to optimize SQL, since loading a page in busy forums was taking 10-20 seconds. On the other hand, in mid-2008 we began to work on our own product – Nerrvana – in dotProject but we decided to try using phpBB instead of internal dotProjects’ forum. Soon it became clear that it was a great idea. Therefore, we advise you to use dotProject for project tasks, time tracking and specifications, and a forum (of your choice – phpBB, vBulletin, Vanilla) – for discussions. Later we needed to create forums and provide access to our client. phpBB allowed us to do it with ease. I would hesitate to show customers dotProjects’ forum. It does not have an advanced editor (rich editor) and the ability to insert images etc. Learn more about the advantages of using a specialised forum platform for project management and collaboration in my next post in this series – “How do we use a forum”.

Why did we need to provide access to a forum for our client, you may ask. The answer is simple – work on projects with customers via e-mail box is obsolete. Employees are changing, and in a year or two no one remembers why a decision was made. The one who wants to “change the world” often does not know why this world was built that way in a first place. Clients manager does not know why the work on the specification stopped. A forum, in contrast to an email, gives answers to all these questions. I did not like the lack of permanent storage for requirements, questions and answers on the project. I wanted to have a place where a new clients’ employee even in a year or two after application launch could easily jump start and be up to date about decisions that were made a long time ago before making suggestions about changing an existing application. A place I can use to get an old project and re-read all by myself if necessary. When our discussions and communication with the customer is in one place, it is very convenient.

The modules we actually use are Projects, Tasks and Files (fig. 1 num. 1). Other modules – Calendar, Companies, Forum and Contacts are very easy to hide. We do not use them, but as you see, they are not hidden. If you have many clients, use the module Company as an additional filter to view projects and tasks for each client. Tabs for managing users (User Admin) and system (System Admin) are visible only to a user having administrative rights. If you do not want see them – create a separate account for an administrator, and exclude them from your profile.

Fig. 1 Projects
(click to enlarge)

As you can see, we have launched 176 projects since 2004 (fig. 1 num. 2) and completed 158. When a project is completed, we keep it for some time in the status Completed, then we set it Inactive flag which moves a project to Archive tab making it automatically invisible on all other pages, which we’ll talk about further. It is very convenient to be able to get a project from the archive when additional modifications are required. You remove Inactive flag, create a new task and immediately get access to the file specifications, links to the forum. Development space-time continuum stays uninterrupted. It is very easy to add a new status/tab in dotProject if you need it. For example, Ongoing status was added by me. I wanted to store some projects separately from the InProgress tab.

Figure 1 shows projects view. Lots of information, but we only use a combination of status (depends on tab you selected) and a project name on this page. Neither the percentage of completion, or columns starting from ‘Start’ are not used by us. I highlighted project ‘Misc’ (fig. 1 num. 3), which, as you see, was created in 2007 (fig. 1 num. 4) and contains 143 tasks, where 72 are assigned to me (fig. 1 num. 5). Project ‘Misc’ is used for everything we would like to improve in our application. Sometimes it requires additional research we have no time to do now. Tasks are taken and completed in this projects when we have spare time doing business components (the ones visible to our client) or a critical mass is reached and completing some tasks will save us a considerable amount of development time. For some reason this view does not show up an amount of uncompleted tasks but total versus yours regardless of their status. In fact we have 22 unfinished tasks in this project – large number of them also means that we are passionate of things we do too.

How do we decide when to create a project, and when a task? It’s simple. If we will have subtasks, it is a project. If not – a task. Each new application in our system is a project (in our system, applications are added to the platform similar to SalesForce).

If we follow a project name link on Figure 1, we will see project details (View Project) page shown below.

Fig. 2 Project details
(click to enlarge)

I must admit it is full of fields we do not use too. I think it is better to have more than you need and I can say it very easy to ignore stuff you do not need. For example, I just do not notice the fields (fig. 2 num. 7) and columns (fig. 2 num. 3). Tab ‘Forum’ is visible (fig. 2 num. 1) because a lot of old discussions are in dotProjects’ forum, and therefore we can not hide it entirely. The project, shown on Figure 2, is new (created after we started using phpBB as dotProjects’ forum replacement), so you will see a link on our separate internal forum (fig. 2 num. 1), where we discuss sparklines. You see that a project contains five tasks. Details of each tasks can be looked at by clicking on the links (fig. 2 num. 5). If you worked on a specific task and want to update a task log – it is easy ​​(fig. 2 num. 2). Completed tasks are easy to hide, using a filter (fig. 2 num. 6). ‘Gantt Chart’ tab is a feature we also do not use, but the ‘Files’ tab is a smooth transition to the files of the project. You will find specifications, screenshots in there.

If we follow a link marked with an arrow at the top of Figure 3, we will end up on the ‘Tasks’ page.

Fig. 3 Tasks
(click to enlarge)

By definition, dotProject displays all your unfinished tasks on this page. If you have too many of them, you can set a flag called Pinned for some tasks, as I do, and use pinned tasks filter to see only those that are directly in the work. Right here you can add an entry into the task log (fig. 3 num. 2), or go to the task itself using the task name link (fig. 3 num. 3). To show you what information we actually use I again marked the fields that we do not use (fig. 3 num. 1).

Now let’s look at the task page (View Task). I marked the fields that we do not use (fig. 4 num. 6) following already established tradition. Number 1 is showing a field, which can be used to link task and forums’ topic for discussion. Number 4 – total hours worked; an important and real thing. Progress – quite a subjective measure, to me, especially when halfway it turns out that you need three times more functionality and not the one discussed when you started. Even without changes in the task anybody can write 95% and get stuck in some place for three days. So we have all our requirements dynamically changing often, and we do not rely on estimates of completion percentage at all. A new task log can be added in two ways (fig. 4 num. 3). Number 2 shows the Files tab. Place where they really are – but only those which where attached to this task not to the entire project. File can be attached to a project or to a task in dotProject. Logically the Files tab on project view (fig. 2) will show all files attached to a project as well as all files attached to project tasks. For example, if a task is a bug, files attached to this task will be – screenshots with instructions on how to reproduce this error. dotProject developers created plenty of fields to store URL (two fields for a project, one for a task). Even each task log can include a URL (fig. 4 num 5). In task logs you can use URLs to link with a forum too. You will agree if you were discussing how to do something, somewhere in the middle of the topic and finally did it, then repeating the same thing in a task log will be boring and time consuming. It is much easier to write short – “Implemented – see the link”, and include a URL linking the task log update with a specific post in the forum with the details of implementation.

Fig. 4 Task details
(click to enlarge)

And finally, let’s look at a project files page.

Fig. 5 Project files
(click to enlarge)

We practically do not use the link labeled fig. 5 num. 1, because if you open it in our system, you’ll see a paginated page showing the first twenty or so files with mark “693 File (s) (24 Page (s))”. Yes, it contains a convenient filter to display files by project, but there is a better way – to open a project and ‘Files’ tab on it. This page is shown on Figure 5. dotProject is equipped with a primitive, but quite handy version control system for files. If I need to change a specification I checkout (fig. 5 num. 2) it. dotProject notifies all those involved in a project, that document will change. This is perhaps most important, since it alarms everyone to delete local copies of this document. Version control is typically needed for specifications. When you load a new version into dotProject it does not remove the previous ones. Older versions are available by clicking on a number showing total count of file revisions (shown by arrows on fig. 5).

The project, shown on Figure 5, is a good reason for us to talk about dotProject’s connection with the forum. Numbers 3 and 4 mark two links. One of them points to our internal forum for this project. Another one points to a forum created to discuss the project with the client.

Why do we have a separate forum for a client? First, to ask the client one question we often need a dozen of posts to discuss it. Why to load clients with the information that might be misunderstood or confusing? Secondly, Vadym is in the U.S.A, I am in Australia, and Dmitriy and Alex are in Ukraine. On the one hand, the lack of direct communication, on the other – all discussions are recorded. It is a fact that writing takes more time so we all are forced to write to a point only. What more could you want? Third, as you probably guessed, our native language – Russian, but not our clients. It is easier for us to discuss things in our native language.

Thus, since the beginning of using phpBB, we always create two forums for a project. Each task usually requires a new topic. Topic name could be different from the task name, but should clearly relate to the topic and task. However sometimes a task requires several topics. For example, there is a task – “Specification”, and it is immense. I’ll call topics as: “Specification – add/edit/show audit pages” or “Specification – email notifications – 3 types”. Therefore, if you open a forum for this project, you will always clearly see specification topics, without even thinking that a specially selected prefix connects all of these topics with a specific task in a project management system. I have tried to show all of this in the diagram below.

We can have more than one topic to discuss a spec and we do not even need to link because names of tasks and topics speak for themselves
Forum-category (container for other forums)
For own product you do not need a respective forum as there is no client to discuss
Link between Project and Forum
Link between Task and Topic
Hint
This forum is linked to dotProject project but its topics are not
No structure here. We need to ask – we start a new topic

(click to enlarge)

Forum-categories in phpBB conveniently divide the internal (only us) and external (customers and us) forums.

Finally, I would like to provide some statistics from our dotProject setup.
Projects¹ – 176
Tasks² – 997
Task logs³ – 4909
Total hours⁴ – 23958
Total posts in forum before switch to phpBB⁵ – 8247
Files⁶ – 828
Files total size⁷ – 135.5 Mb
Db dump⁸ – 16.8M