ctria is blogging...

Fabric management redesigned!

2012-06-26T19:47:00.001+03:00

People who know me probably know that when it comes to fabric management i'm a fan of Quattor. Quattor is a great tool that can manage nodes from installation part (utilizing pxeboot and kickstart) till fine-tuning service features. While Quattor is a strong tool that could help administering hundreds or even thousands of nodes, it has some weak spots which i'd like to get rid of:

Very steep learning curve
Some operations may be quite time consuming (i.e. doing errata updates)
Use of a custom programming language (PAN) which is usually unknown to even senior admins
Inventory of the assets is based on what you describe and not on what exists

The above usually lead to misuse of the Quattor which in turn leads to unmaintained or difficult to maintain templates.
After a very long time of using Quattor and also evaluating other fabric management tools (i.e. puppet, chef) my conclusion is that while they can help admins to scale their infrastructure, they work pretty much against the way that administrators are used to. Admins are used to ssh and do their stuff while these management systems don't know how to interpret these changes so they simply skip them.
So lets move one step back and redesign it!
What if you had something monitoring your systems and had the ability to upload your local changes from the node to the central repository of your configuration?
I'm thinking of a solution which will have the following features:

A simple PXE image to inventory your nodes (that way you only need to know the MAC address of your nodes before importing them to the system).
PXE and kickstart (or equivalent) configuration to bootstrap nodes
Ability to take-over a pre-installed node (i.e. cloud VM)
Components (per feature) that will run on the nodes and will:

Identify current configuration
Change the configuration
Prepare a feedback commit if the admin wants to push the local changes on the node to the configuration system

A simple tool/daemon that will run on the nodes as a wrapper for the components
All the configuration will be based on a tagging system so that each node will have its own tag (to override configuration) and then a sorted list of tags to include (i.e. a tag called web-server and a tag called production would result a node that will be part of the production web-servers)
The configuration server would store its data on a DB and timestamp every change. Changes should be easily reverted in case it is needed.
The configuration server would provide API interface with read-only access for other tools to consume data (i.e. an dynamic infrastructure monitoring system) and read-write to update the data.

A list of the first components (i.e. before announce alpha release) that i'm thinking of is:

package managing
file generator
daemon handler
user management

And a demo could be the installation of a node from scratch with a couple of users and the ssh service up and running.

So in principal this is going to be yet another fabric management tool but with the addition of feedback from the nodes to the server. From the fabric management tools that i've used i found that Chef is closer to what i'm thinking of and basically because of the "ohai" util that feedbacks information to the server. I'll probably depend on this.

PS: This is just an idea for now so please add comments, implementation will start when time allows. The implementation will be done based on RedHat rules (thus compatible with RedHat Enterprise Linux and its clones) but should be modular enough to be extended to other distributions.

A successful failure

2012-03-31T01:37:00.000+03:00

This post is about a really special failure that i had recently which i find as a success.

About a month ago i had a totally unexpected contact from a nice guy from Mountain View. He send me an email telling me that he is part of Google Staffing team and asking me if i'm interested to join the Google.com Engineering team.

At first i was pretty sure that this was a mail scam and i replied so, but he insisted with a complete description of the Google.com Engineering (a.k.a. SRE) team. I really got excited by that moment and i wanted some more information on how he found me. What made Google be interested in me? It turned out to be my involvement to opensource projects especially about machine deployment (via quattor) and monitoring (via nagios).

We proceeded to an (unfortunately quick) round of interviews. First i had an interview with him as my recruiter, he introduced me to the team once more and told me that there were two groups in that team, one focused on development and another focused system administration. That was a pretty good tip as i couldn't see my development skills on languages like C, Java etc to match Google's expectations. Of course he also knew this so we focused on the sys-admin side. This first interview was successful. He asked me to rate myself in a series of things like programming languages, administrating, scripting etc in a scale of 1 to 10 keeping in mind that 1 is like never heard about it and 10 that i've written some books about it. I'm not good in evaluating myself so shortly after this i was waiting a response like "Thanks Christos, we'll contact you again when you raise some of them", instead he continued with a list of questions on system administrating. To this point everything went well. I answer all (or almost all) of them correctly so we both were happy about the results.

He arranged the next interview about 10 days later with a member of the SRE team. He also sent me some interview tips and wished me good luck. I studied the resources he pointed me which were the google research documents about the infrastructure that (i believe) Google still uses like Google Filesystem, BigTables and MapReduce but also books both on development and system administration. These were some of the busiest but also exciting 10 days i had as i continued my normal work while i was studying things during night. I think at that point the Google dream was planted: "I will work on all these! I'll be a googler!".

The days passed and i had the second interview. This was with the SRE team member. He first made sure that i was expecting this call and then i totally lost him. He had proceed on the interview but i was still at the "HEY i'm talking with a Google engineer!"-world. At that point i introduced him my stressful side. He asked first about any strong skills that i wanted to outline, i've chosen infrastructure monitoring. We had a conversation on this for about 5 minutes (feeling like about half an hour). Then he asked me some network questions. Nothing real special, but things that normally computer scientists learn in college. It was about network blocks but i was still on "he asks me something, i have to reply soon or ..."-mood. I completely lost it at that point and started answering things that i knew to be wrong (but still wrong is better than nothing, or not?).

Next he proceed to a more familiar question about system load. I answered quickly and was happy that i passed this point till his next question "How would you calculate the system load?". Ok the theory is clear but how you do this in practice? This is totally kernel's job! To make it even worse he asked me to write a "simple" script to do so in the shared google docs we had both open. There my mistakes started one after another:

a) I used bash, awk etc for this. Why this is bad? because you get extra processes running :)

b) I only checked the Running processes from ps output. Completely forgot about blocked ones.

c) In order to get load average, i used expr for the final result, which didn't give me any decimals.

d) Probably many other things that i didn't realize, like use of so many external programs.

We discussed it a little bit and during our discussion i tried to fix some bits that would lead to syntax errors but the scope was already lost.

After this he gave me some time to ask whatever i wanted about Google and interview ended. By the time i hang up the phone every mistake, that at least i know about, came in my mind and found the correct answer (or at least a better one), but it was too late. I hoped that there will be another interview to try to fix things but the next day i got a call from my recruiter telling me politely that i'm not their guy and that they will keep my profile for possible future match. Fair enough considering their expectations.

So why i consider all these as success? Well after the interview i got really disappointed. I felt that the google dream was vanished. But then i got back a month... would i ever submit for an opening in Google? No! They found me! I got contacted by a leading corporation in IT world because of my previous (public) work. My first ever real (outside academic world) job interview was done by Google engineers! And most important this gave me the needed push to realize that what i do is important not only in academic world but also to the IT industry out there.

I no longer feel that the Google dream is over. Google may contact me in future again (given that the percentage of people being dropped at first and hired by the second time is quite high), it may also not. But does it really matter if it would be Google? Knowing that my expertise is in need by such corporations makes my day.

Developing the proper way

2011-07-04T13:01:00.001+03:00

Recently i wanted to submit some patches to the Lustre Filesystem to allow the compilation of the kernel modules (patchless client) for Fedora 14 kernels. I initially posted my patches to the discuss list but i was pointed to the company that maintains the community based Lustre fork (whamcloud).

There i was amazed by the way that code is maintained as it uses all the current best practices that i could think of for developing. Thus i wanted to share them with my undefined readers.

Git repository:
First the code is publicly available on a git repo. People can clone the repository locally, do their modifications and they push them to Gerrit.

Gerrit:
Gerrit is a code reviewing web tool for git repositories which allows people to submit via "git push" patches. Patches are verified and reviewed. In Lustre's case, code verification is done by Jenkins while review is done by at least 2 code reviewers

Jenkins:
Jenkins monitors execution of repeated jobs. In Lustre's case it is used for automated builds of the current master branch + the committed patchset.

Finally when patches are approved they are merged to the master branch.

Could you please nslookup my CRL?

2011-02-08T10:26:00.000+02:00

How do you publish your CRLs?

The common answer to this is via an http(s) URL or from an LDAP server.

If you hosted a CA that serves Grid community you would find out soon that your CDPs are hit very hard. According to our logs we get more than 250 hits per minute in order to download a file sized in most of the times less than 100 KB.

On the other hand Relying Parties to these CAs many times blame CAs for their outages or unavailability and want to have a reliable caching mechanism of the CRLs distributed in the whole world. Currently caching depends on CA's webserver configuration AND at client's willing to cache things. While big sites use squids to save their bandwidth, you still see many requests from the same origin for the same file getting the same response "cache this for 1 hour" but still trying to get a fresh one out of our server.

So yesterday on my way home i was thinking what if we host all the CRLs on a DNS server?

DNS resolvers have a reliable caching mechanism which is spread usually to one per site, the only limitation is how to store this info. TXT records have limit of 1300 bytes (max recommended size by a recent draft of IETF in order to fit in a single 1500-byte Ethernet package). One way to move is to split them. CRLs in PEM format are in base64 form which means that they have 65 bytes per line (64 data + newline) thus up to 20 lines per TXT record. Will this scale?

Our current infrastructure (as part of EGI) supports ~ 100 CAs. The majority of the CAs are covered by less than 10 TXT records. Less than 10 CAs need up to 40 TXT records while we have about 5 CAs that need more than 100 TXT records with the top reaching the 1770 records!

So yes this should cover the needs of the "small" CAs in terms of CRL size but would be unreliable for the "big" ones.

I'm planning to do some testing on this on my next spare time period and report here with my results (hopefully sooner than my last post).

Hex-editing your GPFS Terabytes

2009-09-15T15:31:00.001+03:00

I remember the old times where you could find a those geek guys that would use a hex editor to "patch" your favorite game and become bullet-proof or with 99...9 (still counting) lives or ... do what ever you wanted in order to win. At that time i was sure that hex editors are powerful to save you from a "disaster" but i couldn't think what a disaster can be.

We are using GPFS as network file system for our clusters and except from dummy scratch space for MPI jobs it is also used for some local user's home directory. A local team need to expand their GPFS filesystem so we had to add a few disks to our array. The procedure sounded trivial, adding the new disks to the array, create a new logical volume and finally add the new raw device to GPFS filesystem.

But of course something went wrong. The new volume was about 10TB in size which due to a GPFS limitation we had to partition in at least 2 partitions. Easy work via parted but what happens when parted (for a reason still unknown) "modifies" the partition table of other logical volume which is part of the GPFS filesystem as a whole (without partition table).

Well the parted simply ruins the first sectors of a GPFS NSD which means it ruins all the valuable information (Disk ids and NSD ids as well as the filesystem definition) from this disk. The users report "We are receiving 'Input/Output error' when using the X file" and everything gets worse and worse.

Fortunately there IS a solution to this disaster. Although we couldn't find any official IBM documentation on this (apart from some posts in GPFS's forum), there is a way to recover from this situation. What you need is a hex editor, the famous "dd" and a lot of patience.

First copy the sector 8 from each disk within the GPFS filesystem. This sector is the File System Descriptor and it is common on all disks. Next we have to recover sector 2 and sector 1. Sector 2 is GPFS disk identifier (know also as the NSD-ID). Finally the sector 1 contains information about the disk which is called the disk descriptor.

Due to legal reasons i'm not sure if i'm allowed to reveal more information on how to do this but studying carefully the sectors starting from 8 and going to 2 and then 1 you are able to recover your FS.

Big brother secures our transactions?

2009-09-14T16:15:00.001+03:00

I'm at Germany for a meeting and i found out that i was out of money....

Well i thought this is easy, let's visit an ATM and get some. I went at the ATM after a few random clicks i was able to find the "English" button and i requested my money.

A few more clicks to find the "OK" button at the machine and there i had my money.

After a few minutes i received a phone call. The person introduced to me as member of the security incident team of my bank. Asked me if i got these money from Germany or if my card was stolen!

In one way i felt safe but ... do they really monitor what i do all the time???

I should also mention that all this happened at 3 a.m. Monday morning.

Bye bye SL3! Bye bye gLite 3.0...

2009-07-17T01:27:00.004+03:00

The last piece of gLite 3.0 node was decommissioned this week with the shutdown of node001.grid.auth.gr.

This node was the sBDII/lcg-CE for GR-01-AUTH for long time now and was serving as Torque server for local PBS queues.

A new node (XEN guest node with SL4-x86_64 and gLite 3.1) was setup in April to take over this task but the migration of the local users/queues was postponed till July. The node has already processed tens of thousands jobs.

With this migration we finally achieved the milestone where ALL our Grid nodes are controlled by our Quattor installation.

Semaphore limits...

2009-07-15T10:17:00.007+03:00

Living in a world of powerful Worker Nodes with many cores per node and many GBs of RAM i thought that running "big" jobs won't be a problem.

A strange call reached our helpdesk where a user was able to submit jobs to multiple 2-core WNs but was but was unable to submit a job to a single x-core node (where x > 6). After some debugging the errors that the job was getting was due to a semaphores limit.

In specific the job needed many semaphore arrays (21 per core) where only 128 were available (on our SL4 WNs) and the following error kept appearing:

p4_error: semget failed for setnum: 0

The solution came via sysctl where one can set these limits via the kernel.sem parameter:

# sysctl kernel.sem

kernel.sem = 250 32000 32 128

The limit we were hitting is the last number which is the SEMMNI (how many semaphore arrays can be allocated). Using the following command we were able to adjust this. The magic number we choosed was "512":

#/sbin/sysctl -w kernel.sem="250 32000 32 512"

Migrating to gmail...

2009-07-07T19:17:00.003+03:00

It's been long time since my last post... This post is about a migration i'm trying for my mailbox as lately i'm experiencing some issues with my primary mail account...

It's a strange situation because although my mailbox was migrated to another imap server, which clearly puts the blame at the server, i was not the only one who was migrated but i was the only one that experienced issues. (Using same OS and imap client software).

At the beginning the issue was due to the postfix's default max connections per IP given that Mail.app at os X opens many connections in parallel.

After raising this limit things were better but again it wasn't as it was.... So i though why not outsourcing the imap server work at Google? This is something that a colleague is already doing. I didn't want to migrate all my mails at gmail at once thus i just added the following to procmailrc at my account at our mailbox in order to copy all messages to my gmail account:

:0c:

! myemail@gmail.com

That did the trick and i soon realized how world would be without mail filters...

I started to add filters and "labels" (i really loved gmail labels! MUCH MUCH better than imap folders) and i realized that there is no way to filter my mails using non-standard headers (i did hard work on my imap client to find the best "custom" mail header to categorize my email sources and now this is unusable :( ).

Anyway the feeling after 2 days using gmail in comparison to my primary email is that gmail is much much faster but the spam filtering is not as "educated" as the one i was used to. Of course as the anti-SPAM software runs before procmail i can rely on both Google's and my previous mail provider anti-SPAM assertions.

Another very interesting comment that i have about gmail is that it can find which mails have arrived more than once for my email account (i.e. when i get a reply from a mailing list and the sender is using both my email and mailing list as recipients) and gives me only one of them. I thought that i was losing mails at first but actually all the information is preserved as the gmail recognizes both sources and if i have a filter that adds a label to each source the the mail will have all the labels.

You can do the DB scheme work online!

2009-06-28T19:38:00.002+03:00

I'm working on a project with colleagues from other institutes. Within this project we decided to some work on a client-server model application with a database back-end.

We had many mail exchanges, nice figures to describe workflows, phone meetings, video conference meetings but it was time to start doing some work.

One first thing we had to agree on was the DB scheme we are going to work on and i was surprised by a tool that one of the colleagues used to give us his SQL model. The tool is called wwwsqldesigner and it is open-source. Of course there is a demo installation to use if you don't want to install it your self.

What i liked most is that you are able to get your design in XML format. Then you can send the XML file to the rest developing team who can upload it again either to their local installation or the demo one, do their changes and publish a new version etc... Of course you are able to save your model at the server and then others can just select it from a list in order to view it and change it.

I liked it so much that i'm thinking of installing it locally and upload our local project's schemes.

I don't want all these mails on my iPhone!

2009-06-26T13:49:00.005+03:00

A week ago i was writing on this blog about about the iPhone 3.0 OS update and as a disadvantage i had that there is no mail filters yet.

I really hate it when my laptop is not connected to the imap server (in order to filter all my mails) and i get all these SPAM and mailing list mails on my iPhone. It makes it totally useless.

The first thought i had was to use procmail which seem to be powerful with one "show-stopper" for me. It requires to have access to your mailbox server in order to upload your procmail configuration.

Then i thought to search for a simple client that will connect to IMAP, filter my mails and then logout. The client that i found is imapfilter which is actually developed a Greek guy!

Its operation does EXACTLY what i want. You feed it with an easy to read configuration (LUA):

---------------

-- Options --

---------------

options.timeout = 120

options.subscribe = true

----------------

-- Accounts --

----------------

-- Connects to "imap1.mail.server", as user "user1" with "secret1" as password.

account1 = {

server = 'imap1.mail.server',

username = 'user1',

password = 'secret1',

ssl = 'ssl3',

}

---------------

-- Filters --

---------------

spam = {

'new',

'header "X-DSPAM-Result" "Spam"',

}

----------------

-- Commands --

----------------

-- Get status (messages, recent, unseen) of the mailbox.

-- check(account1, 'INBOX')

-- Move messages between mailboxes at the same account.

results = match(account1, 'INBOX', spam)

move(account1, 'INBOX', account1, 'SPAM', results)

I just setup a cronjob for this and works perfect!

Lets cut some (gLite) Hydra heads

2009-06-25T18:24:00.007+03:00

You may be familiar with the Lernaean Hydra. The complexity of this beast was perfect to name a gLite service that is used to encrypt/decrypt data.

This service is based on the "Shamir's Secret Sharing" algorithm where a the encryption/decryption key is divided to X parts and Y parts of them (where Y <= X) are needed to reconstruct the key.

A requirement for data encryption was raised sometime in the previous years and we had deployed 3 gLite Hydra servers (each one will hold a part of every user's key and only 2 of them would be required for encryption/decryption operations) with clear geographic and administration separation.

A software update to one of them led to a "funny" situation where no new keys were able to be registered and no old ones could be unregistered. (These are the only operations that require all the servers to be up and responding). The tool that was provided to (re)configure the service had the very interesting operation of dropping every DB table and re-create them using the predefined schema.

A re-configuration of the updated server gave us a "everything just doesn't work" state, which we had to resolve under user community pressure. Note that if the service just didn't work, users may have lost lots of human/cpu hours because they are just able to get an encrypted output which they can't decrypt.

Analysis to the DB at another gLite Hydra instance gave us an idea of how this service stores its data. Due to luck the actual keys were not deleted by the configuration script but only the relation between users and keys was deleted.

A copy of the user database and some reverse engineering at the relation DB at a working Hydra instance was enough to recover the service with (almost?) no cost.

That reminded me that common Murphy's law where the backup you have is either unreadable at the time you needed or was last updated BEFORE your critical data was stored.

OpenMP jobs on Grid? (The LCG-CE - PBS approach)

2009-06-20T01:32:00.017+03:00

There was a user support requirement for OpenMP jobs in Grid. OpenMP is a shared-memory implementation which means that all processes must run on the same box.

Well this can easily achieved at PBS side by using the directive:

#PBS -l nodes=1:ppn=X

Where "X" is the number of requested processes. But the main issue is HOW can we get this requirement based on what WMS gives to us on submission?

After googling this, the "correct" solution can only be achieved at CREAM CE where users can select a number of requirements that will not only be used for job matching process at WMS but also passed to the CE. You can find more info on this here.

LCG CEs on the other hand are only getting a poor RSL which doesn't carry almost any of the user's requirements. So lets get in LCG CE's internals...

First a job reaches the globus-gatekeeper. At this phase user's proxy is matched to a pool account. GateKeeper's task is to authenticate the user and the job and pass it to the globus job manager.

The globus job manager uses the GRAM protocol to report the job state and submits the job to the globus-job-manager-marshal which is using a perl module to talk to the relevant queuing system.

This perl module is responsible for the creation of the job (shell script) that will be submitted to the PBS server. In this module the CpuNumber requirement is translated by default to:

#PBS -l nodes=X

So this is the part we need to change in order to create OpenMP jobs. The next issue now is how we find out if user has asked for OpenMP job. I've noticed that the JDL option "Environment" is passed to the job executable that will be submitted thus a definition like the following:

Environment = {"OPENMP=true"};

can do the trick.

The whole above approach works but for sure needs a lot of work but as proof of concept is more than ok...

In the (near) future i would like to test the CREAM CE which, as i said before, has a more clear way to support requirements from JDLs using the CeForwardParameters definition.

Coding on multiple SVN repositories...

2009-06-19T12:53:00.005+03:00

As a developer i use to use repositories (mainly SVN) for code versioning and to interact with other developers.

Involvement in developments from other teams within a project usually require that the (production) repository is hosted somewhere centrally. This give us the advantage of having one code-base where all developers to work. The main disadvantage of this implementation though is that usually developers doesn't commit till they have something really stable and working.

Another disadvantage is that it is not clear to someone outside your mind to find out on what you are working on (and usually "manager" guys need to do so).

I was proposed to use a local repository for every developing i do where it would be easy to have "every change" commits and commit stable versions to the central repositories. This will give us both frequent commits (thus clear history view) and other are able to see on what you are working and probably comment on this work. At first i was highly against this... It's clear that it adds a lot of additional work without giving us many clear advantages.

As this was a "manager's" proposition i had to try it. The initial thought was to work on our local repository and then, when i have something stable, take a diff since last sync of the repositories to apply it to the remote (central repository).

But ... thinking on this again, is it the "svn tagging" procedure having different server for trunk and tags?

An implementation to test:

Creation of a local test repository with trunk and tag trees
Create a new repository to serve as "the remote central repository"
Create a new tag at the first repository which will have an svn:external link to "the remote central repository"
Start tagging as normal on the local test repository but always at the same tag.

iPhone 3.0 OS is here...

2009-06-19T01:50:00.002+03:00

It took me about 8 hours "check for updates" clicking and finally around 8pm on Wednesday it was available!

For the 230 MB download, the first 200MB were downloaded within a few seconds while the last 30 took about half an hour! (was one of the first downloaders?)

About 24 hours experience on iPhone 3.0 and i think that finally there is all the missing "phone" functionality.

Thumps up:

I can write Greek!
iPod shake! (shake to shuffle)
"Search iPhone" or the iPhone spotlight. Everything is about a text box away from your screen.
mms (never used it before but it was a pity that iPhone was unable to do something that 30 euro mobiles do)

Thumps down:

Some apps 3rd party report "compatibility errors" (fortunately without any (visible) malfunction).
Still no background applications (no skype on background)
No email filters (should I consider procmail?)