britxardo

Broken buckets in Eucalyptus

2010-04-14T23:19:00.003+02:00

I have experienced an issue with Eucalyptus Walrus when uploading big data chunks, which has at some point failed without finishing and left the bucket in an inconsistent state, so that when I try to list its contents I get an error message such as:

dd1e48ad-45ea-44e3-9227-e9fd67309ab0adminBukkitListBucketTypejava.lang.NullPointerException

The steps I followed to fix it:
a) Stop both Cloud Controller and Walrus
b) Physically remove the bucket from the Walrus storage
c) Remove the affected entries from the CLC database
d) Restart both components

The tricky one is fixing the database, which I did following these steps:
On the CLC server the Eucalyptus database files are in:

/var/lib/eucalyptus/db

the .script files contain the data to be loaded to the in-memory database at CLC startup time.
The relevant file for buckets is eucalyptus_walrus.script, where certain entries must be removed:

1.- Look for the entry where the affected bucket is inserted and remove it:

INSERT INTO BUCKETS VALUES(10,'image-bucket',0,'2010-03-05 18:16:29.965000000',FALSE,FALSE,FALSE,FALSE,FALSE,'US','admin')

the first value in the INSERT is the bucket ID (10), which we'll need later.

2.- Look for the entries where the objects in the affected bucket are inserted and remove them (recognisable by the name of the bucket in the INSERT):

INSERT INTO OBJECTS VALUES(16,'image-bucket',NULL,'application/xml','c0a490cf7d28955fd3e8d0b249c64c60',FALSE,FALSE,FALSE,FALSE,'2010-03-05 18:16:31.012000000','debian.5-0.x86-64.img.manifest.xml','debian.5-0.x86-64.img.manifest.xmlLCc2jw..','admin',6293,'STANDARD')NDARD')
INSERT INTO OBJECTS VALUES(17,'image-bucket',NULL,NULL,NULL,FALSE,FALSE,FALSE,FALSE,NULL,'debian.5-0.x86-64.img.part.0','debian.5-0.x86-64.img.part.0nOsjWQ..','admin',NULL,NULL)

The first value in each insert is the object ID (16 and 17), which we will need later.

3.- Look for the entry where access is granted to the bucket (recognisable by the bucket ID) and remove it.

INSERT INTO BUCKET_HAS_GRANTS VALUES(10,26)

The insert goes into the BUCKET_HAS_GRANTS table and the first values (10) is the bucket ID we found out earlier. The second value is the access
granted ID (26) which we will need later.

4.- Look for the entries where access is granted to the objects (recognisable by the object IDs) and remove them.

INSERT INTO OBJECT_HAS_GRANTS VALUES(16,27)
INSERT INTO OBJECT_HAS_GRANTS VALUES(17,28)

The INSERTs are into the OBJECT_HAS_GRANTS table and the first value is the Object IDs we found out earlier. The second values are the access
granted IDs (27 and 28) which we will need later

5.- Look for the entries where access rights are defined for the bucket and the objects (recognisable by the access grant IDs) and remove them:

INSERT INTO GRANTS VALUES(26,TRUE,TRUE,TRUE,TRUE,NULL,'admin')
INSERT INTO GRANTS VALUES(27,TRUE,TRUE,TRUE,TRUE,NULL,'admin')
INSERT INTO GRANTS VALUES(28,TRUE,TRUE,TRUE,TRUE,NULL,'admin')

The INSERTs are into the GRANTS table and the first values are the access grant IDs we found out earlier.

(Also posted in the Eucalyptus forum)

1752

2010-03-30T13:00:00.000+02:00

The change from the Julian calendar to the Gregorian calendar took place at different times in different places. In 1752 it was applied in the US, meaning that the day after the 2nd of September 1752 in the US is the 14th of September 1752.
Type 'cal 1752' in any UNIX-like system and take a look at September...

Source IP

2010-03-24T00:16:00.000+01:00

Today we had a discussion regarding source IP NATting. On the one hand personally I don't like relying on the source IP of a request for anything, let alone authentication, but on the other hand I think it is good to have the source IP of a request on your logs.
Load balancing is the disruptive feature, it hides the real server IP from the client but imposes restrictions on the internal network setup...

Twitter

2010-03-24T00:01:00.000+01:00

Of all those "social" networks around Twitter seems like not that bad, so I've just decided to give it a try, created an account and added the updates here.
If I remain as lazy as lately you won't be seeing many updates, though...
You can follow it here!

How to do the same on CentOS

2010-01-07T11:26:00.000+01:00

This is related to this change. I have done the same on CentOS 5.4 and there are a few differences but essentially the modifications are the same.

The root.war file is on the same location and the change to be done is the same as in Ubuntu (remove the confidentiality requirement). The funny thing is that there is another file with the same content as the modified web.xml in etc/eucalyptus/cloud.d/gwt-web.xml but it doesn't seem to have any effect (modifying root.war is enough)

The Jetty connector change is a bit trickier, though. In this setup (I installed from RPMs) the Jetty configuration file (eucalyptus-jetty.xml) is not within the configuration files in /etc/eucalyptus but inside one of the Eucalyptus JAR files:
/usr/share/eucalyptus/eucalyptus-www-1.6.1.jar
This file must be uncompressed, the eucalyptus-jetty.xml modified as in Ubuntu (replacing the HTTPS connector with and HTTP connector) and compressed back.
I don't especially like this setup since it makes it more difficult to fine tune Jetty not only for this HTTPS/HTTP change but also for some other connector parameters that you might want to modify in order to do some fine tuning.

How to disable SSL in the Eucalyptus web interface in Ubuntu

2010-01-06T22:46:00.021+01:00

I recently installed Eucalyptus on Ubuntu 9.04 to do some tests, and the first thing I tried was to disable SSL in the admin interface.

Why would I want to disable SSL and use plain HTTP? You might wonder... The reason is quite simple, from the beginning I found a bit strange that a system such as the Eucalyptus Cloud Controller, the "heart" of Eucalyptus, was intended to run exposed to the Internet.
Coming from a traditional three-tier architecture mindset my first thought was that such a system, handling user credentials, business logic and many other "sensitive" topics would best fit into a backend - secure backend kind of setup.
As it is today the system looks quite tightly coupled, therefore a "secure backend" with a database, LDAP or similar seems not so easy to achieve without a deeper knowledge, but I though that a frontend - backend approach should be easy to apply.
In any case the modularity introduced with release 1.6.1 allowing a separate installation for each component and some comments in the Eucalyptus forums make me think something might happen in that area in the future.

Therefore I went for a frontend - backend approach, where the plan was to setup a frontend layer using Apache in the frontend performing the SSL offload (with my own certificates) and acting as a reverse proxy towards the Eucalyptus setup in the backend.
I setup a standard reverse proxy configuration in Apache, taking care of the SSL listening on the standard 443 port and the only thing left was to remove the SSL layer from Eucalyptus and have it listen to plain HTTP on its standard web GUI port 8443.

In order to do that I just edited a couple of files from the standard Eucalyptus installation:

1.- /etc/eucalyptus/cloud.d/eucalyptus-jetty.xml

In this file I removed the Jetty SSL connector and replaced it with an HTTP connector:

Removed

<Call name="addConnector">
<Arg>
<New class="org.mortbay.jetty.security.SslSelectChannelConnector">
<Set name="Port">
<SystemProperty name="euca.www.port" default="8443"/>
</Set>
<Set name="maxIdleTime">30000</Set>
<Set name="keystoreType">pkcs12</Set>
<Set name="truststoreType">pkcs12</Set>
<Set name="keystore"><SystemProperty name="euca.var.dir" default="conf"/>/keys/euca.p12
</Set>
<Set name="truststore"><SystemProperty name="euca.var.dir" default="conf"/>/keys/euca.p12
</Set>
<Set name="password">eucalyptus</Set>
<Set name="keyPassword">eucalyptus</Set>
<Set name="trustPassword">eucalyptus</Set>
<Set name="ThreadPool">
<New class="org.mortbay.thread.QueuedThreadPool">
<Set name="minThreads">10</Set>
<Set name="maxThreads">25</Set>
<Set name="lowThreads">5</Set>
<Set name="SpawnOrShrinkAt">2</Set>
</New>
</Set>
</New>
</Arg>
</Call>

Replaced with:

<Call name="addConnector">
<Arg>
<New class="org.mortbay.jetty.nio.SelectChannelConnector">
<Set name="Port">
<SystemProperty name="euca.www.port" default="8443"/>
</Set>
<Set name="maxIdleTime">30000</Set>
<Set name="ThreadPool">
<New class="org.mortbay.thread.QueuedThreadPool">
<Set name="minThreads">10</Set>
<Set name="maxThreads">25</Set>
<Set name="lowThreads">5</Set>
<Set name="SpawnOrShrinkAt">2</Set>
</New>
</Set>
</New>
</Arg>
</Call>

2.- /var/lib/eucalyptus/webapps/root.war

I uncompressed this WAR file and extracted the /META-INF/web.xml file which I then edited to comment the following section:



Then I compressed back the WAR containing the updated web.xml and put it back where it belongs.

After those two changes I restarted the cloud controller and voilá! Eucalyptus is now serving the web admin interface over HTTP and the SSL offload is performed in a frontend Apache.

Then I configured Eucalyptus (first time admin login) with the frontend DNS name as "cloud IP". Although the text in the form only talks about your cloud controller's IP I wrote the DNS name and didn't have any problem, so I guess it works just fine.

Since the credentials seem to have some dependency on this last parameter I find a DNS name much more convenient than the IP itself.

Anyway, I've just started testing it and although the web part works I might run into issues later on, time will tell.

AdSense

2010-01-05T23:36:00.002+01:00

I just found out that AdSense registration might be trickier that expected, when trying to login to the account I created earlier today I get the following message:

"Account Not Active
An AdSense account does not exist for this login, as your application is currently in review. Within a week of your application date, we'll review your application and follow-up with you via email. Once you are approved to join AdSense, you'll be able to log in to your account and get started."

I guess I'll have to wait and see.

Update (11:55 PM): I've just received an email saying that my AdSense account has been approved and it is already active. That was fast!

Analytics

2010-01-05T21:58:00.005+01:00

Keeping the experimentation rush I've just added Google Analytics tracking to the blog, let's see what happens.
It seems that I need to wait 24 hours to be able to get some results, so more to come...

Ads in the blog

2010-01-05T19:41:00.000+01:00

I have added some Google AdSense and Amazon ads to the blog. I know it looks awful but it's just a test, that's not the final look & feel.
I've been wondering for a while how this online advertising thing works, how complex it is to setup an account, how do you track the activity, what kind of money is involved, etc.
Setting up the accounts has been pretty easy, especially for AdSense (since I'm already using a Google service), where my account is said to be pending approval. For Amazon it is also integrated into Blogger, except for the initial registration page, but everything was smooth.
In both cases there are many options such as ad content, size, etc. so it might take a while until I figure what fits best., if i ever get there.
So far the experience has just started, so I can't tell yet but I'll try to keep posting about it.

Urteberri on!

2010-01-05T17:42:00.000+01:00

New year new blog.