Problems in the caffeinecirculationsystem

cd under development packages nested folders without repeating yourself

2011-04-10T21:28:00.000+03:00

To me Plone development environment means Fedora 12, terminator and vim. As I do lots of development in the shell there has been one annoying little thing which I'm tired of - cd:ing under development packages inner structure to where the actual code lives.

Doesn't seem much but if you repeat that many times a day you'll get tired of it. Of course you can always launch vim for the package root and open the file you wanted from there, but it's still extra effort.

Few weeks ago me and my colleague Jussi Talaskivi started to think that there has to be a script which takes you to your destination simply by parsing dotted name and using information from there to cd you to the correct place. After spending some time googling around we found none. Annoyed by this we decided to do this missing script by ourself.

Soon it turned out that this wasn't that easy task as it sounds - simple shell or python script where you just parse correct path from arguments wouldn't do the trick as they're launched in a separate process which doesn't have any effect on the working dir of your shell. Little research and try - error approach later we found working solution.

1. Create shell script (eg. pcd.sh) to somewhere on your $PATH which contains following lines:



#!/bin/sh

cd `echo -n $1/;echo $1 | sed -e 's/\./\//g'`

2. Chmod your shell script so that you can execute it.

3. Edit your .bashrc and add following alias:

 alias pcd=". pcd.sh"

Now pcd command should work as expected:



$ pwd

/home/jutaojan/workspace/

$ pcd very.long.packagename

$ pwd

/home/jutaojan/workspace/very.long.packagename/very/long/packagename

Excellent!

The key element is the first dot in the alias. This makes your command to execute in the current shell session. As alias doesn't cope with arguments you'll need a script (in this case pcd.sh) for parsing to work. I know this isn't the most sophisticated piece of code and there probably are better ways to achieve the same result, but I couldn't find one.

I hope this example is useful for someone frustrated for the long nested structures we're all so familiar with.

Plone RPM deployment

2011-01-23T17:03:00.002+02:00

It's been ages since I wrote my last post. I guess now it's time to make that up with post about Plone deployment. In a last few months I've had chance to invest part of my time at work to develop our Plone deployment model and since there hasn't been that many posts about the subject I thought I'd share my experiences for the community.

Before I get to the details here's some background information about the environment I've been dealing with. I'm working at the University of Jyväskylä, Finland. Our university is a heavy weight Plone adopter here in Finland. Plone is in use by every faculty and department. We have about 500 - 700 content managers and about 50 - 60 separate Plone instances. Our front page gets about 2 - 3 million ~~hits~~ page views / month. The important information in previous data is the amount of instances. 50 - 60 is just my estimation about our current status and that amount is increasing every year. I'm sure you got the idea so I'll continue to our previous deployment model which I'm sure is familiar to anyone who has had the pleasure to deploy Plone sites.

Current setup

We've got quite normal setup of RHEL servers, Apache + Varnish combination and loads of Plone sites. Our sites use mainly Plone 3.3.5, but we have increasing amount of Plone 4 sites and also few Plone 2.1 and 2.5 sites. We've used buildout to deploy new sites and update the old ones and as I've mentioned in my previous posts (Managing multiple plone buildouts, problems with plone version pinnging) it hasn't always been enjoyable experience. Don't get me wrong - buildout is awesome tool if you need to put up few sites every now and then and don't need to look after them, but running about 30 buildout scripts within few hours scheduled update window isn't fun. I know many of you think now that why didn't they script that? We did. We also developed tools to update packages automatically to avoid all the manual steps - yet we hit problems in updates, some of them were plain human errors (eg. wrong version pinnings) and some of them were because of some unpredictable behavior of software.

Reinventing the wheel?

Every now and then we've been thinking better deployment story for Plone-sites until we realized that the things we'd want to achieve sounded just like package management software. As we're using Red Hat os in our servers I hold my breath and jumped head first to the RPM world. There isn't too much information about deploying Plone sites as RPM-package on WWW. I knew folks at Weblion have developed environment for Debian based distros, but that didn't help much. Luckily Google revealed that Nikolay Kim from Enfoldsystems had been in talks with Fedora packagers about packaging Plone as RPM. As far as I understood from the mailing list messages this didn't work out well and Enfoldsystems packages never ended up to Fedoras repository due to packaging policy disagreements. Nevertheless Enfoldsystems have kindly published their Plone buildouts and RPM specs in their Subversion repository.

RPM setup

I used Nikolay Kims work as a base structure to fit things to our environment. Nikolay had divided Plone to two separate RPMs. First one (plone-base) contains all the common packages Plone needs. Another package contains the Plone-instance and its eggs. Nikolays packages should work out of the box as they are, but we needed to modify them a bit to suit our needs. I created own RPM-package for the Python virtualenvs - one for Python 2.4.6 and one for Python 2.6.5. I also created RPM-packages for few python libraries (python-imaging, python-lxml, python-ldap) so that they install to virtualenvs instead of system python. With these ready I modified Nikolays spec-files to use my virtualenv packages and modified buildout.cfg file so that it extends to our shared configuration and the initial setup was done. There was lots of manual work to do to add and test RPM-specs for our buildouts but now when it's done life seems to smile at me. To make my workload even lighter and to avoid dull copy-paste work I created custom paster template which creates buildout folder with RPM-spec ready for rpmbuild to do it's magic. I attached the rpmbuild process to Hudson by creating parametrized builds - when our developers includes certain commit message and pushes his changes to Mercurial this launches hudson build, tests and if everything went ok, finally rpmbuild. This is done with the help of hghudson package which I customized a bit to allow parametrized Hudson builds.

What we achieved?

You may wonder what makes RPM deployment better than buildout?

Here are some pros listed in no particular order:

It fits well with the regular operating system maintenance.
Our system administrators can now update servers/packages automatically without any knowledge about Plone.
If something goes wrong we can easily rollback to previous package version.
Deploying new site happens just like installing any other package with package management software.
We can be sure there isn't going to be any trouble with pypi or some other critical site being down when we're updating/creating new instance.
We still have buildout in the background to make developers life easy.
It saves time, nerves and makes it possible for me to once again enjoy Wednesdays (guess which day our scheduled maintenance window is?) :)

I know some of these can be achieved by using buildout as well - eg. by mirroring pypi, using eggproxy etc., but nevertheless my experience is that running buildouts on a production server has always a bigger chance to fail than updating tested RPM-package. I still have to admit that at the moment this deployment model is new to us and I probably won't know all the cons. Here are few which are more related to the setup process of RPM deployment model:

You'll need a way to create RPM-spec automatically.
RPM deployment isn't as straightforward than using buildout.
You'll get most out of the pros only when you're serving several instances.
You'll need to know at least basics about how to create RPM-packages to set up this kind of environment.

I hope this post will be useful to someone struggling with similar problems. I will be posting updates about the subject when our new deployment model has seen more use. I'd also like to thank Nikolay Kim and Enfoldsystems for the excellent work they have done to make Plone RPM-packaging being as easy as it is now. Thank you!

Plone 4, ZEO and supervisor

2010-05-13T14:11:00.001+03:00

This post belongs also to the "lessons learned" category.

With Plone 3, ZEO and supervisor combination you've probably configured your supervisor to start plone instances by running $BUILDOUT/parts/client1/bin/runzope.

Problem is that with Plone 4 your $BUILDOUT/parts/client folder doesn't contain anything else than etc folder. You know starting instances by targeting supervisor to use $BUILDOUT/bin/client1 fg doesn't work like you'd expect (supervisor would control the client1 script - not the actual plone process).

My colleague Jussi Talaskivi figured that using 'console' argument instead of 'fg' for bin/client1 script should do the trick. With 'console' argument stopping, starting and restarting Plone 4 instances with supervisor works like a charm.

Below is full example of working supervisor configuration.

[buildout]

parts = supervisor

[supervisor]
recipe = collective.recipe.supervisor
port = 8200
user = xxxx
password = xxxx
programs =
    10 zeo       /buildout/xxx/parts/zeoserver/bin/runzeo  true zope
    20 instance1 /buildout/xxx/bin/instance1 [console]     true zope
    20 instance2 /buildout/xxx/bin/instance2 [console]     true zope

Plone 4, ZEO and tempstorage

2010-05-12T21:03:00.004+03:00

This is reminder for myself and hopefully for other people too so that they don't have to waste several hours of valuable time looking for this simple piece of information.

If you have experience setting up Plone 3 with ZEO and tempstorage (so that your sessions won't disappear in the middle of hardcore content managing) you might have trouble setting up same kind of setup while using Plone 4.

Thanks to David Glick who enlightened me that plone.recipe.zeoserver doesn't include whole Zope 2 in it's Python environment anymore. In this specific scenario this means that even though you have tempstorage in your eggs folder it isn't included in pythonpath when zeoserver part is being processed and this leads to below error message:

Error: could not load package tempstorage: No module named tempstorage

Package name: 'tempstorage'

File name: 'component.xml'

Package path: None

For help, use bin/zeo-server -h

Solution to this is to add this simple oneliner to your buildout.cfg zeoserver part:



[zeoserver]

...

eggs = tempstorage

...

Usability and Plone

2010-01-24T15:00:00.002+02:00

I've seen in here and there someone mentioning that usability of Plone is very good. Lot's of people - me included - like the fact that in Plone you don't have separate content management interface compared to some of Plones rivals. That counts for something when we're talking about good usability. Still that is only one quite small part of the whole picture. So what else is there? What do people like in Plone and where are the rough edges for end users? If general consensus is that Plone does have good usability, where is the actual proof of that?

On plone.org I found one page in developer documentation mentioning following: "Plone differentiates itself on usability. The intuitiveness of the user interface is what attracts people to Plone the most."

I interpret this sentence meaning about the "one view for all" approach. What bugs me in this is that this whole sentence about good usability is about how the UI works compared to Plones competitors. Saying that our approach has more intuitive UI than our competitors is one thing - proving that our claim is true is another thing. Yet I think the most important thing is missing from here. Proving that the UI we have has good usability in itself. Is the user experience better than our competitors and good enough for our end users or is it actually, really and without any doubt so good that it wouldn't be better without rethinking whole thing and designing even more intuitive UI?

In launchpad.net/plone it reads that "Plone is easy to use. The Plone Team includes usability experts who have made Plone easy and attractive for content managers to add, update, and mantain content."

Good usability is not just that someone says or writes it. You'll need to prove it too - and not only by comparing how our product does against competitors products - getting the end users opinion weights so much more.

Why I'm bringing this topic up? I'm doing my masters thesis about Plone usability and in the past week I've been searching usability studies about Plone to get some background material. When I started this I thought it'd be easy to find at least few through reports - knowing that people have mentioned Plones good usability quite often in separate posts. To my surprise I found only few short blogposts (Plone Usability Makes It As Easy As ABC, Usability of CMS Home Sites) and few evaluation videos, but none of those are scientifically accurate enough to be material in masters thesis. Basically I didn't find any actual usability study made about Plone.

I hope you don't get me wrong in this post. I'm being over critical about this matter as the subject is maybe too close my job and interests. I've been working with Plone about 4 years now and this includes developing new products, giving end user support and administrating sites which gets millions hits/month. I'm trying to forget all this in my masters thesis which should to be scientifically approvable study. If I won't find any proof about the good usability I have to think (scientifically) that it is all just words and nothings been proven - yet.

If you happen to know some study what I've (obviously) missed I'd be more than thankful to know about it.

Speed comparsions between Plone and Wordpress

2010-01-20T22:56:00.001+02:00

Jon Stahl wrote recently a blogpost about Plone being three times faster than Dropal, Joomla and Wordpress. We had a small discussion about this in my workplace and as my colleague pointed out this wasn't a really that comprehensive test that you could state Plone being 3 times faster than it's competitors. This seemed a bit unfair test considering how fast this has been spread in tweet/blogosphere, so I decided to repeat the test with a bit more critical viewpoint.

What's wrong in the original test? No one would consider opinion poll with 10 answers nowhere near trustworthy - it's all the same with requests. I didn't want to put up all the cms so I just set up second best (Wordpress 2.9.1) and compared that to my Plone development site (4.0a3). As a comparsion I did the first test with same ab command Jon used and my iMac gave following results:

Wordpress: 7.13 requests / second
Plone: 17.10 requests / second

So far we have clear winner and Jons data holds up... almost. Anyways.. to do a bit more realistic test I used 100 requests - first with no concurrency and then concurrency value set to 3 (ab -n 100 -c 3 http://...) Results are a bit different:

No concurrency, 100 requests
Wordpress: 7.18
Plone: 16.46

Concurrency 3, 100 requests
Wordpress: 13.04 requests/second
Plone: 16.50 request/second

To add some more requests I got just small differences anymore. With ab -n 200 -c 3:

Concurrency 3, 200 requests
Wordpress 13.09 req/s
Plone 16.37 req/s

Concurrency 4, 200 requests
Wordpress: 13.04 req/s
Plone 16.39 req/s

So when I just added some more requests situation was quite the same, but when we have (in the real world) that much requests in shortest time possible there are bound to be some concurrency. Adding concurrency to my test situation doesn't look that bad for Wordpress anymore.

Even though Wordpress did better job with more concurrency it still got beaten by Plone in my rough testing. Yet it's not that spectacular to be 1.2 times faster than being three times faster. And don't get me wrong - as a Plone developer and administrator I'm really thankful for all the people who have contributed to Plone tuning in one way or another. It's still good to be a bit critical before we claim that good results :)

To be clear, I'm not really familiar with Wordpress or anything which has something to do with PHP. As Jon I also put up MAMP environment and installed Wordpress there using PHP 5. There were no other customizations so I don't know how tuned it was for production use. My Plone on the other hand is my development instance and it was running in debug mode on the foreground so it was in worst possible situation for Plone to run... or well.. I could have enabled very specific logging, but that would have been overkill for this test.

Domain name registration through Google - when things go wrong.

2010-01-18T23:19:00.000+02:00

Not too many people know, that you can register new domains through Google. This can be done when you're registering for Google Apps Standard Edition which is free and somewhat stripped version of their Google Apps Premium Edition. Latter one is tailored more to suit business needs.

With $10/year prize tag it's not cheapest option, but you'll get "private domain registration to protect against spam at no extra cost, full DNS control and domain management, automatically configured to work with Google services, email, calendar, instant messaging, web pages and more also at no extra charge".

As a comparsion GoDaddy offers .org domains at $14.99/year so it's actually not that bad offer. Google actually is just collecting the data and sending it to their partners (godaddy, enome) which does the registration. I decided to give it a try at January 11th. To my big surprise things didn't went that smoothly. It's been now one week since my order - the domain I registered is actually not registered at all - it's up for grabs for anyone out there. All I got was account to Google Apps Standard Edition. After one days waiting I figured out there's something wrong with my order as my domain was freely available and in my Google Apps Dashboard my status was "Not active, pending payment", so I asked Google a status request about my order. I got automated reply stating that my orders payement was pending and I should wait them to process it. I waited one more day and since things didn't get better I sent another mail stating that my order was invalid. Once again I got automated reply without any useful info (except the fact that by replying to that mail Googles personnel should contact me). I replied to the mail stating that there really is something wrong as my payement hasn't gone through. I waited few days for reply which I never got.

At this point all I wanted was to delete my google apps account so I can retry, so I browsed through google help center and forums for information. What I found was that you actually CAN'T delete your account if you're ordered your domain through Google (http://www.google.com/support/a/bin/answer.py?hl=en&answer=56236). Wait what? So.. now I have Google apps standard edition account for my nonexisting domain which I can't delete so that I could try registering again. Also, I can't get any contact to Googles "customer service" to get someone to solve this.

I did a quick search from Google Help forums with words "Not active, pending payment" and got quite a lot of results. It seems that Google staff reads those forums and can "flag" some posts which some support personnel can then pick and solve so I left their forum message describing my case and asking for any help. I haven't got any replies yet.

As a summary about everything above: I'm a victim of extremely automated process which had some hickups when it was processing my order and I'm not too happy about that. I know Googles strength isn't in customer service but in their extremely scalable systems. What pisses me most is that their system didn't work for me.

Based on my experience I really can't recommend anyone registering their domain names through Google.

End of rant.

New Finnish translations for Plone 4

2009-12-22T00:06:00.000+02:00

Today I committed last modifications to Plone Finnish translations to Plone collective. Overall there were quite a lot of things to translate as Finnish translations were reviewed last time over 2 years ago when Plone 3 first time appeared.

Plone has developed quite a lot since that and 3.x branch has had missing translations for Finnish users far too long (and still has - I made translations for the 4.x branch only). One thing what troubles me is that I'm not entirely happy about all the translations I made. For some words I found it damn difficult to try find the same - or at least almost the same - expression from Finnish dictionary. The expression power of our language isn't just there yet and trying to browse how people have translated same words in other projects proved the case. There just aren't Finnish equivalent words for the certain English words. What those difficult words were then? Here's few ones popping in my mind: commit, checkout, revision, portlet. If any Finnish readers have an idea how to translate those, let me know :)

Last one has been translated since Plone 3 came out, but the translation isn't just good. What it currently means in Finnish is practically same as java applet. I'd like to get rid of that translation, but haven't figured out better - yet.

Problems with Plone version pinning

2009-11-18T22:13:00.010+02:00

Since we splitted our one huge Plone site to several smaller ones we've been looking for the correct procedure to keep things running smoothly and to make updates as painless as possible. So far we haven't found the right way and every update has had some problems. Some of the problems are related to the fact that we have about 24 sites to maintain and adding a new content type to all of them means lots of work. Below are all the steps needed for update (and solutions we've founded so far).

1. Add your new product to buildout.cfg eggs and zcml list.

Do this 24 times and you'll find yourself thinking there has to be better way. Luckily there is; you can extend your buildout.cfg with some general config file (eg. deployement.cfg). Most tutorials describe a way where you have those two files living side by side in filesystem and you run buildout with -c deployement.cfg option, but this doesn't remove the actual problem.

We have our deployement.cfg living in different server on the web, but this brings us some new challenges. Actual extending of buildout.cfg:s parts won't work, or at least I haven't found the way for that). So if your buildout.cfg includes something like this



[instance]

...

eggs =

Plone

...

and you want to extend that eggs list from deployement.cfg you'd think following would work:



[instance]

eggs +=

collective.yourproduct

Well - for our disappointment - it won't. Instead of extending you'll need to remove eggs attribute listing from original buildout.cfg and add it to your deployement.cfg with all the eggs the original buildout.cfg had. That'll do what you want and now you can share that deployement.cfg by adding url to it in your buildout.cfg:s extends attribute.

2. Run the actual buildout command

Once again - repeat this 24 times and if you didn't swear editing all those buildout.cfg:s I mentioned in step I'm sure you'll do that by now. We didn't find any ready solution for this so we made our own called jyu.bomanager. Our tool has buildout.cfg where we can configure all the sites we have. By convention we've named our buildout directorys with the same name our sites are. What jyu.bomanager does is that it runs buildouts automatically for all our sites and restarts them. All seems to be good here, but wait - there are problems too. I'll come to that later.

3. Update/install the new addon.

Now when we have our 24 sites up and running we just need to update or install our new product. I guess you already see what's the problem so I won't repeat myself anymore. To solve this problem we developed a tool similar with jyu.bomanager called jyu.sitemanager. What jyu.sitemanager does is that - depending our mission - it can log in to all of our sites as admin and either install specified new product(s) or upgrade those which needs updating. If everything has gone ok this far we don't have any problems with this step either.. unless we have failed testing our new addon and there are bugs in it, but that'll never happen to us :)

Weakest link...

...is step 2. So far running buildout is usually the part where things fall apart - even when on testing server all went well few minutes ago. It won't help much if we've automated all the steps so that you'd need only few commands to watch the process just happen if buildout fails. There are many points where we can fail - plone.org could be down, PyPI could be down or there could be some nasty surprise where we have some unpinned version updated - maybe brought to our buildout by some addon requirements - or just forgotten to pin.

Today we had scheduled 2 hours timeframe to update all our sites and we hoped that this time everything would go smoothly. Maybe next time... it seems that even though we've mirrored http://dist.plone.org/release/3.2.2 to our own server and pinned versions we still had some loose version out there. Plone 3.2.2 pins zc.buildout to version 1.1.1, but running buildout got us new version of plone.recipe.zope2zeoserver which expected to find 'relative_paths' attribute from zc.buildout. That attribute has appeared on zc.buildout since 1.2 versions, so you see the problem. We modified versions.cfg so that we'll get zc.buildout 1.4.1 version and things went smoothly. Another problem came from LDAPUserFolder. We have python-ldap 2.3.8 installed in our production environment, but LDAPUserFolder had version requirement for anything bigger that 2.3.0 (I'm not sure about exact version number and I'm too lazy to find out) so our buildout happily fetched 2.3.10 which needs libraries our production server doesn't have. Crash. Another version pinning and we were good to go.

Although buildout is blessing sometimes it feels like wild wild west. It isn't funny when we're supposed to update our site and our buildout hangs when it's supposed to fetch data from pypi or plone.org. Now we're thinking setting up eggproxy and/or own pypi mirrors so maybe next time we have smooth experience.

Kanban vs. Scrum

2009-09-24T20:54:00.002+03:00

Nice summary about Kanban and Scrum.

http://www.crisp.se/henrik.kniberg/Kanban-vs-Scrum.pdf

..and to make things less serious: http://blog.crisp.se/henrikkniberg/2009/06/26/1246053060000.html

Managing multiple Plone buildouts

2009-04-19T21:47:00.002+03:00

I recently had situation in work where I needed to update several of our Plone sites with few addon products. So far we haven't thought how we'll manage our sites update story as we've basically just recently entered to the amazing world of buildouts and several Plone-sites instead of one huge plone-site with zeo.

Gotta say I had few ideas popping in my head when I was manually running buildout scripts for each of our sites and stopping and restarting them after buildout was completed. My first though was to just simply write some script which does the job and then I was already imaging some web based service so we could manage our sites buildouts ttw. Well.. that might be a bit overkill.. at least for the starters, so I went back to roots and wrote a small python application which does the job. Basically it just takes the list of buildouts and runs buildout-script with -Nv parameters and restarts the site.

Source code is below and in the github.


#!/usr/bin/python2.5

from subprocess import check_call
from subprocess import CalledProcessError

#BUILDOUT_LIST is a list of folders where you're sites are
BUILDOUT_LIST = ['Site1',
                 'Site2']
class Buildout:
    """Takes care of managing one buildout"""

    def __init__(self, path):
        self.path = path
        self.dev_null = open('/dev/null','wb')

    def instance_cmd(self, command):
        try:
            if command == 'start':
                print "Starting %s instance..." % self.path
            elif command == 'stop':
                print "Stopping %s instance..." % self.path

            check_call(['./' + self.path + '/bin/instance', command],)

        except CalledProcessError, e:
            print e.returncode

    def run_buildout(self):
        try:
            print "Trying to run buildout for %s " % self.path
            dev_null = open('/dev/null','wb')
            check_call(['bin/buildout', '-Nv'],
                       cwd='./'+self.path,
                       stderr=dev_null, stdout=dev_null)
            print "Buildout for %s completed succesfully" % self.path

        except CalledProcessError, e:
            print "Error while running buildout for %s: %s" % (self.path,
                                                               str(e))

class BOManager:
    """ Launches buildout scripts """

    def __init__(self, buildouts):
        self.buildouts = buildouts

    def giveitago(self):
        for buildout in self.buildouts:
            print "-"*40
            bo = Buildout(buildout)
            bo.run_buildout()
            bo.instance_cmd('stop')
            bo.instance_cmd('start')
            print "-"*40+"\n"

if __name__ == '__main__':
    buildout = BOManager(BUILDOUT_LIST)
    buildout.giveitago()

Ubuntu, Plone and problems with openldap and python-ldap

2009-04-06T15:20:00.002+03:00

For some reason I've always had problems with python-ldap and openldap with Plone when I'm using Ubuntu. I've always managed to solve the problem by somehow building openldap from sources and been happy ever since. Now with fresh Ubuntu install I bumbed once again to the same problem and decided to write a reminder how I get things working again.

This time I used Panyasan's buildout example with one extra environment line. Without that line openldap won't compile on ubuntu. Instead of compiling succesfully it throws me an error "ubuntu getpeereid.c:52: error: storage size of ‘peercred’ isn’t known"

Here are the important parts:


[buildout]

parts =
    zope2
    productdistros
    instance
    zopepy
    openldap
    python-ldap

...

[versions]
python-ldap = 2.3.6

[openldap]
recipe = zc.recipe.cmmi
url = ftp://gd.tuwien.ac.at/infosys/network/OpenLDAP/openldap-stable/openldap-stable-20071118.tgz
extra_options = --with-sasl --with-tls --enable-slapd=no
environment = CPPFLAGS=-D_GNU_SOURCE

[python-ldap]
recipe = zc.recipe.egg:custom
egg = python-ldap
include-dirs = ${openldap:location}/include
library-dirs = ${openldap:location}/lib

Plone migration experiences

2009-03-10T11:14:00.002+02:00

1. Migrating is fun fun fun!

Let's face it - migrating Plone site from old version to new one can be either painful or very painful experience. If you need to do it for one site only, you don't even get the reward of the fact that second time will be so much easier. Third time you'll start wondering what was the hard part in it.

This post is intended to be description how we did plone migration from older version to new one with big, live and heavy traffic site. We're still middle of migrating rest of our sites and therefore I'm going to upgrade this description when I'm having more time. Also note that this is written with one go from memory and as doing the migration is quite complex thing to do including many tasks there might be some errors.

I hope this gives some ideas to others struggling with the same topic and also raises some discussion why this is so complex to do when it shouldn't. I'm also happy to get comments how we could have done this better.

2. Backgrounds

In our case we had quite big Plone 2.5.3 site with over 100 000 objects and about 10 GB Data.fs file (zodb database). We have about 500 active users who create content and our front page gets over million hit/month. Amount of data/objects/activity started to have it's effect on our site and we decided it's time to split the site to smaller individual plone sites - and while we're at it - also update hardware behind all this. At those times Plone 3 had matured enough, se we decided to upgrade the Plone version as well. This is where all the fun starts.

3. Old products

Our site has several custom products developed for Plone 2.1 and Plone 2.5 series - we even had some old products from pre-archetypes times (mxmDynamicPage), so we knew we had some migration to do.

3.1 Evaluation of available tools

There are quite a few tools available for migrating Plone content, but at least by those times none of them seemed mature enough for our needs. Some of them didn't work at all with Plone 2.5 versions and other didn't work with Plone 3. Luckily we found json_migrator product which wasn't silver bullet, but at least it was easily modified to suit our needs. Our special thanks goes to json_migrators developer Max M who kindly helped us to get started with it.

3.2 Writing new versions of our own custom products so that they work in Plone 3

This was good chance to get to know Plone 3 world and start doing things Plone 3 way. Besides our old products, we also wrote policy product which has proven to be a life saver when we were actually doing migrations. Author is now a fun of policy products!

3.3 Writing migration scripts

Some parts of the migrations we did with json_migrator and some by using Product.contentmigrator product. Reason for this divide was that we had products with no actual replacements in Plone 3 (well other than Plone 3 itself) like Simple Portlet which functionality is in Plone 3 already. This is why we needed to take data from our old site and create a script which could import the data as appropriate to environment. Eg. in Simple Portlets case we did a script which took the data from saved json-files and added static text portlets to certain folders right or left portlet.

4. Data.fs or zexp export?

Traditionally migration is done by copying old sites Data.fs file under new site and starting a site, running the portal_migration and hoping there won't be any problems and finally be happy migrator.

4.1 ..or both?

Our case was different as we were splitting our site - not migrating the whole thing. So what we had was the following:

Live site which was no no area to tweak
Identical "preview" site which had latest backup from our live-site
Helper site with identical versions from everything in above two
Brand new Plone 3.1.7

4.2 Preparing the export

While doing the migration we didn't touch our live site at all - it was all the time in use and users knew what was going on and were ready for updating same info to two sites at the same time. On our preview site we created new plone site and named it with the final name it was going to be in our new site. Then we cutted from the preview-site all the content we were going to split and pasted it in this - let's call it preview2 - site.

This took awhile... even with pretty heavy class server.

4.3 Cleaning out the mess and doing the export

After cutting and pasting was ready we ran json_export script which took care of all the data we couldn't migrate other ways. When this was ready we ran scripts which deleted all that content which was now safe in json-files on the server. This was also good point for removing products which were unneeded in Plone 3. In our experience eg. Plone Language Tool caused some random problems for us while we were doing first migrations so this is where we now get rid of it. We also removed all those products whose data we exported as json-files - there's no reason to take dead weight to new Plone :) After cleanup we're ready to export our preview2 site as zexp to server. Luckily this didn't take that long.

5. w00t.. what to do with this zexp

We moved zexp-file inside our helper sites var -folder and started the site. Now all we needed to do was log in to ZMI and go to export/import form. From there under import section we had our zexp-file. We selected it from the dropdown box and made sure we had the "Retain existing ownership information" checked and clicked import-button.

This can also take some time to complete.

After import is ready you can shutdown this helper site and copy its Data.fs file under you new Plone 3 site.

6. Running the final steps of migration

Now we're finally in the point where things either work or they don't. If luck isn't on your side today you can end up in the situation where all the work and waiting was for nothing. We were optimistic and started the new Plone 3.1.7 site and hopeed for the best. While site is starting it is building the indexes so this takes awhile, but nothing compared to the paste or import times mentioned in previous chapters.

After awhile we had zope running but when we tried to see Plone we're greeted with horrible errors. We logged in as admin to ZMI and opened our plone site in the ZMI and clicked portal_migration object. Now we had portal_migrations form open in our web browser where there are some version information in the upper part of the window. There was old plone sites version and current plone sites version. Now we simply just clicked the migrate-button which started the actual migration. Luckily everything went fine and we had migrated our old Plone site to new version.

7. Final cleanup

7.1 Old products

Migration leaves all sort of stuff from old plone still around. There are probably some products in the ZMI Control panels product management page which are remainings from the old Plone. Easy solution to clean them is to select everything from that page, click delete, stop and restart the site.

7.2 Workflows

For some reason migrations doesn't touch workflows so if you have used default plone workflow/folder workflow in your old plone you'll likely want to create new temporary site under Plone 3 and copy its plone and folder workflows under your actual sites portal_workflow (after you've deleted the old ones). Reason for this is that in the Plone 3.1 you don't give owner, or manager roles anymore via sharing tab - instead you give them roles like can edit, can add etc. Old plones workflows doesn't give a damn about those new roles and likewise doesn't have owner or manager roles in sharing tab. So unless you don't do this your newly assigned roles won't work. After replacing those workflows you'll be still blind of what roles users have in your site as sharing tab doesn't include old roles. Our solution to this was custom script which walked through all folderish objects and converted owner/manager roles to can add/can edit roles and Member roles to can read. Problem solved.

7.3 Policy product

Now it's time for our shiny new policy product which installs our products to our site and takes care of turning on/off every little switches. After installing this you should have almost ready site, but there are still few things to do. Now when we have all the products installed we still have one migration to run - contentmigration.

If you still remember we decided to migrate some of the contents by using contentmigrator product. We still have some content on our site using old product and we need to get rid of it so we create external method to portal_skins/custom folder and fill the needed information to point our external method of our new replacing product. We save the external method and hit the test tab. When done we can remove remainings of our old product by deleting the product from products folder and restarting site.

8. Enjoy

What? It's already done? Whew.. it wasn't so bad was it?

Plone and multi-site management

2009-03-07T13:21:00.000+02:00

CMS Wire recently posted a light weight comparsion of open source CMS and their multi-site management capabilities. According to article none of the three popular CMS (Plone, Drupal and Joomla) include multi-site management in their core CMS. Article was more about third party addons and what they have to offer for multi-site management needs.

Six Feet Up companys Lineage -product shines on this comparsion with it's easy to use separation of certain folder inside Plone site as independet site with certain users, workflows, theme and navigation. Although Lineage is already released it still has some dependencies which are not met with Plone 3.2.1 (but should be in Plone 3.3), so installation needs some additional lines to buildout. You can find more information from Plone Software Center.

In the Plone-sites we're developing and maintaining, we've traditionally wanted to keep sites theme pretty much the same within all different departments and faculties. With Lineage change to this restriction policy would be pretty easy I still believe our current way is best option for us. We have students who are studying in several minors in different departments. Now if every department would have their own theme this would just broke the ways people have learned to use our pages. Still we've encountered situations when there are some projects which aren't tied to our university so tightly and would want custom theming. I can see how easily this could be done with Lineage.

Migrating Plone products to Plone 3

2008-10-03T09:47:00.001+03:00

Calvin Hendryx Parker Tips For Migrating Apps To Plone 3

View SlideShare presentation or Upload your own. (tags: plone ploneconf2007)

Martin Aspeli Extending And Customising Plone 3

View SlideShare presentation or Upload your own. (tags: plone ploneconf2007)

About this blog and the writer

2008-03-26T14:21:00.000+02:00

I should knock down the barriers and finally put some content to this blog. I intend to make this as my technology biased blog partly as a reminder for myself about things and problems I've encountered and hopefully also solved.

About me... I'm Finnish information technology science student currently writing my masters thesis about web site usability. To be exact I'm doing research about usability of Ajax based web applications. My intention is to get some data if much hyped myth that Ajax applications have better usability have actually some ground in it. My research application is Plone web CMS (Content Management System) which is probably best open source web CMS currently available.

Reasons for me picking this certain web application is my backgrounds as a Plone developer in University of Jyväskylä. I've been working with Plone since Autumn 2006 and in that time I've managed to do quite a few things with Plone from translating ready products to developing new ones. Products in Plone are basicly different containers for different types of publications (eg. News, Events etc.) which have specific content, workflows and views. Since Plone 3.0 Plone got some more Ajax-stuff behind it which makes it perfect research application for me. University of Jyväskylä is using currently Plone 2.5 which isn't using heavily any Ajax-stuff. This makes them a perfect usability comparsion target for my research.

Well that's about my masters thesis. I'm going to use this blog for posting stuff about stuff I encounter in my work and hobbies. It may be about Plone, but also about other web technologies or programming as well. Or just about latest TV-show I've seen.

Now I'm going to go back to my masters thesis. Good byebyes!