Zopatista

Portlets as ESI include

2012-06-14T00:00:00-07:00

Using ESI includes to cache Plone portlets separately.

To help with making a large and busy intranet website perform better, we’ve used a light sprinkling of ESI (via Varnish’s ESI support) to improve the cacheabilibty of pages in the site. By delegating assembly of parts of the page to the Varnish cache, pages become much more cacheable as frequently changing chunks such as the personal bar at the top are requested separately.

Portlets via ESI include

One such chunk we separated out is the right-hand portlets column. Varnish has been configured to set a special header so that we can detect that ESI is supported:

sub vcl_recv {
    ...
    # Indicate that a varnish capable of doing ESI is in front...
    set req.http.X-ESI = "esi";
    ...
}

Using this header we can then conditionally swap out the portlets column with an <esi:include> statement; this makes site development much easier as we do not have to run Varnish just to see the site working. Here is the relevant section from the main_template.pt file:

<td id="portal-column-two"
    metal:define-slot="column_two_slot"
    tal:condition="sr">
  <div class="visualPadding"
       tal:define="
           esi_header request/HTTP_X_ESI | nothing;
           base context/@@plone_context_state/current_base_url | nothing;
           location python:base and base.rstrip('/').split('/')[-1].lstrip('@');
           esi python:esi_header and (location not in (
               'manage-portlets', 'manage-content-type-portlets'));
           queryString request/QUERY_STRING;
           queryString python: queryString and '?' + queryString or '';
           ">
    <metal:portlets define-slot="portlets_two_slot">
      <esi:include tal:condition="esi"
          tal:attributes="src string:${context/absolute_url}/@@right-column${queryString}" />
      <tal:noesi condition="not: esi"
                 replace="structure provider:plone.rightcolumn" />
    </metal:portlets>
    &nbsp;
  </div>
</td>

Note that we are making sure that ESI is also not applied when using the portlet management views.

The @@right-column view is simply a template:

<html tal:omit-tag="">
<body tal:omit-tag="">

<tal:block replace="structure provider:plone.rightcolumn" />

</body>
</html>

This whole setup was working swimmingly; we could cache pages for extended periods of times with things like the portlets updating much more frequently and with caching keyed to specific groups of users.

Where did that portlet go?

This being a large and complex intranet, it took some time for someone to notice that some lightly-used portlets were no longer showing up. These were portlets that depend on certain content being there, so their absence was not necessarily a problem. However, it was becoming clear that even when their specific conditions were being met, they were not being rendered still. This was quickly narrowed down to the ESI-included portlet rendering; if you bypassed the cache the portlets would show up.

So what went wrong?

Portlets are essentially rendered as part of the Zope viewlet framework. Viewlets are snippets of page output that are looked up by a key consisting of the current context, the current request, the current view and the viewlet manager. Portlets thus have access to these same pieces of information, and you can thus register portlets that only show for certain contexts (particular content types, marker interfaces, etc.), browser layers (usually themes), and even only for specific views or portlet managers (tying the portlet to the left, right or dashboard portlet wells).

With the lesser-known <plone:portletRenderer /> directive, you can also vary the way portlets are rendered for the above keys. Thus, a portlet can look different in different themes, different portlet managers, or when a certain extra marker interface is present on your content objects. This is what had happened to the vanished portlets here; they had been tied to specific views:

<configure
    xmlns="http://namespaces.zope.org/zope"
    xmlns:plone="http://namespaces.plone.org/plone"
    />

  <plone:portlet
    name="foobar.portlets.localcalendar"
    interface=".localportlet.ILocalCalendarPortlet"
    assignment=".localportlet.Assignment"
    renderer=".localportlet.Hidden"
    addview=".localportlet.AddForm"
    />

  <!-- My HQ page -->
  <plone:portletRenderer
    portlet=".localportlet.ILocalCalendarPortlet"
    class=".localportlet.Renderer"
    view="foobar.types.browser.mychain.MyChainView"
    />

  <!-- My Store page -->
  <plone:portletRenderer
    portlet=".localportlet.ILocalCalendarPortlet"
    class=".localportlet.Renderer"
    view="foobar.types.browser.store.StoreView"
    />
</configure>

The above plone:portlet declaration registers a portlet that is hidden by default. The two plone:portletRenderer declarations then assign new renderers when certain views are being used instead. This neat trick allows for the portlet to be targeted very specifically.

This all works great, unless you use a dedicated view for ESI rendering of the portlets. Suddenly the current view is no longer MyChainView or StoreView, but rather @@right-column. Thus the dedicated renderer is skipped in favour of the .localportlet.Hidden renderer, which does what it says on the tin: not render.

Reconstruct the whole context

The solution is of course to reconstruct the whole context; the @@right-column view already had most things right, only the current view is wrong. With a simple set of TAL declarations we can set up a new value for the view variable when rendering the portlets. Here is the reworked main_template.pt code:

<td id="portal-column-two"
    metal:define-slot="column_two_slot"
    tal:condition="sr">
  <div class="visualPadding"
       tal:define="
           esi_header request/HTTP_X_ESI | nothing;
           base context/@@plone_context_state/current_base_url | nothing;
           location python:base and base.rstrip('/').split('/')[-1].lstrip('@');
           esi python:esi_header and (location not in ('manage-portlets', 'manage-content-type-portlets'));
           viewContext string:?__view_context=${view/__name__};
           queryString request/QUERY_STRING;
           queryString python: queryString and viewContext + '&amp;' + queryString or viewContext;
                   ">
    <metal:portlets define-slot="portlets_two_slot">
      <esi:include tal:condition="esi"
                   tal:attributes="src string:${context/absolute_url}/@@right-column${queryString}" />
      <tal:noesi condition="not: esi"
                 replace="structure provider:plone.rightcolumn" />
    </metal:portlets>
    &nbsp;
  </div>
</td>

We use a GET parameter to pass along the name of the view to look up; I’ve used a double-underscore prefix here to reduce the chances we clash with a query string parameter used elsewhere in the site. The @@right-column view then restores this view for portlet rendering (with a fallback to the Plone default view context @@plone):

<html tal:omit-tag="">
<body tal:omit-tag="">

<tal:block
    define="viewname request/__view_context | nothing;
            viewname python:viewname and '@@' + viewname or '@@plone';
            view nocall:context/?viewname"
	replace="structure provider:plone.rightcolumn" />

</body>
</html>

Et voilà, our portlets are showing up good and proper again.

Portlets As Esi Include

2012-06-09T00:00:00-07:00

Unicode in RTF documents

2012-06-06T00:00:00-07:00

How to encode unicode codepoints in RTF documents using PyRTF.

Some time ago I had to output some nicely formatted reports from a web application, to be usable offline by Windows users. Naturally, I used the aging but still reliable PyRTF module to generate RTF documents with headers, tables, and a consistent style.

As my application users are mostly Norwegians, however, I quickly discovered that the PyRTF module does not handle international characters (i.e. anything outside the ASCII codepoints), at all. There is no unicode support at all (it has been on the TODO list since forever), let alone converting unicode codepoints to whatever RTF uses to represent international characters.

Recently, a Stack Overflow question reminded me of how I solved this problem at the time, and clearly this question has come up before. Because some approaches I’ve seen can actually produce incorrect or overly verbose output (including pyrtf-ng), I wanted to explain and expand on my solution to provide a definitive answer to the problem, and also see how my original method faired in terms of speed.

So does RTF handle unicode?

Since PyRTF doesn’t filter the text you add to a document at all we can just encode unicode strings ourselves. Lucky for me, the Wikipidia entry on RTF has a fairly detailed section on how RTF handles characters outside the ASCII range. Together with the published RTF 1.9.1 specification (PDF) there is plenty of information on how to encode unicode codepoints to RTF control sequences.

There basically are two choices:

The \'hh control sequence; a backslash and single quote, followed by an 8-bit hexadecimal value. The value is interpreted as a code-point in a Windows codepage, limiting it’s use. You can assign different codepages to different fonts, but you still cannot use the full range of unicode in a paragraph.
The \uN? control sequence; backslash ‘u’ followed by a signed 16-bit integer value in decimal and a placeholder character (represented here by a question mark). The signed 16-bit integer number here is consistent with the RTF standard for control characters, a value between -32768 and 32767.

This control sequence can properly represent unicode, at least for the U+0000 through to U+FFFF codepoints. This sequence was introduced in the 1.5 revision of the RTF spec, in 1997, so it should be widely supported. The placeholder character is meant to be used by readers that do not yet support this escape sequence and should be an ASCII character closest to the unicode codepoint.

The \uN? format is the easiest to produce, especially if you ignore the replacement character (just set it to ‘?’ at all times, surely most RTF readers support the 1.5 RTF standard by now, it’s been out there for 15 years).

Encoding the slow (and incorrect) way

A quick search with Google showed me how pyrtf-ng encodes unicode points:

    def writeUnicodeElement(self, element):
        text = ''.join(['\u%s?' % str(ord(e)) for e in element])
        self._write(text or '')

Unfortunately, the above snippet does a few things wrong: it uses a control code for every character in the unicode string, producing output that is at least 5 times as long as the input, and it doesn’t produce negative numbers for codepoints over \u7fff:

>>> example = u'CJK Ideograph: \u8123'
>>> ''.join(['\u%s?' % str(ord(e)) for e in example])
u'\\u67?\\u74?\\u75?\\u32?\\u73?\\u100?\\u101?\\u111?\\u103?\\u114?\\u97?\\u112?\\u104?\\u58?\\u32?\\u33059?'

A recent Stack Overflow answer improved on this by only encoding characters over \u007f (decimal 127) but it still iterates over every character in the string to do so:

>>> ''.join(['\u%s?' % str(ord(e)) if ord(e) > 127 else e for e in example])
u'CJK Ideograph: \\u33059?'

This outputs unicode because codepoints < 128 are left untouched; numbers are not properly converted to signed shorts either. Here is my variation that remedies these things, and dispenses with the str() call:

>>> ''.join(['\u%i?' % (ord(e) if e < u'\u8000' else ord(e) - 65536) if (e > '\x7f' or e < '\x20' or e in u'\\{}') else str(e) for e in example])
'CJK Ideograph: \\u-32477?'

This feels like rather a waste to me, and must be slow as well. I wanted to see how my own solution stacks up against the character-by-character, naive implementation.

Encoding the lazy way

While casting around for my own solution, I also looked into the Python codecs module to come up with ideas on how to do this more efficiently. Of course, the codecs provided by that module are all implemented in C, but the unicode_escape codec did produce output quite close to what I needed for RTF; codepoints between \u0020 and \u007f are left alone, the rest are encoded to one of the \xhh, \uhhhh or \Uhhhhhhhh 8, 16 or 32-bit escapes (with the exception of \t, \n and \r). Would there be any way to reuse this output?

Well, if you combine this with a bit of re.sub magic, you can in fact produce convincing RTF command sequences:

>>> import re
>>> import struct
>>> _charescape = re.compile(r'(?<!\\)\\(?:x([0-9a-fA-F]{2})|u([0-9a-fA-F]{4}))')
>>> def _replace_struct(match):
...     match = match.groups()
...     # Convert XX or XXXX hex string into 2 bytes
...     codepoint = (match[0] and '00' + match[0] or match[1]).decode('hex')
...     # Convert 2 bytes into a signed integer, insert into escape sequence
...     return '\\u%i?' % struct.unpack('!h', codepoint)
... 
>>> escaped = example.encode('unicode_escape')
>>> escaped
'CJK Ideograph: \\u8123'
>>> _charescape.sub(_replace_struct, escaped)
'CJK Ideograph: \\u-32477?'

Using the struct module gave me a quick means to re-interpret the hexadecimal notation as produced by the unicode_escape format as a signed short, but I did have to make sure there were 2 bytes at all times.

Of course, the above trick does not handle newlines, returns or tabs (\n, \r and \t respectively) correctly, nor does it escape existing backslashes yet, but I hoped back when that this proof of concept should operate several orders of a magnitude faster than the naive character-by-character method when dealing with mostly-ASCII input; most of the work is done in C by the codecs and re modules, after all.

So this time around I decided to time these:

>>> import timeit
>>> def test1(): ''.join(['\u%i?' % (ord(e) if e < u'\u8000' else ord(e) - 65536) if (e > '\x7f' or e < '\x20' or e in u'\\{}') else str(e) for e in testdocument])
... 
>>> def test2(): _charescape.sub(_replace_struct, testdocument.encode('unicode_escape'))
... 
>>> declaration = u'\
... Alle mennesker er f\xf8dt frie og med samme menneskeverd og menneskerettigheter. \
... De er utstyrt med fornuft og samvittighet og b\xf8r handle mot hverandre i brorskapets \xe5nd.'
>>> testdocument = declaration * 100
>>> timeit.timeit('test1()', 'from __main__ import test1', number=500)
5.982733964920044
>>> timeit.timeit('test2()', 'from __main__ import test2', number=500)
1.4459600448608398

Cool, so my hybrid encode plus regular-expression based solution looks to be around 4 times as fast, at least when it comes to simple Norwegian text with a handful of latin-1 characters, my most common case. Note however that I am not handling the RTF escape characters properly, nor are the \n, \r and \t characters handled correctly.

Can I do better?

But I am actually being too clever by half (read: pretty dumb really); why did I encode to unicode_escape in the first place? I was still in the process of fully understanding the issues and saw a shortcut. My regular expression isn’t particularly clever, I dabbled with the struct module to get my signed short values, and with all this hocus-pocus I lost sight of the goal: to escape certain classes of characters to RTF command codes.

But aren’t regular expressions quite good at finding those classes all by themselves? I may as well use a decent expression that selects what needs to be encoded directly:

>>> _charescape_direct = re.compile(u'([\x00-\x1f\\\\{}\x80-\uffff])')
>>> def _replace_direct(match):
...     codepoint = ord(match.group(1))
...     return '\\u%s?' % (codepoint if codepoint < 32768 else codepoint - 65536)
>>> _charescape_direct.sub(_replace_direct, example).encode('ascii')
'CJK Ideograph: \\u-32477?'
>>> def test3(): _charescape_direct.sub(_replace_direct, testdocument).encode('ascii')
...
>>> timeit.timeit('test3()', 'from __main__ import test3', number=500)
0.5356400012969971

Suddenly we have an 10 times speed increase! Not only that, I am now also properly escaping the three whitespace characers \n, \r and \t, and as an added bonus, the RTF special characters \, { and } are now also being escaped! I call this a result, and a lesson to learn.

Perhaps we can translate instead

We could also use a translation table to do my escaping for me. This is simply a dict that maps unicode codepoints to a replacement value. To create a static dict for all unicode values could be somewhat tricky, requiring either a custom __missing__ method or loading a generated structure on import.

Before digging into clever solutions to that, I should perhaps first test the speed of a simple translation table, one that only covers codepoints up to ‘\u00ff’, or latin-1:

>>> _table = {i: u"\\'{0:02x}".format(i) for i in xrange(0, 32)}
>>> _table.update({ord(c): u"\\'{0:02x}".format(ord(c)) for c in '\\{}'})
>>> _table.update({i: u"\\u{0}".format(i) for i in xrange(128, 256)})
>>> len(_table)
163
>>> timeit.timeit('testdocument.translate(_table).encode("ascii")', 'from __main__ import testdocument, _table', number=500)
2.66812801361084

Unfortunately, using .translate turns out to be slowing us down considerably. Reducing the table to just a few codepoints doesn’t help either:

>>> _basictable = {ord(c): u"\\'{0:02x}".format(ord(c)) for c in '\n\r\t\\{}'}
>>> len(_basictable)
6
>>> timeit.timeit('testdocument.translate(_basictable)', 'from __main__ import testdocument, _basictable', number=500)
2.0113179683685303

So it looks like I might want to avoid using .translate if at all possible.

Worst-case scenario

So far, I’ve compared methods by testing them against some Norwegian text, typical of many European languages with a generous helping of ASCII characters.

To get a more complete picture, I need to test these methods against a worst-case scenario, a UTF-8 encoded test set from a great set of UTF-8 test documents:

>>> import urllib
>>> utf8_sequence = urllib.urlopen('https://raw.github.com/bits/UTF-8-Unicode-Test-Documents/master/UTF-8_sequence_unseparated/utf8_sequence_0-0xffff_assigned_printable_unseparated.txt').read().decode('utf-8')
>>> len(utf8_sequence)
58081
>>> testdocument = utf8_sequence
>>> timeit.timeit('test1()', 'from __main__ import test1', number=10)
0.7785000801086426
>>> timeit.timeit('test3()', 'from __main__ import test3', number=10)
0.8913929462432861

Interesting! So in the worse-case scenario, where the vast majority (99.8%) of the text requires encoding, the character-by-character method is actually a little faster again! But this also means that for most cases, where you insert shorter text snippets into an RTF document, and where a far larger percentage of characters do not need escaping, the regular expression method will beat the character-by-character method hands down.

So what about non-BMP Unicode?

So far I’ve focused only on characters within the BMP. You can apparently use a UTF-16 surrogate pair, at least according to Wikipedia for codepoints byond the BMP. However, the RTF specification itself is silent on this, and no endian-nes is documented anywhere that I can find. The Microsoft platform uses UTF-16-LE throughout, so perhaps RTF readers support little-endian surrogate pairs too.

However, I cannot at this time be bothered to extend my encoder to support such codepoints. On a UCS-2-compiled python there is a happy coincidence that codepoints beyond the BMP are treated mostly like UTF-16 surrogate pairs anyway, so they are sort-of supported by this method:

>>> beyond = u'\U00010196'
>>> _charescape_direct.sub(_replace_direct, beyond).encode('ascii')
'\\u-10240?\\u-8810?'

Note, however, that the first byte is -10240, or 0xd800 in unsigned hexadecimal, making this a big-endian encoded surrogate pair. Presumably on Windows that’ll encode the other way around.

On a UCS-4 platform the codepoint will be ignored by the regular expression and the .encode('ascii') call will raise a UnicodeEncodeError instead.

I am calling this ‘unsupported’ and a day. Suggestions for implementing this in a neat and performant way are welcome!

Off to PyPI we go

I am quite happy with the simple regular expression method, and prefer it over the character-by-character loop.

So I packaged up my regular expression method as a handy module on PyPI, complete with Python 2 and 3 support and a miniscule test suite; the source code is available on GitHub.

The module in fact registers a new codec, called rtfunicode, so after you import the package all you need do is use the new codec in the .encode() method:

>>> import rtfunicode
>>> declaration.encode('rtfunicode')
'Alle mennesker er f\\u248?dt frie og med samme menneskeverd og menneskerettigheter. De er utstyrt med fornuft og samvittighet og b\\u248?r handle mot hverandre i brorskapets \\u229?nd.'

Hopefully it comes in handy for others. Feedback is most welcome, as are patches!

The dreaded plone.relations IntId KeyError

2011-06-29T00:00:00-07:00

This article was originally published on jarn.com.

When IntIds go missing, the going gets tough. Specifically, plone.app.relations and related packages do not deal gracefully when a relationship source or target is missing. Here is how we clear such broken relationships.

We’ve been experimenting with plone.app.relations to manage relationships between objects for a few years now. This package uses zc.relations to lay the links between content items in your site, which in turn relies on zope.app.intid to indirectly create those links. Basically, intids are pointers to the real objects and lets you handle the linking efficiently.

Water in the Bilge

The relations machinery is not very forgiving if any intid has gone AWOL. Normally, the relations data structures are kept in sync through Zope events, but this doesn’t always work out. In our experience, you can end up with objects and their intids removed, but the relationships pointing to the now-gone intids still in place. When this happens, things break, and you get trackbacks ending in the dreaded KeyError: <long number> in getObject of zope/app/intid/__init__.py. The traceback line before that will be zc/relationship/index.py in the method resolveToken.

Now, the zc.relations package is very powerful and very, very flexible. This comes at a price, as it’s internal data structures are quite daunting to the uninitiated. If you have to repair these relations and all you have is the missing intid at one end of the relation, it’ll be a long hard slug through a maze of 3 or 4 different packages and opaque TreeSets.

Bucket by Bucket

Luckily, we already did the deep code dive for you. The following method, if passed an intid, will find any references to it in the relations data structure and remove these for you:

from plone.relations.interfaces import IComplexRelationshipContainer
from zope.app.intid.interfaces import IIntIds

def removeKeyErrorRelationship(iid):
    """Remove all relationships that point to a intid no 
       longer in the site
    """
    intids = getUtility(IIntIds)
    relationships = getUtility(IComplexRelationshipContainer, 
                               name='relations')
    relIndex = relationships.relationIndex
    for direction in ('target', 'source'):
        data = relIndex._name_TO_mapping[direction].get(iid)
        if not data or data[0].value == 0:
            continue # Empty set for this direction
        for relid in list(data[1]):
            keyref = intids.refs.get(relid)
            if keyref is None:
                # Not even the relationship exists anymore
                relIndex._remove(relid, (iid,), direction)
            else:
                relation = keyref.object
                try:
                    relation.__parent__.remove(relation)
                except AttributeError:
                    # The relation object only exists in the intid utility;
                    # in this case __parent__ is None.
                    relIndex.unindex(relation)
                    relIndex.unindex_doc(relid) # be doubly sure
                    intids.unregister(keyref)

Note that this method assumes you already have the local site manager set up properly. This is a great little method to get rid of individual KeyError problems.

Man the Pumps

It would be better, if you could clear out all missing intids from the relations tool altogether, before they become a problem and things fall down. Luckily, there is! The following code will hunt down and remove all missing intids from the tool. Note that it’ll take a while (it’ll scan through two whole relations indexes), so you better sit back and relax while the work is done.

from plone.relations.interfaces import IComplexRelationshipContainer
from zope.app.intid.interfaces import IIntIds
from BTrees.IOBTree import difference

def clearAllMissingLinks():
    """Find and remove all missing intids in the
       relations tool.
    """
    intids = getUtility(IIntIds)
    relationships = getUtility(IComplexRelationshipContainer, 
                               name='relations')
    relIndex = relationships.relationIndex
    rtotal = itotal = 0
    for direction in ('target', 'source'):
        idx = relIndex._name_TO_mapping[direction]
        for iid in difference(idx, intids.refs):
            itotal += 1
            for relid in list(idx[iid][1]):
                keyref = intids.refs.get(relid)
                if keyref is None:
                    # Not even the relationship exists anymore
                    relIndex._remove(relid, (iid,), direction)
                else:
                    relation = keyref.object
                    try:
                        relation.__parent__.remove(relation)
                    except AttributeError:
                        # The relation object only exists in the intid utility;
                        # in this case __parent__ is None.
                        relIndex.unindex(relation)
                        relIndex.unindex_doc(relid) # be doubly sure
                        intids.unregister(keyref)
                rtotal += 1
    return itotal, rtotal

Note that this method returns the total number of intids identified, as well as the total number of relationships removed.

Patch the leak?

Instead of pumping out the water, we should of course patch the leak. We have yet to find it though, but if we do, we’ll make sure the affected packages receive the patch!

April 2012 Update: clean-up methods fine-tuned.

I’ve found that in practice some relationships only were still referenced by intid keyrefs and present in the relationships index, but no longer were present in the relationship utility itself. These have to be manually unindexed and removed; the code examples above have been updated to reflect this.

Saving the day: recovering lost objects

2008-12-18T00:00:00-08:00

This article was originally published on jarn.com.

When a customer discovers over a week later that an important object was accidentially deleted, what do you do?

Oh noes!

A customer discovered that an important entire section of his site was missing and asked us to bring it back. This was in a heavily edited site, with loads of writes each day, but we quickly located the offending transaction: someone had deleted the object in question 9 days earlier.

Undo was no longer an option, though: too many things had changed, not least the catalog. Truncating the Data.fs (removing all transactions since, including the offending one) was not only undesirable, but impossible as the site stores the data in Oracle through RelStorage.

Time travel

So, instead of permanently removing transactions, we used a handy little package to do some time traveling: zc.beforestorage.

zc.beforestorage does require a ZODB version 3.8 or 3.9; the customer installation is on Plone 3.0, so a newer ZODB3 egg was necessary for this operation. A small additional buildout configuration file (saved as beforestorage.cfg) helps out:

[buildout]
extends =
    buildout.cfg
eggs +=
    zc.beforestorage
    ZODB3
    zope.proxy

[versions]
ZODB3 = 3.8.1
zope.proxy = 3.4.2

[relstorage-patch]
recipe = plone.recipe.command
command = 
    cd ${buildout:eggs-directory}/ZODB3-3.8.1-py2.4-linux-i686.egg/ZODB
    curl -s http://svn.zope.de/zope.org/relstorage/tags/1.1c1/poll-invalidation-1-zodb-3-8-0.patch | patch -N -p0
    cd ${buildout:directory}
update-command = ${relstorage-patch:command}

[instance]
zope-conf-additional +=
    enable-product-installation False

The relstorage-patch section in the above code ensures that our ZODB3 egg is patched with the RelStorage additions, and the zope.proxy egg is needed because ZODB 3.8 requires a newer version. The enable-product-installation line is required because zc.beforestorage puts your ZODB in read-only mode (understandibly); the option tells Zope not to try and write product information to the ZODB.

Once buildout has been run with this configuration (with the -c switch), you’ll still need to edit the zope.conf file for your instance, usually in parts/instance/etc/zope.conf. You need to edit the <zodb_db main> section to wrap the storage in the beforestorage. Ours looked something like this:

<zodb_db main>
    # Main database
    cache-size 650000
%import zc.beforestorage
%import relstorage
    <before>
    before 2008-12-08T10:29:03
    <relstorage>
        <oracle>
            dsn RELSTORAGE_DSN
            password xxxxxxxxx
            user xxxxxxxx
        </oracle>
    </relstorage>
    </before>
    mount-point /
</zodb_db>

Any line with the word ‘before’ in it is new. The timestamp we learned from the undo log, simply converted to UTC. Now, when you start the instance, you are in the past. You can’t alter this past (no killing of grandfathers), but you can read it. And lo and behold, the deleted object is back.

Recovery

Now that we have found the lost object, we can recover it. We simply exported it; in the ZMI, choose the Export/Import button, and save the export on the server. Remove the zc.beforestorage configuration (just run buildout with your regular buildout file), restart, import the .zexp file, done!

Note that you’ll need to reindex the imported content and that any related data that lives outside of the object itself is gone. For example, its intid are gone and all relationships to it will have to be recreated etc. But you just saved your customers bacon, I’m sure they won’t mind a little manual work!

One cookie please, but hold the pickles

2007-11-09T00:00:00-08:00

This article was originally published on jarn.com.

The python pickle module is dangerous, didn’t you know?

All your base are belong to us

By now you all should have installed last Tuesday’s Hotfix. If you haven’t yet, but are running Plone 2.5 or Plone 3.0 websites, you should do so yesterday, or at least as soon as humanly possible.

The Hotfix patches a serious security problem in the statusmessages and linkintegrity modules, where network-supplied data was interpreted as pickles. “Network-supplied” data in this case means both cookies and form data, and no authentication is required to exploit the holes.

What happen ?

The basic problem with the holes is that the Plone community was totally unaware of how dangerous the pickle module really is. Hanno Schlichting did file a report a few months ago stating that the code was potentially dangerous, but even he didn’t fully appreciate that pickles are a security hole only waiting for attacker input. The scary thing here is that the code in question was written by extremely capable and experienced developers, but none of them were aware of the fact that you cannot ever use pickles to load user-supplied data.

What is needed then, is education. This is my contribution.

You are on the way to destruction

So what is wrong with pickles? They are just a damn handy way to serialize arbitrary data into binary strings and back again, right?

Yes, they are that, but the pickle format used is also a simple stack language that allows the creation of arbitrary python structures, and execute them. This stack language allows you to import modules (the ‘c’ symbol), and apply arguments to callables (the ‘R’ symbol), thus causing code to be run. Combine this with the python built-in methods eval and compile and you have the perfect vehicle for an attacker to have the pickle loader routine execute arbitrary python code when loading a well-crafted pickle. Just image what an attacker could do with that to your Zope server. Do you think you’ll ever be sure you got all the backdoors out of your Data.fs?

We get signal

So next time you need to preserve data across HTTP requests, please do not be tempted to use the pickle module to create strings for you. Rarely will you have anything more than a handful of simple datatypes to pass along anyway, so just invent a simple dataformat and use that instead. (No, using a subclass of the python implementation of pickle is not a simpler solution).

With statusmessages for example, each message consists of a message and a type string, both unicode. So we changed to a hand-rolled format using a 2 byte length header (11 bits of message length, 5 for the type) directly followed by the message and type strings (encoded to utf-8). When reading this from a cookie again later, the decoder simply has to read the lengths from the first 2 bytes, then read the right amount of characters to get the message and type back. A similar method was used to encode the linkintegrity data. Simple, effective, and impervious to attacks.

Congratulation!!
A.D.2111
All bases of CATS were destroyed.
It seems to be peaceful.
But it is incorrect. CATS is still alive.
ZIG-01 must fight against CATS again.
And down with them completely !
Good luck.

Zero Wing, 1989

Small change, big effect

2007-11-04T00:00:00-07:00

This article was originally published on jarn.com.

How changing one line halved the time it took to rename a Plone folder.

Here at the Plone Performance sprint, Matt Hamilton and Sasha Vincic are homing in on the Catalog and folder renaming. As Sasha already reported earlier, they identified the object_provides index as a potential bottleneck.

The object_provides index

The index is filled with interface identifiers, strings representing the actual interfaces. The data for the index comes from a small method in Products.CMFPlone.CatalogTool, object_provides, which looked like this:

def object_provides(object, portal, **kw):
    return [interfaceToName(portal, i) for i in providedBy(object).flattened()]

So, for each interface declared by an object, interfaceToName is invoked. The purpose of interfaceToName is to provide a way to turn an interface to a string that can be used to later turn that string back into an interface, through the queryInterface method, interfaceToName’s sibling.

Now the problem with interfaceToName is that it has to iterate over the whole utilities registry to find all interfaces registered as utilities just to find out what name it was registered with. This is a slow process, but a necessary one; although the default name for an interface is it’s dotted name (the python identifier path to their definition), some special classes of interfaces are registered with a different name. For example, when registering a Zope3-style browser menu, an interface is generated for the menu, and registered with a zope.app.menus prefix.

No need for interfaceToName

Luckily, for the object_provides index use-case, interfaceToName is overkill. First of all, object_provides indexes declared interfaces on content objects only, and therefore will never encounter any of the “special” interfaces.

But more importantly, the index contents are never used to find the original interfaces again. Quite the contrary, it is only used to search what objects provide a given interface, and the developer querying the catalog will have to generate the same string format every time they search. So, with the index using interfaceToName to fill the index, searching the index also requires developers to use interfaceToName to query the index. Search for IATFolder? Pass in interfaceToName(IATFolder) and hit the same performance problem.

Unique identifier

So if interfaceToName is overkill, what unique identifier should we use then? As we already mentioned, when you register an interface in the first place, the default name is the dotted name of the interface. It’s a unique identifier, as it’s the name under which python stores it in memory. It is available as the __identifier__ attribute on the interface. As it’s unique, and available directly from the interfaces themselves, it’s ideally suited for both indexing and searching.

Of course, this means that if we use __identifier__ then you should use the same attribute when querying the index. Because __identifier__ (or __module__ + '.' + __name__, which is the same) is already the default for interfaceToName, this is what Plone developers have been using anyway.

So we changed the indexing method to:

def object_provides(object, portal, **kw):
    return [i.__identifier__ for i in providedBy(object).flattened()]

and presto, indexing was more than twice as fast, as shown by the pretty graph on the right. We tested this by having JMeter rename a folder with 20 documents in it, 40 times.

Not bad for a one-line change.