hvelarde

Look Ma! No Resource Registries

2018-09-12T15:25:00.001-05:00

Over the past years I have spent a lot of time trying to make our add-ons compatible with Plone 4.3 and Plone 5. This has been difficult for many reasons including differences in the API of default content types frameworks and registration of static resources, just to mention some.

On the latest topic I've been very critic of the approach taken and even tried to propose a different one to avoid some well known problems on PLIP 1896. Unfortunately this was not understood at the time and I just gave up because we didn't had use cases to justify the migration to Plone 5.

This has started to change in latest months and we can't continue kicking the can down the road any longer: Python 2.7 will be EOL in 2020, Zope 4 has been released, and there has been a lot of work lately to make Plone compatible with Python 3.

IMO, Plone 5 resource registries are complex and must be avoided for many reasons as stated in the PLIP aforementioned:

the revolutionary approach chosen (rewrite everything) has proven buggy and difficult to maintain, as shown by the many issues reported over the last years (the latest opened just 2 days ago)
some of the JavaScript tools used at the time (Bower, Grunt, RequireJS…) are less attractive nowadays than more modern options like npm, Yarn and webpack
resource bundling was a workaround for a limitation of the HTTP/1.1 protocol and not a CMS feature; HTTP/2 made this unnecessary and even undesirable
LESS is now way less popular than SASS

I still think that every add-on author must be able to use the tools of their choice, and I still think Plone should not have to worry about resource bundling.

In early 2016 we selected webpack for our projects and in early 2017 we developed a Buildout recipe (called sc.recipe.staticresources) to make the inclusion of it into the Plone ecosystem easier. Over the past 2 years we have been developing, testing, and enhancing this approach in many add-ons and projects. But the resource registries were still there…

Not anymore.

I'm pleased to announce the release of collective.lazysizes 4.1.1.1, the first add-on that uses a new and very opinionated approach on how to handle static resources in Plone: we just deprecated resource registries in favor of a viewlet registered in plone.htmlhead. This simplifies maintenance among multiple Plone versions and avoids bundling of unrelated resources.

How does it work?

We use webpack to generate all static resources (in our case, a JavaScript file, a page template and an icon; check the webpack.config.js file). The page template is used for the viewlet and includes the following code on it:

<script async="" src="https://www.example.com/++resource++collective.lazysizes/lazysizes-43c36fc.js"></script>

As you can see, the JavaScript id already includes a hash (lazysizes-43c36fc.js), so you don't have to worry about cooking resources, neither caching the wrong file. The hash will only change if the related code changes.

(To understand better the work done, check the following pull requests: Deprecate resource registries and Create JS resource with unique ID.)

What are the benefits of this for developers?

you can work on your project using latest state-of-the-art JavaScript technologies like ES2015 and Vue.js
you don't have to worry about resource registries anymore as this approach works out of the box in all Plone versions: no more duplicated registrations, cleaner and simpler ZCML and XML files
you don't have to worry about upgrade steps anymore (at least the ones related with resource registry cooking/bundling)
you end up with better static resources: all CSS and JS files minimized, all images optimized
your code will be faster, as rendering a viewlet is less expensive than rendering a resource in the registry (at least, in Plone 4.3)

What are the benefits of this for system administrators?

less steps to upgrade your sites
less unneeded cache invalidations
faster sites as resources can be loaded in parallel

See how easy is to add a new feature to our resources: Add lazysizes print plugin; and how easy is to update the version of the JavaScript library we're using: Upgrade lazysizes to 4.1.1.

Now you can concentrate in the things that matter and stop worrying about the ugly parts; isn't that fun?

Share and enjoy!

Configuring better load balancing and health checks for Plone with HAProxy

2017-12-18T15:53:00.003-06:00

In the previous post of this blog I enumerated some problems we face when using normal load balancer health checks on Plone instances and I described a possible solution using five.z2monitor. In this post I'm going to show you how to configure HAProxy and other components of the stack in order to get the most of it.

As mentioned previously, we can do load balancing using nginx, Varnish or HAProxy. Both nginx and Varnish provide more features on their commercial versions, but I'm going to focus only on the community versions. In my opinion, if you ever need more features in your load balancer you should try HAProxy before buying one of those subscriptions.

On nginx the load balancer is handled by the ngx_http_upstream_module, it's very easy to enable and configure and it has some nice features (like backup servers and the least connections method), but also many limitations (like basic health checks and no queue control). This works very well on small sites.

On Varnish the load balancer is handled by the directors module, and I have to admit that I'm not impressed by its features as it lacks support for least connections which is the chief method with Plone instances. On the other side, backend definitions and health checks are more configurable, but support HTTP requests only. You may consider Varnish as a load balancer if you already use it on high traffic sites and is going to do a good and honest work.

(Previously, I was arguing in favor of the hash method, but later tests showed that the standard round robin is less prone to instance overloads.)

As you can see both nginx and Varnish can only handle HTTP requests for health checks and we have very limited control of request queues, something that in my experience has prove to be an issue.

(I never care about the ngnix queue, but in Varnish I always configure a max_connections directive to avoid flooding instances with requests that could make them unreachable for a long time.)

Let's go back on track: HAProxy is a very complex piece of software; it's so complex and its documentation so huge that I preferred just to removed it from the equation when I first had to take care of the infrastructure in our company some years ago.

That choice proved to be right at the moment, but I always like to investigate and act as my own devil's advocate from time to time. So, when one of our main sites started receiving some very nasty DoS attacks I started playing with it again.

It took me a whole week to prepare, test and review the configuration I'm going to share with you, but first I'm going to talk a little bit about our infrastructure and rationale.

The sites I'm talking about use (at least) 2 servers with a full stack: nginx, Varnish, HAProxy and Zope/Plone; both servers are configured using ZODB Replicated Storage (ZRS), one acting as the master ZEO server and the other as the slave. All Plone instances point to both ZEO servers. When the master ZEO server fails, all instances connect automatically to the slave in read-only mode. When all instances in one server fail, Varnish connects to the HAProxy in the other server as a backup. When Varnish fails, nginx tries to use the Varnish on the other server. That's more or less how it works.

This is our working haproxy.cfg and I going to explain some choices I made:

First the standard listen section: we use option tcp-check (layer 4) to make the health checks as is way faster that doing a HTTP (layer 7) check: we ask Zope on the alternate binded port for the ok command and expect the OK string as a result.

In my tests layer 4 checks took just a couple of milliseconds to finish, 2 orders of magnitude less than layer 7 checks. Faster, non-blocking responses mean you can check more frequently and you're going to detect sooner any failure on the instance.

Health checks are done every 2 seconds and, in case of failure, the slowstart parameter avoids flooding an instance during it's defined warming up period of one minute.

We inform the status of the backend upstream to Varnish using the monitor-uri directive: if we have no backends alive, HAProxy's monitor will also fail.

In Varnish we have have configured the backend like this:

Varnish will detect any backend failure in 2 seconds, at worst; Varnish will then try to use the HAProxy backend on the other server as a backup.

Back into HAProxy's configuration, we set maxconn 4 on backend servers to avoid sending more that 4 request to any instance; similarly, we set timeout queue 1m to keep the queue size under control.

Finally, the global section: you'll see we're using maxconn 256; in my opinion there is no reason to use a bigger value: if you have more that 256 requests on queue, you obviously have a problem and you need more instances to serve the traffic.

The typical stats screen of HAProxy with this configuration looks like this:

As you can see, over a period of almost 25 days we have no backend failures in this server, even with a large number of individual instance failures.

Last, but not least: if you want to cut the start up time of your instances you have to include the following directives on your buildout configuration (more about that on this Plone forum thread):


[instance]

…

# play with this value depending on the number of objects in your ZODB

zeo-client-cache-size = 256MB

zeo-client-client = zeoclient

Share and enjoy!

We have been doing health checks wrong in Plone

2017-12-11T20:16:00.002-06:00

In the previous post of this blog I was arguing on how to increase the performance of high-traffic Plone sites. As mentioned there, we have different ways to do so: increasing the number of server threads, increasing the number of instances, and increasing both.

Increasing the number of threads is easier and consumes less memory but, as mentioned, is less resilient and can be affected by the Python GIL on multicore servers. Increasing the number of instances, on the other side, will increase the complexity of our stack as we will need to install a load balancer, a piece of hardware or software that distributes the incoming traffic across all the backend instances.

Over the years we have tried different software load balancers depending on the traffic of a site: we use nginx alone as web server and load balancer on smaller sites and we add Varnish as web accelerator and load balancer when the load increases; lately we started using HAProxy again to solve extreme problems on sites being continuously attacked (I'll write about on a different post).

When you use a load balancer you have to do health checking, as the load balancer needs to know when one of the backend instances has become unavailable because is quite busy to answer further requests, has been restarted, is out for maintenance, or is simply dead. And, in my opinion, we have been doing it wrong.

The typical configuration for health checking is to send requests to the same port used by the Zope HTTP server and this has some fundamental problems: First, the Zope HTTP server is slow to answer requests: it can take hundreds of milliseconds to answer even the most simple HEAD request.

To make things worst, the requests that are answered by the Zope HTTP server are the slower ones (content not in ZODB cache, search results, you name it…), as the most common requests are already being served by the intermediate caches. In the tests I made, I found that even a well configured server running a mature code base can take as many as 10 seconds to answer this kind of requests. This is a huge problem as health check requests start queuing and timing out taking perfectly functional instances out of the pool, and making things just worst.

To avoid this problem we normally configure health checks with long intervals (typically 10 seconds) and windows of 3 failures for every 5 checks. And this, of course, creates another problem as the load balancer takes, in our case, up to 30 seconds to discover that an instance has been restarted when using things like Supervisor and its memmon plugin, leading to a lot of 503 Service Unavailable errors in the mean time.

So, it's a complete mess no mater how you analyze it and I needed to find a way to solve it: enters five.z2monitor.

five.z2monitor plugs zc.monitor and zc.z3monitor into Zope 2, enabling another thread and port to handle monitoring. To install it you just need to add something like this into your buildout configuration:


[buildout]

eggs =

    …
    five.z2monitor
zcml =
    …
    five.z2monitor


[instance]

…
zope-conf-additional =
    <product-config five.z2monitor>
        bind 127.0.0.1:8881

    </product-config>

After running buildout and restarting your instance you can communicate with your Zope server over the new port using different commands, called probes:


$ bin/instance monitor help

Supported commands:

  dbinfo -- Get database statistics

  help -- Get help about server commands

  interactive -- Turn on monitor's interactive mode

  monitor -- Get general process info

  ok -- Return the string 'OK'.

  quit -- Quit the monitor

  zeocache -- Get ZEO client cache statistics

  zeostatus -- Get ZEO client status information

Suppose you want to get the ZEO client cache statistics for this instance; all you have to do is use the following command:

$ bin/instance monitor zeocache main
417554 895451465 435095 900622900 35429160

You can also use the Netcat utility to get the same information:

$ echo 'zeocache main' | nc -i 1 127.0.0.1 8881
417753 896710955 435422 901905068 35467686

It's easy to extend the list of supported commands by writing you own probes; in the list above I have added to the default command set one that I create to know if the server is running or not; it's called "ok", and here is its source code:

We have now a dedicate port and thread that can be used for health checking:

$ echo 'ok' | nc -i 1 127.0.0.1 8881
OK

With this we solved most of the problems I mentioned above: we have faster response time and no queuing; we can decrease the health check interval to a couple of seconds and we are almost sure that a failure is a failure and not just a timeout.

Note we can't use this with nginx, nor Varnish, as their health checks are limited and expect the same port used for HTTP requests; only HAProxy supports this configuration.

So, in the next post I'll show you how to configure HAProxy health checks to use this probe and how to reduce the latency to a couple of milliseconds.

Plone performance: threads or instances?

2016-12-23T11:28:00.003-06:00

Recently we had a discussion on the Plone community forum on how to increase performance of Plone-based sites.

I was arguing in favor of instances, because some time ago I read a blog post by davisagli taking about the impact of the Python GIL on performance of multicore servers. Others, like jensens and djay, were skeptical on this argument and told me not to overestimate that.

So, I decide to test this using the production servers of one of our customers.

The site is currently running on 2 different DigitalOcean servers with 4 processors and 8GB RAM; we are using Cloudflare in front of it, and round-robin, DNS-based load balancing.

Prior to my changes, both servers were running with the same configuration:

nginx, doing caching of static files and proxy rewrites
Varnish, doing caching and URL-based load balancing
4 Plone instances running on ZEO client mode, with 1 thread and 100.000 objects in cache

Both servers where running also a ZEO server on ZRS configuration, one as a master and the other as a slave, doing blob storage replication.

First, here we have some information from this morning, before I made the changes. Here are some graphics I obtained using New Relic on the master:

Here is the same information from the slave:

As you can see, everything is running smoothly: CPU consumption is low and memory consumption is high and… yes, we have an issue with some PhamtomJS processes left behind.

This is what I did later:

on the master server, I restarted the 4 instances
on the slave server, I changed the configuration of instance1 to use 4 threads and restarted it; I stopped the other 3 instances

I also stopped the memmon Supervisor plugin (just because I had no idea on how much memory the slave server instance will be consuming after the changes), and killed all PhamtomJS processes.

The servers have been running for a couple of hours now and I can share the results. This is the master server:

And this is now the slave:

The only obvious change here is in memory consumption: wow! the sole instance on the slave server is consuming 1GB less than the 4 instances in the master server!

Let's do a little bit more research now. Here we have some information on database activity on the master server (just one instance for the sake of simplicity):

Now here is some similar information for the slave server:

I can say that I was expecting this: there's a lot more activity and the caching is not very well utilized on the slave server (see, smcmahon, that's the beauty of the URL-based load balancing on Varnish demonstrated).

Let's try a different look, now using the vmstat command:

Not many differences here: the CPU is idle most of the time and the interrupts and context switching are almost the same.

Now let's see how much our instances are being used, with the varnishstat command:

Here you can see why there's not too much difference: in fact Varnish is taking care of nearly 90% of the requests and we have only around 3 requests/second hitting the servers.

Let's make another test to see how quickly we are responding the requests using the varnishhist command:

Again, there's almost no difference here.

Conclusion: for our particular case, using threads seems not to affect too much the performance and has a positive impact on memory consumption.

What I'm going to do now is to change the configuration used in production to have 2 instances and 2 threads… why? because restarting a single instance on a server for maintenance purposes would let us without backends during the process if we were using only one instance.

Share and enjoy!

Using pep257, a Python docstring style checker, with Buildout

2014-04-30T19:52:00.000-05:00

PEP 257 documents the semantics and conventions associated with Python docstrings. According to it:

A docstring is a string literal that occurs as the first statement in a module, function, class, or method definition. Such a docstring becomes the __doc__ special attribute of that object.

This is specially useful, for instance, when you are introspecting code and you want to know what a specific method or function does.

pep257 is a Python docstring style checker.

I already filled a feature request on plone.recipe.codeanalysis to add support for it; meanwhile, if you want to use it in your Buildout-based projects, you can add the following to your buildout configuration:

After running bin/buildout you will find a bin/pep257 script.

A typical output will look like this:

(python-2.7)# hvelarde@nanovac (master * u+1) ~/collective/polls 
# bin/pep257 src/
src/collective/__init__.py:1 at module level:
        D100: Docstring missing
src/collective/polls/js_i18n_helper.py:1 at module level:
        D100: Docstring missing
src/collective/polls/js_i18n_helper.py:8 in public class `LegendOthersTranslation`:
        D101: Docstring missing
src/collective/polls/js_i18n_helper.py:14 in public method `render`:
        D102: Docstring missing
src/collective/polls/testing.py:1 at module level:
        D100: Docstring missing
src/collective/polls/testing.py:10 in public class `Fixture`:
        D101: Docstring missing
src/collective/polls/testing.py:14 in public method `setUpZope`:
        D102: Docstring missing
src/collective/polls/testing.py:19 in public method `setUpPloneSite`:
        D102: Docstring missing
src/collective/polls/config.py:1 at module level:
        D100: Docstring missing
src/collective/polls/polls.py:1 at module level:
        D100: Docstring missing
src/collective/polls/polls.py:21 in public class `IPolls`:
        D101: Docstring missing
src/collective/polls/polls.py:24 in public method `recent_polls`:
        D400: First line should end with '.', not 's'
src/collective/polls/polls.py:24 in public method `recent_polls`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:27 in public method `uid_for_poll`:
        D400: First line should end with '.', not 'l'
src/collective/polls/polls.py:27 in public method `uid_for_poll`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:30 in public method `poll_by_uid`:
        D400: First line should end with '.', not 'd'
src/collective/polls/polls.py:30 in public method `poll_by_uid`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:33 in public method `voters_in_a_poll`:
        D400: First line should end with '.', not 'l'
src/collective/polls/polls.py:33 in public method `voters_in_a_poll`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:36 in public method `voted_in_a_poll`:
        D400: First line should end with '.', not 'd'
src/collective/polls/polls.py:36 in public method `voted_in_a_poll`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:39 in public method `allowed_to_edit`:
        D401: First line should be imperative: 'i', not 'is'
src/collective/polls/polls.py:39 in public method `allowed_to_edit`:
        D400: First line should end with '.', not 'l'
src/collective/polls/polls.py:39 in public method `allowed_to_edit`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:42 in public method `allowed_to_view`:
        D401: First line should be imperative: 'I', not 'Is'
src/collective/polls/polls.py:42 in public method `allowed_to_view`:
        D400: First line should end with '.', not 'l'
src/collective/polls/polls.py:42 in public method `allowed_to_view`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:45 in public method `allowed_to_vote`:
        D400: First line should end with '.', not 'l'
src/collective/polls/polls.py:45 in public method `allowed_to_vote`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:48 in public method `anonymous_vote_id`:
        D400: First line should end with '.', not 'd'
src/collective/polls/polls.py:48 in public method `anonymous_vote_id`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:52 in public class `Polls`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/polls.py:52 in public class `Polls`:
        D400: First line should end with '.', not 's'
src/collective/polls/polls.py:52 in public class `Polls`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:60 in public method `ct`:
        D102: Docstring missing
src/collective/polls/polls.py:64 in public method `mt`:
        D102: Docstring missing
src/collective/polls/polls.py:68 in public method `wt`:
        D102: Docstring missing
src/collective/polls/polls.py:72 in public method `member`:
        D102: Docstring missing
src/collective/polls/polls.py:75 in private method `_query_for_polls`:
        D400: First line should end with '.', not 's'
src/collective/polls/polls.py:75 in private method `_query_for_polls`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:81 in public method `uid_for_poll`:
        D400: First line should end with '.', not 'l'
src/collective/polls/polls.py:81 in public method `uid_for_poll`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:85 in public method `recent_polls`:
        D400: First line should end with '.', not 's'
src/collective/polls/polls.py:85 in public method `recent_polls`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:98 in public method `poll_by_uid`:
        D400: First line should end with '.', not 'd'
src/collective/polls/polls.py:98 in public method `poll_by_uid`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:109 in public method `voted_in_a_poll`:
        D400: First line should end with '.', not 'd'
src/collective/polls/polls.py:109 in public method `voted_in_a_poll`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:128 in public method `allowed_to_edit`:
        D401: First line should be imperative: 'I', not 'Is'
src/collective/polls/polls.py:128 in public method `allowed_to_edit`:
        D400: First line should end with '.', not 'l'
src/collective/polls/polls.py:128 in public method `allowed_to_edit`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:133 in public method `allowed_to_view`:
        D401: First line should be imperative: 'I', not 'Is'
src/collective/polls/polls.py:133 in public method `allowed_to_view`:
        D400: First line should end with '.', not 'l'
src/collective/polls/polls.py:133 in public method `allowed_to_view`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:137 in public method `allowed_to_vote`:
        D401: First line should be imperative: 'i', not 'is'
src/collective/polls/polls.py:137 in public method `allowed_to_vote`:
        D400: First line should end with '.', not '?'
src/collective/polls/polls.py:137 in public method `allowed_to_vote`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:156 in public class `PollPortletRender`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/polls.py:156 in public class `PollPortletRender`:
        D204: Expected 1 blank line *after* class docstring, found 0
src/collective/polls/polls.py:156 in public class `PollPortletRender`:
        D400: First line should end with '.', not 'w'
src/collective/polls/polls.py:181 in public method `render_portlet`:
        D202: No blank lines allowed *after* method docstring, found 1
src/collective/polls/polls.py:251 in public method `render`:
        D400: First line should end with '.', not 'e'
src/collective/polls/__init__.py:1 at module level:
        D100: Docstring missing
src/collective/polls/subscribers.py:1 at module level:
        D100: Docstring missing
src/collective/polls/subscribers.py:17 in public function `fix_permissions`:
        D401: First line should be imperative: 'Thi', not 'This'
src/collective/polls/subscribers.py:17 in public function `fix_permissions`:
        D400: First line should end with '.', not 'f'
src/collective/polls/subscribers.py:17 in public function `fix_permissions`:
        D205: Blank line missing between one-line summary and description
src/collective/polls/subscribers.py:17 in public function `fix_permissions`:
        D208: Docstring is over-indented
src/collective/polls/subscribers.py:17 in public function `fix_permissions`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/subscribers.py:37 in public function `remove_votes`:
        D401: First line should be imperative: 'Thi', not 'This'
src/collective/polls/subscribers.py:37 in public function `remove_votes`:
        D400: First line should end with '.', not 't'
src/collective/polls/subscribers.py:37 in public function `remove_votes`:
        D205: Blank line missing between one-line summary and description
src/collective/polls/subscribers.py:37 in public function `remove_votes`:
        D208: Docstring is over-indented
src/collective/polls/subscribers.py:37 in public function `remove_votes`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/setuphandlers.py:1 at module level:
        D100: Docstring missing
src/collective/polls/setuphandlers.py:12 in public class `HiddenProfiles`:
        D101: Docstring missing
src/collective/polls/setuphandlers.py:18 in public method `getNonInstallableProfiles`:
        D102: Docstring missing
src/collective/polls/setuphandlers.py:23 in public function `updateWorkflowDefinitions`:
        D103: Docstring missing
src/collective/polls/setuphandlers.py:29 in public function `setupVarious`:
        D103: Docstring missing
src/collective/polls/portlet/voteportlet.py:1 at module level:
        D100: Docstring missing
src/collective/polls/portlet/voteportlet.py:24 in public function `PossiblePolls`:
        D103: Docstring missing
src/collective/polls/portlet/voteportlet.py:46 in public class `IVotePortlet`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/portlet/voteportlet.py:46 in public class `IVotePortlet`:
        D400: First line should end with '.', not 't'
src/collective/polls/portlet/voteportlet.py:46 in public class `IVotePortlet`:
        D205: Blank line missing between one-line summary and description
src/collective/polls/portlet/voteportlet.py:87 in public class `Assignment`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/portlet/voteportlet.py:102 in public method `__init__`:
        D102: Docstring missing
src/collective/polls/portlet/voteportlet.py:111 in public method `title`:
        D401: First line should be imperative: 'Thi', not 'This'
src/collective/polls/portlet/voteportlet.py:111 in public method `title`:
        D400: First line should end with '.', not 'e'
src/collective/polls/portlet/voteportlet.py:111 in public method `title`:
        D205: Blank line missing between one-line summary and description
src/collective/polls/portlet/voteportlet.py:118 in public class `Renderer`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/portlet/voteportlet.py:129 in public method `utility`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/portlet/voteportlet.py:135 in public method `portlet_manager_name`:
        D102: Docstring missing
src/collective/polls/portlet/voteportlet.py:149 in public method `poll`:
        D102: Docstring missing
src/collective/polls/portlet/voteportlet.py:170 in public method `poll_uid`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/portlet/voteportlet.py:176 in public method `getVotingResults`:
        D102: Docstring missing
src/collective/polls/portlet/voteportlet.py:184 in public method `can_vote`:
        D102: Docstring missing
src/collective/polls/portlet/voteportlet.py:193 in public method `available`:
        D102: Docstring missing
src/collective/polls/portlet/voteportlet.py:202 in public method `is_closed`:
        D102: Docstring missing
src/collective/polls/portlet/voteportlet.py:208 in public class `AddForm`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/portlet/voteportlet.py:208 in public class `AddForm`:
        D204: Expected 1 blank line *after* class docstring, found 0
src/collective/polls/portlet/voteportlet.py:217 in public method `create`:
        D102: Docstring missing
src/collective/polls/portlet/voteportlet.py:221 in public class `EditForm`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/portlet/voteportlet.py:221 in public class `EditForm`:
        D204: Expected 1 blank line *after* class docstring, found 0
src/collective/polls/portlet/__init__.py:1 at module level:
        D100: Docstring missing
src/collective/polls/Extensions/Install.py:1 at module level:
        D100: Docstring missing
src/collective/polls/Extensions/Install.py:7 in public function `uninstall`:
        D103: Docstring missing
src/collective/polls/Extensions/__init__.py:1 at module level:
        D100: Docstring missing
src/collective/polls/tests/__init__.py:1 at module level:
        D100: Docstring missing
src/collective/polls/content/__init__.py:1 at module level:
        D100: Docstring missing
src/collective/polls/content/poll.py:1 at module level:
        D100: Docstring missing
src/collective/polls/content/poll.py:34 in public class `InsuficientOptions`:
        D101: Docstring missing
src/collective/polls/content/poll.py:38 in public class `IPoll`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/content/poll.py:38 in public class `IPoll`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:80 in public method `validate_options`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:90 in public class `Poll`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/content/poll.py:90 in public class `Poll`:
        D400: First line should end with '.', not 'e'
src/collective/polls/content/poll.py:90 in public class `Poll`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:101 in public method `annotations`:
        D102: Docstring missing
src/collective/polls/content/poll.py:105 in public method `utility`:
        D102: Docstring missing
src/collective/polls/content/poll.py:109 in public method `getOptions`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:115 in private method `_getVotes`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:133 in public method `getResults`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:144 in private method `_validateVote`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:155 in private method `_setVoter`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:180 in public method `voters`:
        D102: Docstring missing
src/collective/polls/content/poll.py:186 in public method `total_votes`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:192 in public method `setVote`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:217 in public class `PollAddForm`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/content/poll.py:217 in public class `PollAddForm`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:223 in public method `create`:
        D102: Docstring missing
src/collective/polls/content/poll.py:235 in public class `PollEditForm`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/content/poll.py:235 in public class `PollEditForm`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:241 in public method `updateWidgets`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:258 in public method `applyChanges`:
        D102: Docstring missing
src/collective/polls/content/poll.py:270 in public class `View`:
        D101: Docstring missing
src/collective/polls/content/poll.py:277 in public method `update`:
        D102: Docstring missing
src/collective/polls/content/poll.py:346 in public method `can_vote`:
        D102: Docstring missing
src/collective/polls/content/poll.py:357 in public method `can_edit`:
        D102: Docstring missing
src/collective/polls/content/poll.py:362 in public method `has_voted`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:371 in public method `poll_uid`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:377 in public method `getOptions`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:382 in public method `getResults`:
        D200: One-line docstring should not occupy 2 lines

How to get statistics about your contributions on a GitHub organization

2014-01-08T14:34:00.000-06:00

GitHub, the web-based hosting service for software development projects, lets you check statistics on your contributions on specific projects easily.

You can check, for instance, the statistics on collective.cover just by visiting the following page:

https://github.com/collective/collective.cover/graphs/contributors

GitHub gives you a count of commits and a visual representation of them among time.

If you want to check only the number of commits by author, you could use the git-shortlog command:

The -s option gives you a summary output and the sort command shows the information sorted in reverse order (-nr).

You could improve this listing by using the .mailmap feature to add commits belonging to the same author using two different email addresses.

Image now that you want to get a list of all your contributions on a specific organization, lets say, the Plone collective.

Enter the GitHub API and github3.py, a Python wrapper for it.

Until today, the Plone collective has more that 1.1k repositories so we need to use the authenticated access to the API to avoid depletion of requests (rate limit allow us to make up to 60 requests per hour for unauthenticated requests and 5,000 for authenticated requests). In this specific example we used 1,171 requests to the API.

Using github3.py we iterate over all of the organization repositories and get information of each one only if my user name is listed among the contributors. We get the results as a list of tuples (repo, total).

After that we sort the list in reverse order using the total commits as the key and make the sum of all commits in general.

As you can see I have contributed to 74 repositories on the Plone collective making 2,975 commits in total.

Not bad, isn't it? :-)

Integrating Travis CI with your Plone add-ons hosted on GitHub

2012-08-08T12:50:00.001-05:00

Update 15/9/2012: Kudos for Asko Soukka who has developed an alternative method of installing Plone using the old good universal installer that reduces the amount of time needed by half. So go and read his post instead of loosing your time with mine.

I took me a little bit but, with the help of Mikko and Martin, I've got a couple of add-ons running tests with Travis CI.

Before setting up Travis CI, you have to make some changes to the Setup Script of your package.

In my case, my add-on package only works for Plone versions 4.1 and later, so I have added Products.CMFPlone as a dependency:

    …
    install_requires=[
        'setuptools',
        'Products.CMFPlone>=4.1',
        ],
    extras_require={
        'test': ['plone.app.testing'],
        },
    …

Products.CMFPlone contains a cut down feature set: just the things I need to run my tests.

Setting up Travis CI is pretty easy: just sign in and activate your GitHub Service Hook

Now, you need to configure your Travis CI build with a .travis.yml file in the root of your repo:

In my case I'm running tests for Plone's latest stable release (4.2 at I write this post) on top of Python 2.7.

Let's take a look at the travis.cfg buildout configuration file:

The main issue I experimented on my first tests was timeouts, so I have a couple of tricks here for you: first, we are extending the standard Plone testing buildout configuration that includes most declarations for us and takes care of always running the latest stable version; we are only going to use the test part. You need to add the package-extras just if your add-on is using plone.app.testing on the test option in extras_require of your package declaration as mentioned above.

zope.globalrequest is needed to run the tests and it was not included on Products.CMFPlone (this is already fixed on Plone's branches for versions 4.2 and 4.3). You may also need to include Pillow in test-eggs; just uncomment it the line.

We also need to add a socket-timeout of 3 seconds (only available on zc.buildout >= 1.5.0) and a list of allow-hosts to download the dependencies. This is pretty important and will avoid timeouts as Travis CI has hard time limits and timeouts are between 10 and 15 minutes for test suite runs (1).

Last, we have to replace the eggs option on the test part; we need to do this because we don't want to include neither Plone or plone.app.upgrade on the tests.

Finally, you can add a Status Image with a link back to the result of your last build on your README.txt file:

.. image:: https://secure.travis-ci.org/collective/your.package.png
    :target: http://travis-ci.org/collective/your.package

To run the tests you only need to make a push to your GitHub repo. Easy, isn't it?

For a live example of all I mentioned above, take a look at the collective.prettydate package.

Travis CI is really easy to set up and fun to use; I strongly recommend it and, if you like it, please show your love donating.

Ray Bradbury in memoriam

2012-06-07T13:05:00.002-05:00

The Illustrated Man turned in the moonlight. He turned again… and again… and again…
Ray Bradbury (The Illustrated Man, 1951)

Ray Bradbury, author of Fahrenheit 451 and one of the greatest science fiction writers of the 20th century, died at 91 two days ago.

I leave with you, as an homage, a fragment of one the most beautiful stories written by him.

Kaleidoscope (fragment)
[…]

The many good-bys. The short farewells. And now the great loose brain was disintegrating. The components of the brain which had worked so beautifully and efficiently in the skull case of the rocket ship firing through space were dying one by one; the meaning of their life together was falling apart. And as a body dies when the brain ceases functioning, so the spirit of the ship and their long time together and what they meant to one another was dying. Applegate was now no more than a finger blown from the parent body, no longer to be despised and worked against. The brain was exploded, and the senseless, useless fragments of it were far scattered. The voices faded and now all of space was silent. Hollis was alone, falling.

They were all alone. Their voices had died like echoes of the words of God spoken and vibrating in the starred deep. There went the captain to the Moon; there Stone with the meteor swarm; there Stimson; there Applegate toward Pluto; there Smith and Turner and Underwood and all the rest, the shards of the kaleidoscope that had formed a thinking pattern for so long, hurled apart.

And I? thought Hollis. What can I do? Is there anything I can do now to make up for a terrible and empty life? If only I could do one good thing to make up for the meanness I collected all these years and didn’t even know was in me! But there’s no one here but myself, and how can you do good all alone? You can’t. Tomorrow night I’ll hit Earth s atmosphere.

I’ll burn, he thought, and be scattered in ashes all over the continental lands. I’ll be put to use. Just a little bit, but ashes are ashes and they’ll add to the land.

He fell swiftly, like a bullet, like a pebble, like an iron weight, objective, objective all of the time now, not sad or happy or anything, but only wishing he could do a good thing now that everything was gone, a good thing for just himself to know about.

When I hit the atmosphere, I’ll burn like a meteor.

“I wonder,” he said, “if anyone’ll see me?”

The small boy on the country road looked up and screamed. “Look, Mom, look! A falling star!”

The blazing white star fell down the sky of dusk in Illinois. “Make a wish,” said his mother. “Make a wish.”

(1949)

Photo: Eneas.

Running pep8 before any commit on Git

2012-06-05T17:56:00.001-05:00

Readability counts.

The Zen of Python

One of the things that made me choose Python as a programming language in the first place was its readability.

PEP 8 —the Style Guide for Python Code— gives coding conventions for the Python code and pep8 is a tool to check your code against these conventions.

I use gedit as my editor and I have installed the developer plugins to check my code against PEP 8 every time I save a file but, as not everyone does this, Érico asked me today about a way to enforce this practice.

I started searching the web and I have compiled (from 1, 2 and 3) a nice solution using a pre-commit hook with Git (this is possible also with Subversion, but I'm not pretty interested on it right now):

First you need to be sure you are running Git version 1.7.1 or later, and that you have pep8 installed in your system (check the package documentation).

Create a directory to store the hooks globally:

mkdir -p ~/.git_template/hooks

Tell Git all new repositories you create or clone will use this directory for templates:

git config --global init.templatedir '~/.git_template'

Put the following script in the ~/.git_template/hooks directory:

Make the file executable:

chmod +x ~/.git_template/hooks/pre-commit

If you want to use this hook on an existing repository all you have to do is reinitialize it:

git init

Now the pre-commit hook script lives in the .git/hooks directory of your repository.

Test it trying to commit some changes: if the files you are trying to commit comply with PEP 8 (excepting the list of errors or warnings to ignore), the commit will be done as usual; if there are any issues, the commit will be aborted until you fix them.

Feel free to modify the list of errors and warnings to ignore, globally or from project to project, to fit your personal needs.

Remember PEP 8:

A style guide is about consistency. Consistency with this style guide is important. Consistency within a project is more important. Consistency within one module or function is most important.

But most importantly: know when to be inconsistent -- sometimes the style guide just doesn't apply. When in doubt, use your best judgment. Look at other examples and decide what looks best. And don't hesitate to ask!

Two good reasons to break a particular rule:

When applying the rule would make the code less readable, even for someone who is used to reading code that follows the rules.

To be consistent with surrounding code that also breaks it (maybe for historic reasons) — although this is also an opportunity to clean up someone else's mess (in true XP style).

Robot Framework and SeleniumLibrary for Plone developers

2012-05-30T19:36:00.001-05:00

Update 05/06/2012: Added a tip for handling overlays and a reference to datakurre's work.

Let's face it: zope.testbrowser sucks for functional and acceptance tests: tests are boring to write, hard to debug when they fail, and there's no support for JavaScript.

The latest is becoming more important every day, as Plone codebase now includes almost 30% of JavaScript code according to Ed Manlove (and Ohloh).

There are some alternatives that help on solving these issues and Selenium —a portable software testing framework for web applications— stands among the best.

Some years ago I started using Selenium to make some functional tests, but I abandoned the task because it was a little bit slow and boring.

During the sprints held after the Plone Conference 2011 in San Francisco, Godefroid Chapelle introduced us to Robot Framework —a generic test automation framework for acceptance testing and acceptance test-driven development (ATDD)— and SeleniumLibrary —a Robot Framework test library that uses the Selenium web testing tool internally.

Writing tests with Robot Framework is pretty easy: you can create tests by using RIDE —a light-weight and intuitive editor for Robot Framework test case files— or just by using your favorite text editor.

A test case is made with a list of keywords and you can create more keywords using the same syntax used for creating the test cases; you can use natural language and you can reuse your test code. This way you start pretty slow but you become more and more productive with every new keyword you add to your test suites.

After coming back home I started adding some very basic functional tests to some of the packages we have developed lately.

I'm not interested on writing a Robot Framework tutorial; there are many on the web (How to use Robot Framework with the Selenium Library is a pretty good one; part 2 and Extending Robot Framework to check emails, are also available).

The purpose of this post is to help other Plone developers on getting Robot Framework up and running and to share some tricks I've learned over the last few months.

Don't panic

Basically all you need to start playing with Robot Framework and SeleniumLibrary in Plone is a buildout configuration, a small resource file including some basic keywords for Plone, a couple of helper files and some test suites.

You can take a look at the test suites written during the sprint in the 4.1-robot branch of Plone's buildout.coredev on GitHub, and you can grab the basic files from there (pybot.cfg and the whole acceptance-tests and templates directories); we are going to modify them a little bit in a minute.

You will also need to download and install the Selenium IDE Plugins —an integrated development environment for Selenium scripts implemented as a Firefox extension— in your Firefox browser (sorry, but I don't know if there are other browser options available at the moment).

First, here is the simplified buildout configuration I've been working on based on the one created by Godefroid:

[buildout]
extends = buildout.cfg
parts += plonesite robot selenium library-settings

[versions]
selenium-server = 2.22.0

[plonesite]
recipe = collective.recipe.plonesite
profiles = my.package:default

[robot]
recipe = zc.recipe.egg
eggs =
    robotframework
    robotframework-seleniumlibrary
entry-points = pybot=robot:run_cli rebot=robot:rebot_cli
arguments = sys.argv[1:]

[selenium]
recipe = hexagonit.recipe.download
download-only = true
url = http://selenium.googlecode.com/files/selenium-server-standalone-${versions:selenium-server}.jar
filename = selenium-server.jar

[library-settings]
recipe = collective.recipe.template
input = templates/library-settings.txt.in
output = ${buildout:directory}/acceptance-tests/library-settings.txt

We extend our standard development configuration by adding four parts: plonesite, that creates a new Plone site, if there is none, and runs the profiles listed (my.package:default); robot, that downloads the robotframework and robotframework-seleniumlibrary eggs, and creates the commands needed to run the tests (yes, this is 2012 and they are still unaware of egg entry points); selenium, that downloads the Selenium standalone server; and, finally, library-settings, that is used to initialize some variables used by the tests, it takes a template file (templates/library-settings.txt.in) and replaces buildout variables with their values creating a new file (acceptance-tests/library-settings.txt).

Please note the use of a ${versions:selenium-server} variable in the selenium part of the configuration; this helps us on keeping it up to date just like with any other egg.

Let's get started: add the files you downloaded to the buildout you want to test. Run bin/buildout -c pybot.cfg and you will find a couple of new scripts inside your bin directory: pybot and rebot.

Start your instance as usual using bin/instance fg and, on another terminal window, run bin/pybot acceptance-tests. Sit comfortably; let the show begin…

After a few seconds a couple of Firefox windows will appear: the first one is the RemoteRunner which is required to control the Plone site running in the second window.

The tests will run and you will get a complete report of their results as a couple of HTML files: report.html includes summary information, test statistics and details; log.html includes a complete log of all tests executed.

In case of error, log.html includes detailed information about it, the source code of the page where the error ocurred and a screenshot of it.

Test Report

Test Log

Share and enjoy

And now for something completely different… the tips and tricks.

What to include in your .gitignore file

Most of our packages live on GitHub and I like include a list of objects to be ignored from version control using a .gitignore file; the following is what I use to add in packages using Robot Framework and SeleniumLibrary tests:

library-settings.txt
log.html
output.xml
report.html
selenium*

Running a single test suite

If you want to run only one test suite you can do something like this:

bin/pybot -s test_suite acceptance-tests

Handling overlays

Suppose you have to wait for an overlay window to show up in the screen. By default, a page load is expected to happen whenever a link or image is clicked, or a form submitted. In this case we pass the don't wait argument to the keyword.

Delete Item
    [Arguments]  ${title}

    Click Link  ${title}
    Click Link  Delete  don't wait
    Wait Until Page Contains  Do you really want to delete this item?
    Click Button  Delete

As there is no Wait Until Page Does Not Contains nor Wait Until Page Does Not Contains Element keywords, you will have to use something like this in case you want to know if an overlay was closed:

    Wait Until Keyword Succeeds  1  5  Page Should Not Contain  ${text}

The trick here is the Wait Until Keyword Succeeds keyword. If the specified keyword does not succeed within timeout, this keyword fails and waits lapse of time before trying to run the keyword again.

Uploading files and images

This is a typical test case: let's say you developed a new content type that includes a file or image field and you want to test it. A basic test case could be something similar to this one:

*** Settings ***

Resource  plone.txt
Suite Setup  Setup

*** Variables ***

${link_locator} =  a#file
${input_identifier} =  input#file_file

*** Test cases ***

Test Add Audio File
    Goto Homepage
    Add File  ${PATH_TO_TEST_FILES}/test.mp3

*** Keywords ***

Setup
    Log In  admin  admin

Add File
    [arguments]  ${file}

    Open Add New Menu
    Click Link  css=${link_locator}
    Page Should Contain  Add File
    Choose File  css=${input_identifier}  ${file}
    Click Button  Save
    Page Should Contain  Changes saved

The variable ${PATH_TO_TEST_FILES} could be declared in your library-settings.txt.in file as something like this (please note theres are 2 spaces after the equal sign):

*** Variables ***

${dollar}{PATH_TO_TEST_FILES} =  ${buildout:directory}/src/my/package/tests

The trick here is the use of a ${dollar} variable that is defined in the
library-settings part of our pybot.cfg buildout configuration as:

[library-settings]
…
dollar = $

In case you need to upload an image you will replace the CSS selectors ${link_locator} and ${input_identifier} with a#image and input#image_file respectively.

So long, and thanks for all the fish

Robot Framework and SeleniumLibrary makes your life much easier and fun when writing functional and acceptance tests: you become more productive with the addition of every new keyword and debugging failures is child's play with the help of page source code and screenshots.

The main drawback at the moment is that we lack a good resource file including more keywords covering other Plone features, but that can be solved easily as soon as more people start using Robot Framework and we find a way to collaborate on this.

Another important point that I had forgotten to mention, until Asko Soukka remembered it to me, is that we don't have a concept of layers to set up fixtures and a way to clean up global state of the Plone site. This is particularly useful if you want to return your site to its original state without having to remove all changes made by some test case. Asko has made some advances on that lately.

Adding test users programmatically using collective.recipe.plonesite

2012-01-23T23:45:00.000-06:00

I'm writing some functional tests for a Plone project and I need to add a group of test users every time I create a test site.

collective.recipe.plonesite is a cool Buildout recipe that enables you to create and update a Plone site as part of a buildout run.

I added the following lines to my buildout configuration:

parts =
    …
    plonesite

[plonesite]
recipe = collective.recipe.plonesite
profiles = my.project:default
post-extras = ${buildout:directory}/acceptance-tests/add_test_users.py

The code of the add_test_users.py script is pretty straightforward:

"""This script will add a number of test users to a Plone site.

You can used it in post-extras option of collective.recipe.plonesite. It will
be evaluated after running QuickInstaller and GenericSetup profiles.

@param portal: The Plone site as defined by the site-id option
"""

import logging
logger = logging.getLogger('collective.recipe.plonesite')

test_users = [
    # (username, password, group),
    ('username1', 'password1', 'group1'),
    ('username2', 'password2', 'group2'),
    ('username3', 'password3', 'group3'),
    ]

for username, password, group in test_users:
    if username not in portal.acl_users.getUserIds():
        try:
            portal.portal_registration.addMember(username, password)
            portal.portal_groups.addPrincipalToGroup(username, group)
        except ValueError:
            logger.warn('The login name "%s" is not valid.' % username)
        except KeyError:
            logger.warn('The group "%s" is not valid.' % group)

Enjoy!

(Next time I'll give you my first impressions on SeleniumLibrary, a web testing library for Robot Framework, I'm using to write the tests.)

Setting the right permissions on your blobstorage directory

2012-01-13T19:08:00.001-06:00

Have you ever been annoyed by a message saying that the blobstorage directory of your instance has an insecure mode setting?

That happens to me all the time, so today I spent a couple of minutes trying to figure out how to fix it.

In ZODB/blob.py we have the following:

class FilesystemHelper:
    # Storages that implement IBlobStorage can choose to use this
    # helper class to generate and parse blob filenames.  This is not
    # a set-in-stone interface for all filesystem operations dealing
    # with blobs and storages needn't indirect through this if they
    # want to perform blob storage differently.

    …

    def create(self):
        if not os.path.exists(self.base_dir):
            os.makedirs(self.base_dir, 0700)
            log("Blob directory '%s' does not exist. "
                "Created new directory." % self.base_dir)
        if not os.path.exists(self.temp_dir):
            os.makedirs(self.temp_dir, 0700)
            log("Blob temporary directory '%s' does not exist. "
                "Created new directory." % self.temp_dir)

        if not os.path.exists(os.path.join(self.base_dir, LAYOUT_MARKER)):
            layout_marker = open(
                os.path.join(self.base_dir, LAYOUT_MARKER), 'wb')
            layout_marker.write(self.layout_name)
        else:
            layout = open(os.path.join(self.base_dir, LAYOUT_MARKER), 'rb'
                          ).read().strip()
            if layout != self.layout_name:
                raise ValueError(
                    "Directory layout `%s` selected for blob directory %s, but "
                    "marker found for layout `%s`" %
                    (self.layout_name, self.base_dir, layout))

    def isSecure(self, path):
        """Ensure that (POSIX) path mode bits are 0700."""
        return (os.stat(path).st_mode & 077) == 0

    def checkSecure(self):
        if not self.isSecure(self.base_dir):
            log('Blob dir %s has insecure mode setting' % self.base_dir,
                level=logging.WARNING)

Then, the only thing you need to do is run chmod 700 var/blobstorage (owner can read, write and execute) in your installation directory.

Why this directory is created with a different setting (755) is a mystery to solve another day.

Hiding uninstall profiles using Grok

2011-12-20T14:34:00.000-06:00

Tired of seeing your uninstall profile listed even when your package has not being installed yet?

If you have five.grok among the dependencies of your package in setup.py, you can put the following code in a setuphandlers.py file:

from five import grok

from Products.CMFPlone.interfaces import INonInstallable


class HiddenProfiles(grok.GlobalUtility):

    grok.implements(INonInstallable)
    grok.provides(INonInstallable)
    grok.name('your.package')

    def getNonInstallableProfiles(self):
        profiles = ['your.package:uninstall']
        return profiles

Remember that you need to Grok your package using the following line in your configure.zcml:

<grok:grok package="." />

This cool tip has been brought to you by Érico Andrei; enjoy it!

Max Keiser interviews Grateful Dead's John Perry Barlow about activism and open source

2011-11-18T20:04:00.003-06:00

John Perry Barlow is a poet, essayist, retired rancher, political activist, former lyricist for the Grateful Dead and founding member of the Electronic Frontier Foundation.

Max Keiser interviews John Perry Barlow about activism and open source.

A must see!

Plone in news media

2011-11-05T09:57:00.000-06:00

Lightning talk at the Plone Conference 2011.

View more presentations from Héctor Velarde

Plone: pasado, presente y futuro

2011-10-20T11:18:00.000-05:00

Plone: pasado, presente y futuro

View more presentations from Héctor Velarde

Cómo habilitar la selección y copia de texto en el sitio de El Universal

2011-10-13T20:52:00.002-05:00

Hace un rato me topé con la novedad de que algunas páginas de El Universal no permiten seleccionar texto para copiarlo.

Al principio pensé que se trataba de algún script pero pronto descubrí que se era otra cosa: la inclusión en la hoja de estilo de la página de los seudoelementos -moz-user-select y -moz-user-focus.

Si usas Firefox, la solución es muy sencilla: crea un archivo userContent.css dentro la carpeta chrome de tu perfil y agrega las siguientes líneas:



* { -moz-user-select: auto !important }

* { -moz-user-focus: normal !important }

Reinicia tu navegador y listo.

Que nadie coarte tu derecho a compartir conocimiento.

Displaying (and filtering) the contents of a folderish content type

2011-08-02T19:27:00.003-05:00

If you need to list the contents of a folderish content type you can include something like this in your page template:

<fieldset id="folder-listing">
    <legend i18n:translate="">Contents</legend>
    <tal:block define="listing_macro context/folder_listing/macros/listing">
        <metal:use_macro use-macro="listing_macro" />
    </tal:block>
</fieldset>

Note that listing is a huge macro inside the skins/plone_content/folder_listing.pt template in the CMFPlone product.

You can even do things like content filtering or limiting the number of items shown with this macro:

<fieldset id="folder-listing">
    <legend i18n:translate="">Contents</legend>
    <tal:block define="listing_macro context/folder_listing/macros/listing;
                       contentFilter python:{'portal_type': ['File']};
                       limit_display python:5">
        <metal:use_macro use-macro="listing_macro" />
    </tal:block>
</fieldset>

Enjoy!

(Tested under Plone 4.1)

Plone: software libre y comunidad

2011-07-06T20:31:00.002-05:00

Charla sobre Plone y software libre en el marco del OpenSource Day de la XIII Semana de Ingeniería e Informática de la Universidad Justo Sierra.

Plone: software libre y comunidad

View more presentations from Héctor Velarde

Redirecting log output to sys.stdout on tests

2011-03-10T21:53:00.002-06:00

Use the following code inside your tests/base.py file if you need to redirect the log output to sys.stdout for debugging purposes:

import logging
import sys

logger = logging.getLogger('YOUR LOGGER NAME')
logger.setLevel(logging.DEBUG)
handler = logging.StreamHandler(sys.stdout)
formatter = logging.Formatter("%(asctime)s %(levelname)s %(name)s %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)

This way you can see debug(), info(), warning(), error() and critical() messages printed in the console.

More information: Logging facility for Python and Logging on Plone.

An Excel source pipeline section for collective.transmogrifier

2011-01-28T15:17:00.003-06:00

Using xlrd, and based on CSV source section, I've written a pipeline section to read data stored inside Excel spreadsheets.

The code looks like this:

import xlrd

    from zope.interface import classProvides
    from zope.interface import implements

    from collective.transmogrifier.interfaces import ISection
    from collective.transmogrifier.interfaces import ISectionBlueprint
    from collective.transmogrifier.utils import resolvePackageReferenceOrFile


    class ExcelReader:
        def __init__(self, f, s, *args, **kwds):
            self.fieldnames = None
            book = xlrd.open_workbook(f)
            sheet = book.sheet_by_name(s)
            self.reader = list()
            for row in range(0, sheet.nrows):
                self.reader.append([sheet.cell_value(row, col)
                                    for col in range(0, sheet.ncols)])
            self.reader = iter(self.reader)

        def __iter__(self):
            return self

        def next(self):
            row = self.reader.next()
            if self.fieldnames is None:
                self.fieldnames = row
                row = self.reader.next()
            return dict(zip(self.fieldnames, row))


    class ExcelSource(object):
        classProvides(ISectionBlueprint)
        implements(ISection)

        def __init__(self, transmogrifier, name, options, previous):
            self.previous = previous

            filename = resolvePackageReferenceOrFile(options['filename'])
            sheet = options['sheet']

            self.reader = ExcelReader(filename, sheet)

        def __iter__(self):
            for item in self.previous:
                yield item

            for item in self.reader:
                yield item

The pipeline section assumes that the field names are in the first row of the spreadsheet. Note also the use of iter() to convert the list of rows into an iterator.

To use the pipeline section you only need to add something like this in your transmogrifier's configuration:

[excelsource]
blueprint = your.package.excelsource
filename = excel_book
sheet = sheet_inside_the_book

Remember to register the utility in your configure.zcml file:

<utility
    component="your.package.ExcelSource"
    name="your.package.excelsource"
    />

xlrd supports Excel spreadsheets up to version 2007.

Pruebas en Plone: conceptos básicos y ejemplos

2009-11-12T22:18:00.003-06:00

Charla sobre pruebas en Plone para la reunión de noviembre de 2009 del Grupo de Usuarios de Plone en México.

Pruebas en Plone

View more presentations from Héctor Velarde

Plone en La Jornada

2008-11-11T20:55:00.006-06:00

Charla sobre la implementación de Plone en La Jornada para la reunión de noviembre de 2008 del Grupo de Usuarios de Plone en México.

Plone en La Jornada

View more presentations from Héctor Velarde

When adapting content types, schemaextender is the way to go

2008-01-11T12:42:00.007-06:00

Lightning Talks are among the best investments of time you can make on any Plone Conference: there's always a lot of smart people doing great things.

During the Plone Conference 2007, in Naples, Florian Schulze spoke about schemaxtender, a package that allows you to inject new fields into an Archetypes schema using an adapter. When I saw his talk, I knew I was going to use it for what I had in mind.

As I mentioned before in Mapping NITF into Plone's metadata, at La Jornada we were looking for a way to adapt Plone's standard News Item content type to:

add new fields to it (property, section, urgency and byline)
change fields' order among different schematas to make the edition of new content easier for the publishers

Using schemaextender to accomplish these tasks was so easy that I was really excited when I finished. I spent about 40 hours (in fact, a little bit more after the second release) to read Part 1 of Philip von Weitershausen's excellent book Web Component Development with Zope 3, to understand the way schemaextender works, to find out how to make it work on Plone 2.5, to start using the adapted content with Smart Folders, and even to write some tests for it.

All this work is available in a product called nitf4plone in case you want to try it (be aware this is an beta release). The product works on both, Plone 2.5 and Plone 3.0.

Let's analize the code... but first, a disclaimer: don't try this on Plone 2.5.

Why? schemaextender was written to work with Archetypes version 1.5 or later (that's, Plone 3.0 and later). There's a branch to make it work with Plone 2.5 patched by Erik Rose of the WebLion Project Team at PSU, but it will never be merged into the maintenance one. If you use this branch you are on your own in the event of a bug. Also, have in mind that there's no way back on doing this and after installing schemaextender in Plone 2.5, the adapter will be available for all sites in the instance.

So, yes, do as I say, not as I did.

Having warned you, let's dive a little bit into the code; schemaextender includes 3 types of adapters:

ISchemaExtender lets you can add new fields to a schema
IOrderableSchemaExtender lets you add new fields and reorder them
ISchemaModifier is a low-level hook that allows direct manipulation of the schema

You can find information and examples on all of them on the source code. As you might have expected, I decided to use IOrderableSchemaExtender.

To write the adapter, we need to declare the new fields first; let's take a look to the section field as an example:

class SectionField(ExtensionField, atapi.StringField):
     """Named section of a publication where a news object appear
     """

     def getDefault(self, instance):
          …
          return nitf.default_section

     def Vocabulary(self, instance):
          …
          return atapi.DisplayList([(x, x) for x in nitf.sections])

As you can see, we need to subclass from ExtensionField and StringField; please note that it's mandatory to keep this order. ExtensionField will provide standard accessors and mutators which are not generated on the class. StringField will provide standard attributes and the widget for the field. Also we override getDefault() and Vocabulary() methods to set the default value and vocabulary.

Let's take a look to the adapter's class now:

class NITFExtender(object):
    """Adapter to add NITF fields to News Items
    """
    implements(IOrderableSchemaExtender)
    adapts(IATNewsItem)

    fields = [
        …
        SectionField('section',
        languageIndependent = 1,
        enforceVocabulary = 1,
        required = 1,
        widget = atapi.SelectionWidget(
            label = 'Section',
            label_msgid = 'section',
            description = 'Named section where the news object appear',
            description_msgid = 'help_section',
            i18n_domain = 'nitf4plone')),
    …
    ]

def __init__(self, context):
    self.context = context

def getFields(self):
    return self.fields

def getOrder(self, original):
    # we only need to change the order of the fields in Plone 2.5
    if 'metadata' in original:
        # first we remove the fields from whichever schemata they are
        for schemata in original.keys():
            if 'relatedItems' in original[schemata]:
                original[schemata].remove('relatedItems')
            if 'subject' in original[schemata]:
                original[schemata].remove('subject')

        # now we insert them where we want them to appear
        idx = original['default'].index('property')
        original['default'].insert(idx, 'subject')
        original['metadata'].insert(0, 'relatedItems')
    return original

As I mentioned, our adapter implements IOrderableSchemaExtender. In Plone 3.0 adapters can be registered locally at installation time:

sm = portal.getSiteManager()
sm.registerAdapter(NITFExtender, (IATNewsItem, ), IOrderableSchemaExtender)

In Plone 2.5 we can't have local adapters and registrations aren't persistent, so we have to handle this in a different way:

Here you can see the way news items look after applying the adapter:

schemaextender in action: the news item now contains new attributes and the order of the fields is modified to make the work of the publishers easy.

The code of the adapter is pretty clean and easy to understand.

Having finished it, we followed Mikko Ohtamaa's procedure to adding new fields to Smart Folders search in order to display all news articles for a given section and it worked fine.

Right now we are working on the migration of the content of our site to use the new fields; we are also preparing some templates to display the new information and some CompositePack's viewlets to use them to create the front pages in a better way.

Please let me know if you find this development interesting or if you want to participate in some way.

I want to thank all the members of the Plone community who helped me answering my questions at the #plone channel on IRC and the Product Developers forum, specially Martin Aspeli and Florian Schulze (who helped me with the schemaxtender internals and were really patient with me), Mikel Larreategui and Erik Rose (who helped me with the installer), Wichert Akkerman and Andreas Jung (who are always available answering all sort of questions on the forums).

Relating content automatically in Plone

2008-01-02T12:33:00.000-06:00

A question arose today at the Plone general mailing list (a.k.a. Plone-users): it is possible to create a list of related content automatically?

Well, the answer is yes and I'm going to tell you how.

Some time ago Benjamin Saller created a proof-of-concept product called Haystack to do auto-classification of content. Haystack was built around Open Text Summarizer and the haystack_tool included a couple of methods to summarize text and to get a list of "topics" extracted from the content. Haystack also included some portlets to demonstrate its functionality.

We used Haystack in La Jornada for some time with mixed results: the summarizer worked well; we called it to create the description field of our content using Ajax in order to reduce the work of our publishers at edition time.

On the other hand, with the "topics" obtained we were creating a portlet that retrieved the related content. The main problems with this were the low quality of the "topics" and the implementation of the relation. Sometimes we had some embarrassing results relating content from Iraq with some other of, let's say, Shakira, just because they shared some "topic".

Haystack didn't understood the meaning of words and, of course, Ben Saller was aware of that. Last time I saw him was during the Plone Conference 2006 in Seattle. He gave a talk on Haystack 2.0 and he was really excited about its new features: linguistic mapping and automated conceptual mapping, providing high-quality relationships with little or no human effort.

Unfortunately for us, Ben has been a little bit away from the Plone community for some time. So I don't know what's the status on his work.

Going back to the original question in the mailing list, Matt Bowen pointed out to me that Yahoo! has a web service called Term Extraction that does almost the same thing and he even found a python implementation for it.

I tested Term Extraction with some text in Spanish and I was very pleased with the results:

<ResultSet xsi:schemaLocation="urn:yahoo:cate http://api.search.yahoo.com/ContentAnalysisService/V1/TermExtractionResponse.xsd">
    <Result>wong kar wai</Result>
    <Result>stephen frears</Result>
    <Result>festival de cannes</Result>
    <Result>sean penn</Result>
    <Result>25 de mayo</Result>
    <Result>cines</Result>
    <Result>organizadores</Result>
    <Result>evidencia</Result>
    <Result>el presidente</Result>
    <Result>hace mucho tiempo</Result>
    <Result>afp</Result>
    <Result>ya</Result>
</ResultSet>

Implementing this in Plone seems not to be quite complicated: you can trigger a script in a workflow transition, or use Content Rules in Plone 3.0, to fill the Subject field or, better, add an additional field to store this information. Just remember the Term Extraction web service is limited to 5,000 queries per IP address per day.

Yes, I know this solution suffers from the same problems that Haystack, but the "topics" obtained here have better quality and you can always find a better algorithm to do the relation, like testing for more than one "topic" or using only "topics" longer than one word.

Anyway I will put this in my list of pending stuff to test (with a little help of Matt Bowen, of course).