Last week I released the first alpha version of xml4h, a library I have created to make it easier to do non-trivial XML processing in Python.

You can find it on:

• PyPI for installation using pip: pypi.python.org/pypi/xml4h
• GitHub for code and issue tracking: github.com/jmurty/xml4h
• ReadTheDocs for (pretty comprehensive) documentation: xml4h.readthedocs.org

I made xml4h because I enjoy working with Python — it’s been my language of choice for a while now — but I always found it painful to work with XML using the tools available there. Of course, dealing with XML tends to be painful in its own right, but the existing Python tools only made it harder. Although there was an excellent base of fast, efficient and powerful XML processing tools, interacting with them always seemed to be harder than it should be.

For years I kept an eye out for a better way, until finally of got off my arse and built it myself.

Right now xml4h is young, a little rough, and likely to undergo changes: in other words it’s an alpha. My hope is it will grow quickly into a powerful and intuitive tool that makes it as easy to do XML in Python as it now is to do HTTP or to do SQL.

If you’re into Python and have felt XML-induced pain, it would be great if you could try it out and provide feedback, bug reports, and code to help move the project in the right direction.

Let’s make this thing awesome.

This release has been a long time coming, sorry about that. I had intended to get a release out late last year but personal factors left me short of time: relocating back to Australia from the U.S. and helping out with our newly-arrived baby boy have kept me pretty busy.

Still, the 0.9.0 version is now official. Here are some of the major new features and improvements.

HttpComponents Upgrade

A major change in the new version is JetS3t’s use of the newer 4.x generation of the key HttpClient library (now more accurately called HttpComponents). The older HttpClient 3.x had been end-of-lifed and while it still worked fine, relying on the obsolete version was not a good long-term option.

I need to thank two contributors in particular for doing a lot of this work. Cheers to Gilles Gaillard and David Kocher for their invaluable work; the upgrade wouldn’t have happened without them.

Note that since this upgrade involved updates to the core JetS3t HTTP code layer, there is a risk of subtle bugs in the HTTP handling with this release. I think the risk is small and some people have been using the pre-release 0.9.0 code successfully for a while now, but when you update to 0.9.0 it’s worth doing a little more testing than you might normally.

Here are some of the new service-specific features.

Amazon S3

• Support for multiple object deletes in a single request
• Explicit support for new S3 locations: Oregon (us-west-2), South America (sa-east-1), GovCloud US West (s3-us-gov-west-1), GovCloud US West FIPS 140-2 (s3-fips-us-gov-west-1)
• Support for server-side encryption, with per-object setting of algorithm and default algorithm configuration with the new s3service.server-side-encryption property
• Support for Multipart Upload Part – Copy operation, to add data from existing S3 objects to multipart uploads.
• Support for signing S3 requests with response-altering request parameters like response-content-type and response-content-disposition

Google Storage

• Support for OAuth2 authentication mechanism, with automatic access token refresh.

Conclusion

Please grab the latest version, try it out and let me know how it goes.

Visit the JetS3t web site to download the latest packaged release, view the code samples or read the API Javadoc.

For a more complete list of changes see the Release History or Release Notes documents.

Or go to the BitBucket developer site to access the latest code, report issues in the bug tracker, and contribute to the project.

P.S. The latest release is on its way to the official Maven2 repository and should be available within a day or so.

This probably isn’t the best place to put this code but hopefully it will help someone, most likely me at some time in the future. Unless perhaps sanity prevails and everyone starts using UTF-8 everywhere…

Be sure to only feed this method UTF-8 encoded text.

After working through some hairy code merges recently I realised I needed a better approach than relying on inline diffs, where merge conflicts are represented in a single file like so:

Inline diffs are great for resolving relatively simple conflicts but can quickly become confusing if conflicts span many lines or there are significant differences between files.

So I configured Mercurial to open vimdiff upon merge conflicts, but the default three-paned vertical-split view wasn’t quite what I wanted. It didn’t include the base version of the conflicted file, and the default window layout made it hard to see exactly what was going on.

A little research turned up a blog post showing how to better configure vimdiff when using git. We use Mercurial at work so I adapted this hint to work with Mercurial’s MergeProgram configuration:

This will show the merged file in a large window at the bottom with the three pre-merge files of interest — local changes, incoming/other changes, the base file — in a three-pane vertical split at the top. With this set-up and some practice using the vimdiff commands, complex conflicting merges are much easier to deal with.

If you use vim or want to, be sure to check out the excellent Vim Casts video podcasts to learn (or re-learn) how to get the most out of it. Some recent episodes discuss vimdiff in the context of a git workflow but are still full of useful pointers for those not using git.

This release has been a long time coming, mainly due to my reluctance to finish the documentation. But it’s finally here and comes with some great new features.

Goodies

• Support for Amazon S3′s multipart uploads, both at the API level and with a MultipartUtils tool that makes it very easy to upload files in multiple parts.
• Support for Amazon S3′s website configuration, which makes an S3 bucket act more like a traditional website. I’m using this new feature to great effect on JetS3t’s new home domain www.jets3t.org.
  The new domain is served from S3 like the old jets3t.s3.amazonaws.com version, but it works much better if you visit places like the root directory (versus this) or a missing page (versus that).
  Now the new domain just needs some Google-juice, so please update your links to point to www.jets3t.org.
• Massive improvements to the Synchronize application to reduce its memory footprint when syncing large directory hierarchies and improve its efficiency when comparing local and remote files.
  Synchronize also now supports multipart uploads, so you can back up files larger than 5GB and improve reliability by uploading large files in smaller pieces (see the upload.max-part-size configuration setting in synchronize.properties).
• Support for custom (non-S3) distribution origins in the CloudFront API. Note that these service changes are not backwards-compatible
• A number of bug fixes and other tweaks

See the full list of changes in the Release History or Release Notes documents.

Yes Please

Visit the JetS3t web site to download the latest packaged release, view the latest code samples or read the API Javadoc.

Or go to the BitBucket developer site to access the latest code, report issues in the bug tracker, and contribute to the project.

P.S. The latest release is on its way to the official Maven2 repository and should be available within a day or so.

With a Website Hosting configuration applied to an S3 bucket, the bucket can serve static content but will also act in a somewhat dynamic way to serve index and error documents if someone visits URL paths that don’t match a real file.

This makes it much more feasible to serve static website content from S3 without having to worry about users receiving strange XML error messages if they venture off the beaten track or try to access partial URL paths. In particular, it allows you serve an index.html file from the root of a bucket, just like a real web server.

To find out more read these:

• Host Your Static Website on Amazon S3
• New AWS feature: Run your website from Amazon S3
• Developer Guide: Hosting Websites on Amazon S3

To try out the feature in JetS3t, grab the latest development code and read the example test code to see how it works.

I was trying to install the nose-cov unit test coverage package, which depends on coverage, using the standard pip/easy_install commands without much luck. The compilation phase spat out a stream of error lines followed by the final message:
failed with error code 1.

Mac OS X 10.4 Support

The first issue was that the coverage package required the optional “Mac OS X 10.4 Support” component of XCode, as hinted at by this message I found in the middle of the error log:

I did not have the 10.4 components installed on my 10.6 machine despite having the rest of the (presumably 10.5+) components. This is easy to fix by re-running the XCode installation from your OS X disk and choosing the extra option.

GCC Version 4.0 FTW

After installing the 10.4 pieces the installation still failed, this time with the following message near the end of the error log:

[Edit: 2011-02-25]

A much cleaner way to make your system use GCC version 4.0 instead of a later version is to use the CC environment variable. Prefix your pip/easy_install command with CC=/path/to/desired/gcc-version like so:

Try the CC environment variable approach above before you attempt the hack below.

Google lead me to this post which provided the solution, though it feels like a nasty hack.

My version of OS X (10.6.6) includes two versions of the gcc compiler executable in /usr/bin:

The gcc symlink points to gcc-4.2 but only the gcc-4.0 version successfully compiles the package. I temporarily moved the original gcc symlink out of the way and created a new one pointing to the 4.0 version, like so:

I was then able to install the nose-cov package using pip and it ran just fine. Great!

Finally I replaced the original symlink, because forgetting to do so would almost certainly cause something else to break sooner or later:

So in the end my /usr/bin directory looks exactly the same as it used to and the package is installed. Time to go do some real work.

Real Google Storage support

Amongst many other things, this release is the first to include native support for Google Storage for Developers (GS). This means that you can now interact with GS using its own API instead of relying on its (mostly) S3-compatible API. Take a look at the GS sample code.

I would like to thank the Google developers who helped make this possible.

Major changes — Take care

This release includes more changes to the underlying JetS3t codebase than I’ve made in a long time, hence the move to version 0.8.x. A broad refactoring was necessary to add the brand new GoogleStorageService, and it’s related Google-specific features, in a relatively sane way.

The advantages of all these changes are:

• JetS3t now has generic storage service, bucket, object, and multi-threading classes to simplify working with different providers, Amazon S3 or Google Storage
• We have finally dropped support for JDK version 1.4 and moved to Java 6+, so we can take advantage of newer language and library features (generics, at last!)

The disadvantages of the rework are:

• We have definitely broken backwards-compatibility for some folks who use older JDKs. Sorry about that
• Despite our best intentions we may have broken backwards-compatibility in non-obvious places

This release is likely to be more buggy and less stable than you’ve come to expect from JetS3t. Before you upgrade, be sure to read the full RELEASE_NOTES.txt document and do plenty of testing.

And, of course, file bugs if you find any.

Get the code

Visit the JetS3t web site to download the latest release, and to view the latest code samples and API Javadoc.

The latest version should be available from the official Maven2 repository within a few days.

Change Summary

Here is a summary of new features and changes in this release.

Potentially backwards-incompatible changes:

• Wide-ranging code changes made adding support for Google Storage may lead to API incompatibilities
• JetS3t now requires Java 6+, it is no longer compatible with JDK 1.4
• Changed property names:
  • s3service.internal-error-retry-max => storage-service.internal-error-retry-max
  • s3service.stream-retry-buffer-size => uploads.stream-retry-buffer-size
  • s3service.defaultStorageClass => s3service.default-storage-class
• Removed antiquated and unsupported SOAPS3Service
• Methods for setting and using AWS DevPay credentials moved from S3Service to RestS3Service
• Deprecated a range of S3-specific classes, where provider-agnostic implementations are now available

New support for Google Storage provider:

• Native Google Storage service and ACL implementations: GoogleStorageService, GSAccessControlList
• Generic bucket, object, and service classes for interacting with either of the S3 or Google Storage services
• Provider-agnostic multi-threaded services: ThreadedStorageService, SimpleThreadedStorageService
• Almost all utility classes are compatible with both the S3 and Google Storage services

Bug fixes:

• CloudFrontService: URL signing API changed to support HTTPS and RTMP resources, not just HTTP ones

New toolkit functionality:

• RestS3Service: supports for bucket access policies
• CloudFrontService: support for Default Root Object and object invalidation
• FileComparer utility and JetS3t apps now uses a trailing slash to denote directory place-holder objects, instead of the custom “application/x-directory” content-type

Cockpit application updates:

• Log in to either Amazon S3 or Google Storage services via GUI
• Manage a CloudFront distribution’s Default Root Object

Synchronize application updates:

• Fully supports new GoogleStorageService, allowing synchronization with Google Storage via its native API
• Choose target service end-point, Amazon S3 or Google Storage, with the –provider command argument

Bug fixes:

• Shell scripts are more compatible with Cygwin

New toolkit functionality:

• Added support for the new Reduced Redundancy Storage (RRS) class for objects
• CloudFrontService: Support for HTTPS-only distributions and logging of streaming distributions
• Added support for buckets located in the Asia Pacific (Singapore) location “ap-southeast-1″
• Improved compatibility with Eucalyptus/Walrus

Cockpit application updates:

• Reduced Redundancy Storage class
• Buckets located in the Asia Pacific (Singapore) location
• HTTPS-only CloudFront distributions

Synchronize application updates:

• Allow synchronization with third-party buckets that are not owned by the user

Other notes:

• CloudFrontService API changes may break backwards-compatibility
• SOAPService is deprecated and will soon be removed from the toolkit

Visit the JetS3t web site to download the latest release and view the latest code samples and API Javadoc. The latest version should also be available from the official Maven2 repository by now.

Read about the complete list of changes in the release notes, and visit the development site to submit bug reports or help out with the project.

Accept a little more risk, save some bucks. For many S3 customers this will be an attractive option.

See Amazon’s RRS blog post for more information.

Below is a brief overview of how you can use the feature in JetS3t once you have downloaded and built the latest code.

Cockpit

The Cockpit application has support for the new storage class in a few places:

• In the preferences you can choose the default storage class to apply when uploading objects
• the Copy or Move Objects dialog allows you to choose the storage class for destination objects. This makes it easy to apply the cheaper storage option to many objects, you simply copy the objects in-place after selected the REDUCED_REDUNDANCY storage class
• each object’s current storage class is now shown in the Object Attributes dialog.

API

In the API you set the storage class of an object prior to uploading it:

To apply the new storage class to existing objects you call the normal RestS3Service copyObject or moveObject methods after setting the storage class attribute of each destination object.

To check the storage class an object is stored under you perform a bucket listing then call the S3Object’s getStorageClass method: