Engineered Web

Protocol Relative URLs in Drupal 7

Matt — Tue, 27 Dec 2011 16:09:50 +0000

When I first started tackling the problem of https and Drupal caching I wanted an interim solution and started by making the database caching layer smarter. Neil Drumm had the idea to use protocol-relative (a.k.a, schemeless) URLs. This turns out to be a great idea and thanks to some hooks in Drupal it's something we can implement fairly easily.

Protocol-Relative URL

Protocol relative URLs don't contain the protocol. For example, http://example.com/foo/bar would be //example.com/foo/bar. Browsers will use the protocol for the page it's on when the protocol is absent. For a page that could be displayed under https or http this is a nice solution and can help us avoid caching for both situations.

Protocol-Relative URLs aren't new or novel. For example, Google has been using them for some time. Just take a look at the source of a Google Plus page (or many others).

The Setup

You first need to make sure your server is capable of serving both http and https requests. For URLs to not have a protocol it makes sense to serve with multiple protocols. There is also the case where your content could be served out of http on RSS then viewed in an RSS reader under https. You don't want problems to pop up in cases like this.

IE 7/8 and CSS

Yeah, so older versions of IE have a bug. IE 7 and 8 will download stylesheets twice when the protocol is missing. You can either code around this, not use this technique, or let those users suffer with slightly slower performance.

Implementation Options

What follows are two different implementation options depending on what you want to do and your scenario. You can pick the case you think works best for you or tweak to your hearts delight.

Images Only

Many of the images inside Drupal (e.g., those handled by image styles) have a full URLs. If you want to alter these it's possible through the use of hook_preprocess_image().

/**
 * Implementation of hook_preprocess_image().
 *
 * Make images that use a full url be protocol relative.
 */
function custom_preprocess_image(&$variables) {
 
  // If the image URL starts with a protocol remove it and use a
  // relative protocol.
  $scheme = file_uri_scheme($variables['path']);
  $protocols = array('http', 'https');
  if ($scheme && in_array($scheme, $protocols)) {
    $variables['path'] = '//' . file_uri_target($variables['path']);
  }
}

This code is available as a gist at https://gist.github.com/1523736

All File URLs Generated By Drupal

In Drupal file_create_url() is used to create the full URLs for everything from image to CSS and JavaScript. It handles public and private files, internal paths (which it turns into full URLs), and making sure everything is tidy. To accommodate CDNs there is hook_file_url_alter() which we can plug into in order to make protocol-relative URLs.

/**
 * Implements hook_file_url_alter().
 *
 * Make all URLs be protocol relative.
 * Note: protocol relatice URLs will cause IE7/8 to download stylesheets twice.
 */
function custom_file_url_alter(&$url) {
 
  global $base_url;
 
  static $relative_base_url = NULL, $relative_base_length = NULL;
 
  $scheme = file_uri_scheme($url);
 
  // For some things (e.g., images) hook_file_url_alter can be called multiple
  // times. So, we have to be sure not to alter it multiple times. If we already
  // are relative protocol we can just return.
  // Only setup the and parse this stuff once.
  if (!$relative_base_url || !$relative_base_length) {
    $relative_base_url = '//' . file_uri_target($base_url);
    $relative_base_length = strlen($relative_base_url);
  }
  if (!$scheme && substr($url, 0, $relative_base_length) == $relative_base_url) {
    return;
  }
 
  // Handle the case where we have public files with the scheme public:// or
  // the case the relative path doesn't start with a /. Internal relative urls
  // have the base url prepended to them.
  if (!$scheme || $scheme == 'public') {
 
    // Internal Drupal paths.
    if (!$scheme) {
      $path = $url;
    }
    else {
      $wrapper = file_stream_wrapper_get_instance_by_scheme($scheme);
      $path = $wrapper->getDirectoryPath() . '/' . file_uri_target($url);
    }
 
    // Clean up Windows paths.
    $path = str_replace('\\', '/', $path);
 
    $url = $base_url . '/' . $path;
  }
 
  // Convert full URLs to relative protocol.
  $protocols = array('http', 'https');
  $scheme = file_uri_scheme($url);
  if ($scheme && in_array($scheme, $protocols)) {
    $url = '//' . file_uri_target($url);
  }
}

This code is available as a gist at https://gist.github.com/1524135

Be careful with this option as it will cause CSS to be downloaded twice in IE 7 and 8.

Drupal Does Not Respect https:// When Caching

Matt — Mon, 26 Dec 2011 21:20:33 +0000

One of the problems I recently discovered with the Drupal cache is that it doesn't properly handle https transactions when caching happens. Let's take a look at understanding the problem, an interim solution for the database cache, and finding a long term solution.

The Problem

The page cache properly respects https because it uses a full url including the protocol when generating a cache id. But, Drupal uses multiple layers of caching with the html in a page. For example, the content of blocks can be cached. What if the html for a block has absolute URLs pointing back to the site and those are generated on the http version of the site. Then this is cached. Then this cache is used to create the https version of the page. Then those cached absolute URLs are not using https.

Media Module, Where I Found The Problem

The example that caused me to understand this problem was the media module. With the media module you can embed images and video into a text area (like the body of an article). Media tags are converted to html by the filter system and the results are cached. Images, for example, use absolute URLs (this is part of core and is a good thing for some use cases). If the cache for this text was generated on the http version of the site the path to the image on the https version of the page will use http as the protocol.

This opens up all kids of possible issues. Just imagine someone in a coffee shop thinking they are on https only to have cookies sent back to the server for that image in plain text. Or maybe you have a better imagination than me and can think of something more sinister.

Nested Caching

The issues is nested caching of html and how many of the nested caches don't respect the protocol like the page cache does. This can apply to any html that's cached across any modules.

Fixing Database Caching In The Short Term

To work around this problem I've started a database caching layer that works around this problem by respecting https for most caches. I don't really like this solution and there are some definite hacks in how it works. The real solution should be to have the code saving and retrieving from the html caches be smart enough to include the protocol in the caches. Unfortunately, a cache backend can't fix the places the cache is used.

A Long Term Solution

An issue for a long term solution is already open. If you have any experience in this area, it impacts your sites, or you want to help out please head over there so we can fix this moving forward.

DrupalCamp Michigan Registration Re-Opened

Matt — Mon, 17 Oct 2011 11:41:54 +0000

When we first started planning DrupalCamp Michigan we didn't have high expectations for turnout. Southeast Michigan, where the camp is being held, is not known for a large or active Drupal community. When we sold out less than 3 weeks after we had the idea for the camp it was a very present surprise. But, we knew there were more people who wanted to come and an opportunity to reach more people. So, we've added more space and re-opened registration.

Special thanks to Jason Savino for jumping all over this, coordinating with others in the local community (especially Mike O'Connor from the Commerce Guys who was also working on this), and making the extra space happen quickly.

The First DrupalCamp Michigan

Matt — Mon, 03 Oct 2011 12:37:28 +0000

This December 3rd will be the very first DrupalCamp Michigan. Our initial go at a DrupalCamp will be a one day event at the Crowne Plaza in Novi. The event is being sponsored and attended by the local Drupal shops of Commerce Guys, Mustardseed Media, Switchback, and Commercial Progression.

If you're interested in attending head on over to http://drupalcampmi.org for more details and to register. Space is limited.

Our hope is to bring out many of the local Drupalers so we can better get to know each other. While there are a number of prominent Drupal developers in the area and some prominent shops, the community has yet to extensively connect with each other. We hope this DrupalCamp will be a great opportunity to build connections in the local community.

If you're interested in connecting with other Michigan based Drupal developer in IRC, head over to #drupal-michigan on freenode.

Michiganders, it's time to connect. If you're a local Drupal developer come checkout the first DrupalCamp Michigan.

The Drupal Community and Front End Performance

Matt — Tue, 20 Sep 2011 14:10:32 +0000

Drupal has a presence problem when it comes to front end performance. Drupal has for the most part ignored front end performance. According to a study by Strangeloop, 97% of the time it takes a mobile page to render is in the front end. For desktop browser the front end makes up 85% of the time. These numbers may feel high. But, when pages take 500ms to render in Drupal but 6 seconds to display in an end users browser you can see where this comes from.

The presence problem for Drupal can be seen in several places:

A the past few DrupalCons how many sessions have touched on front end performance? I can only recall one of them while there have been many covering memcache, apc, and other server side technologies.
Take a look at the documentation pages on profiling Drupal. Or, search for documentation pages on performance. You'll find discussions about apache benchmark, learn about varnish, etc. You won't learn about font end performance.
Drupal doesn't provide minified JavaScript. For production environments this is considered a standard practice.
The Drupal 8 development "gates" RFC gives 1 of 6 performance items to font end performance. The other 5 are tips/gates in detail for back end issues we've commonly run into. The front end one is a basic one liner.

Front end performance is a big deal. This is more so true as we enter into the dominance of mobile where mobile devices are low powered and on high latency networks.

What We Can Do About It

Pointing out problems is no good without solutions. The problem is in the amount of face time front end performance gets. So, lets get it some face time.

At DrupalCamps let's start presenting on it.
Drupal companies could benefit from having someone knowledge in house. Come up with ways to add it to your expertise. Maybe hold a book club and discuss a Steve Souders book.
When we learn about useful tools like ImageOptim or Sprite Cow lets share them.
If you see a contrib module serving JavaScript up that has not been minified file a patch. You can use UglifyJS easily through the web. UglifyJS is what jQuery uses.

Front end performance is a big deal. It's the largest part of the performance equation an end user experiences. Companies have done studies showing the financial and usage impact of end user performance. Let's elevate front end performance to the place it needs to be in the Drupal community.

PHP Needs A New Package Manager

Matt — Tue, 13 Sep 2011 14:14:28 +0000

After working with Pear, the PHP package manager, recently one thing has become apparent. PHP needs a new package manager. Over the last couple years I've worked with npm, homebrew, and gem which are great package managers for other languages and systems. Using Pear requires extra thought and work as a user, pain for creating packages, and suffering for those handling deployment outside a standard workflow. So, let's start a conversation about what would be better.

The Problem With Pear

Pear has been around since 1999. The web was an entirely different place. This is back in the day when dial-up internet was common place. After all this time and change in the developer community did Pear become antiquated?

From my perspective I have 3 problems and see dozens of places for small potential improvements.

Decentralization

Pear was also built on decentralization. That means something like:

$ pear install symfony

Pear doesn't know what symfony is or where to get it. Before you can install it you need to know where to get it from and add the channel. For example:

$ pear channel-discover pear.symfony-project.com

If there were just a handful of sites providing packages this wouldn't be an issue. If you use packages from more than a handful of pear channels you could end up needing a manager just for your pear channels much less your PHP packages.

Don't get me wrong, decentralization is nice in theory. But, in practice it's lowering my user experience and is a pain point.

The Pain Of Creating Packages

I don't update my projects often in part because of the pain in creating updates to my pear channel.

While I was lamenting over Pear recently a developer shared this with me. That's when I realized there were others experiencing pain in creating packages. The few times I considered creating a pear channel and packages I started diving into the documentation and walked away after a short bit of screaming. To hear someone who had gone through with creating packages complain about it was new.

Sure, there are tools like Pirum that make it easier. But, creating pear packages is not something many people want to do.

What About Firewalls and Intranets?

Some companies and projects have firewall and packaging restrictions. Sure these can drive developers nuts sometimes but there are often (though not always) good reasons behind them. How do you handle this with Pear?

Well, you could create your own Pear channel and put all the packages you want in there. If you look at the package.xml that comes with each package you'll notice it calls out information like the pear channel it's associated with. So, you need to update each of the packages before moving them to your internal channel?

Let's just say this setup is not one I personally want to work in.

A Picture Of What People (Mostly Myself) Want

Putting the justification for why I'm writing this all to long post aside lets focus on what would be great in a PHP package manager. To put it in a single word, simplicity.

Can you imagine it being as simple for a user as:

$ foo install bar
$ foo info bar
$ foo self-update
$ foo uninstall bar
$ foo pay bar tab

As an end user I want something simple and easy to use. No channel discovering (yes, I want a central repository). If it's going to pull from a corporate repository that could be a command line switch or in a system wide config file.

From a developers standpoint simplicity should come first. Imagine the case where creating a new package worked like this.

$ cd /to/package/root
$ foo create bar

At this point a utility asked you a bunch of questions about the package and built the package.xml file for you. Then to publish it you do something like:

$ foo publish bar

This is the type of tool I would want to use. It might even help PHP in the popularity arena.

A Short List Of Requirements

Now that I've painted a picture of what it should look like here's a short list of requirements that should be covered.

The ability to create my own distribution site (like the central repo) and easily move packages into it. This will help those secure network types.
The ability to import packages into an app. For example:
```
\Foo\Require('bar');
```
This may even include a specific version, for example.
```
\Foo\Require('bar', '1.1.1', '>');
```
The ability to get apps as well as install them on the system. For example:
```
$ foo get wordpress
```
This would go get and download the latest release of wordpress and put it in the directory I'm currently in. Not everything is a package to be installed on a system.

What Do You Think?

I would find a tool like this to be incredibly useful and I've talked to a handful of others who feel the same way. But, nothing has been created in the years and years of PHP so maybe I'm off my rocker. What do you think? Is something like this something you'd want to use? What additional requirements would you like to see? Or, is there something out there that already does all of this that just isn't getting the attention it deserves?

SplFixedArray, An Underutilized PHP Gem

Matt — Thu, 08 Sep 2011 13:55:45 +0000

Arrays in PHP are not arrays per the typical array data type. Instead, as Matt Butcher recently pointed out arrays in PHP are similar to hashes in other languages. This can be a very important point to know when tracking down bugs in code and to programmers coming to PHP from other languages. But, what if we wanted something like a traditional array data type? Maybe something that preserved numeric order. Enter SplFixedArray.

An Example Problem

A simple example of a problem PHP arrays don't handle well is ordering by number. Let's look at a simple example.

$foo = array();
$foo[0] = 'a';
$foo[1] = 'b';
$foo[2] = 'c';
 
// Remove and replace index 1 with something new.
unset($foo[1]);
$foo[1] = 'what?';
 
print implode($foo); // prints "acwhat?"

As you can see numeric order on an array is not preserved. What if we wanted to use something that preserved numeric order? Enter SplFixedArray.

// SplFixedArray is fixed in size but it can be altered. 3 is initial size.
$foo = new SplFixedArray(3);
 
$foo[0] = 'a';
$foo[1] = 'b';
$foo[2] = 'c';
 
// Remove and replace index 1 with something new.
unset($foo[1]);
$foo[1] = 'what?';
 
print implode($foo->toArray()); // prints "awhat?c"

The numeric ordering of elements is preserved. SplFixedArray has the ability to be generated from an array and to have an array be generated from it allowing for interoperability.

The Performance and Memory Impact

Something I didn't expect was for SplFixedArray to be faster and use less memory than an array.

Memory usage is does not scale the same for these two. An array with less than 6 items in it will use less or about equal memory than a comparable SplFixedArray object. Once you pass about 6 items SplFixedArray uses less memory. The memory growth rate SplFixedArray is much lower and you can quickly get to the case in large sets where it uses half the memory. This is of course dependent on what's being stored in the array.

Also note, when you use calls like:

$bar = $foo->toArray();

This case has both an array ($bar) and an object ($foo) in memory.

SplFixedArray is faster at getting and setting information. In some basic testing I found it to be better than 25% faster.

Some Catches

There is a catch with SplFixedArray. It's an object not a primitive data type in PHP. Array specific functions like implode() don't work on SplFixedArray. SplFixedArray does implement the Iterator, ArrayAccess, and Countable interfaces making it useful in a lot of situations. But, it's no drop in replacement. Rather SplFixedArray is another useful tool in our quest for better development.

PHP 5.3 Performance: __toString() vs. A Method

Matt — Wed, 20 Jul 2011 13:45:51 +0000

Ever since Larry Garfield wrote about PHP magic calls and their slow performance I've been hesitant to use them. In the time since his post was written in 2007, magic methods have seen improvements and I've started taking a look at them again. The first one I've started to explore using in the __toString() method on objects. It turns out, as of PHP 5.3 using the __toString() magic method can be faster than using regular method calls.

Understanding `__toString()`

When an object is treated as a string the __toString() method is called. It gives a class the ability to decide how it will react. For example:

/**
 * A class named foo with a magic __toString method.
 */
class Foo {
  public function __toString() {
    return 'foo';
  }
}
 
// $foo is an instance of class Foo.
$foo = new Foo();
 
// $foo is a class being treated like a string. When it is treated like a
// string the __toString method is called.
// Outputs: foo
print $foo;

Testing `__toString()` vs a `render()` method

In versions of PHP prior to PHP 5.3 there have been tests showing magic methods can be slow. I was warned not to use them for anything that requires performance, like working on a framework or CMS where there will be a lot of users. Recently I had heard magic method performance had improved so I decided to test some of them out and see what happens.

The setup

The code I used to perform the tests is:

<?php
 
class Foo {
  public function render() {
    return '';
  }
  public function __toString() {
    return '';
  }
}
 
$foo = new Foo();
 
$iterations = 1000000;
 
$start1 = microtime(TRUE);
for ($i = 0; $i < $iterations; $i++) {
  print $foo->render();
}
$end1 = microtime(TRUE);
 
$start2 = microtime(TRUE);
for ($i = 0; $i < $iterations; $i++) {
  print $foo;
}
$end2 = microtime(TRUE);
 
printf("Render Time: %0.2d\n", ($end1 - $start1) * 1000);
printf("toString Time: %0.2d\n", ($end2 - $start2) * 1000);

This fairly simple code tries to be as minimal as possible to highlight the time to call these methods.

The results

What we care about is the results and how they compare to each other, not so much the amount of time they actually take.

Render Time: 273
toString Time: 253

Using __toString() is faster than calling a method on the object to turn it into a string. Here we have a magic method performing quite nicely.

JavaScript Theme Functions in Drupal

Matt — Mon, 16 May 2011 11:06:28 +0000

Drupal has an extensive theming system which includes theme functions that can be overridden and an extensible template system. There is an expectation that all markup go through the theme system and it is considered a bug when that doesn't happen. Most of the talk surrounding this system focuses on the server side system and what we can do in PHP. But, that's not all there is to the theme system. Drupal provides a theme system for JavaScript as well. One with callbacks that can be overridden by themes, just like on the PHP side.

Theme Functions

The central function to the theme system is Drupal.theme(). It is the JavaScript counterpart to theme() and works in a similar manner to what theme() did in Drupal 6.

The first step is to define a JavaScript theme function within a modules JavaScript file, then Drupal.theme() can use it. To define a theme function simply create a new function in the Drupal theme prototype namespace like:

Dupal.theme.prototype.displayName = function(name, url) {
  return '<a href="' + url + '">' + name + '</a>';
}

Once this is done the function can be used like:

var name = "John Doe";
var url = "http://example.com";
var display = Drupal.theme('displayName', name, url);

Just like the Drupal 6 theme() function usage, the arguments minus the first one are passed to the theme function itself. The first argument is the name of the function in the Drupal.theme.prototype namespace.

Overriding Theme Functions in Themes

Not only does this system allow us to reuse theme functions as templates, it provides a mechanism for themes to override these theme functions. For example, a module defines a theme function like the one above and uses it. But, a theme wants to change the markup. The theme can define an override function like:

Dupal.theme.displayName = function(name, url) {
  return '<a href="' + url + '"><em>' + name + '</em></a>';
}

This theme function has a slightly different name. Drupal.theme is the namespace instead of Drupal.theme.prototype. The function name is the same as is the function argument signature.

When Drupal.theme() sees this override function it uses it instead of the one defined by the module.

What Should Go Through Theme Functions?

The obvious answer is that all markup should go through theme functions. But, there is more to front end presentation than CSS and markup. For example, animations. It's not uncommon for JavaScript to cause something to display. What if a theme wanted that to slide in rather than just show up in the page? This is not something you can accomplish via markup and CSS changes.

So, instead of thinking of these theme functions as a place for markup consider putting from end presentation in them. All of it. Drupal JavaScript theme functions are something we need to use more. It is the system we currently have and one we should embrace.

For more information on the topic please see the Drupal.org handbook page.

Translatable Strings In Drupal JavaScript

Matt — Mon, 02 May 2011 02:01:50 +0000

Drupal is known for being highly translatable. The localization team has done a fantastic job building out tools to make Drupal translatable and it's rare to find a module that doesn't properly wrap its strings in the t() function or the lesser known format_plural() function. While the Drupal community is doing fairly well on the PHP side, many people don't know there is a translation system for Drupal JavaScript as well through the use of the Drupal.t() and Drupal.formatPlural() functions.

Drupal.t()

Drupal.t() is a counterpart to the t() function in PHP. Unfortunately there is no great API documentation site for the JavaScript included in Drupal. Luckily, the usage is almost identical to what we see in PHP.

For example, to make a simple string translatable you could do something like:

var foo = Drupal.t('A translatable string.');

Passing replacements into Drupal.t() works just like the t() function. The replacement keys act based on the first character of the key.

Variables that start with ! will be passed in as is.
When a variable begins with @ it is passed through Drupal.checkPlain(), the JavaScript counterpart to check_plain().
Variables that start with % will be escaped and passed trough the placeholder JavaScript theme function.

For example:

var args = {};
args['!url'] = 'http://example.com';
args['%name'] = 'John Doe';
 
var foo = Drupal.t('%name is the owner of !url', args);

Drupal.formatPlural()

Formatting plural strings is not something we often think about when it comes to translations. But, plural forms is part of translations and something we need to consider. The JavaScript counterpart to the PHP format_plural() function is Drupal.formatPlural(). The API for using the function is the same. For example:

var bar = Drupal.formatPlural(count, '@name has 1 site.', '@name has @count sites.', {'@name': personName});

Drupal has the tools to make our JavaScript translatable. Lets put them to good use. For more details see the Drupal Handbook on translations.