Christopher Cato - my old blog posts on tech

Translating the Site Name in Drupal 7

2014-06-11T03:21:00.001-07:00

Allmost all parts of Drupal are easily translatable but a select few are hidden. The Site name and Slogan are a couple of those.

To translate a Drupal 7 Site name you have to activate 'Variable translation' module (which is in the same package as i18n) and the required ones (like 'Variable store').

After you install these modules, go to Configuration -> Multilingual settings -> Variables tab and choose 'Site name' (?q=admin/config/regional/i18n/variable). Then go to Configuration -> System -> Site information (?q=admin/config/system/site-information) and select the language you. Change the 'Site name' field and save.

You'll find the translated site name in 'variable_store' table.

Script ads and their inherent security flaws

2014-06-06T05:25:00.000-07:00

Ad networks have built a reputation as spammers and spyware-peddlers constantly busy shuffling the latest hot banners onto your screens. But lurking amongst those banners is a big security hole that can fill your site with bad content, steal data or even use your visitors as bots.
But, how is that possible one might ask. Well, if you're at all familiar with the anatomy of ad scripts you will see that they do a couple of things upon loading. First and foremost they display ads. Ads like in flash, images or html code. Second, they register impressions and clicks for the ads. None of this is dangerous in itself allthough Flash is inherently insecure due to the way Flash is allowed to interact with your browser and the underlying OS.

So, what's the danger in this? Well, let's talk about a common scenario. You run site X. It is a well-known site that gets say 30K unique visitors per day. Most of these visitors don't use AdBlock and all of them except a few ~~IT professionals~~ foilhats don't run NoScript. Your site displays ads from one of the ad networks by inserting their provided script tags in your site.
Your site is secure, you think. Not true! Your site is as secure as the remote scripts you load.

Once a hacker gains access to the ad networks server either by a dns exploit, host spoofing or social engineering he replaces the ad with his own code, making sure the ad still functions while adding another script to the mix. Since you included this in your own code, your site is now open to several exploits. Here are a few possible scenarios.

DOM Hijacking
Once a visitor has loaded the ad scripts and the payload is in effect, the hacker can now display any content in the browser by replacing your DOM elements. It could be subtle manipulations or just vandalism. He is in control of the DOM the script is included in. This is mitigated in part by using iframes.

Cross Site Scripting (XSS)
If you use a web-based content system chances are you will be displaying the ad while logged in to your site. With a little homework, the attacker knows which system you are using and can therefore load any url or post any form that requires login credentials. You are logged in so the payload will be able to access everything you can. This is also mitigated by using iframes.

Distributed Denial Of Service (DDOS)
Since almost all of your visitors display the ad, it's a small feat for the attacker to craft a script that will use javascripts AJAX to enlist your 30K visitors in a botnet. By forcing each visitor to repeatedly load a resource from a target site he will in effect be delivering a huge amount of traffic that will bring the target down.

Consider this simple payload that will overload a server until the client closes the session:

while(somecondition){
var x = document.createElement('img');
x.src = 'http://xxx.xxx/img.jpg?' + Math.floor(Math.random() * 10000000);
document.getElementsByTagName('body')[0].appendChild(x);
}

CPU cycle theft
There are already several available javascript bitcoin miners that could be used in such an exploit. Since there is no real CPU or memory limits on iframes or scripts they could theoretically be quite successful. Especially if they manage to somehow enlist the GPU. Maybe hash calculations could be funneled through WebGL?

I can see several other scenarios where an attacker could utilize your ad network to deliver exploit code.
So the real question is how secure the networks are in reality.

Prevention
DOM/XSS by using iframes. DDOS and CPU theft might not be possible other than by monitoring a client view of your website and ensuring that resource usage and network traffic is normal.
CSP (Content Security Policy) might solve the problems if it gets enough traction with the browser devs http://content-security-policy.com/

Fabric fabfile for Varnish deploy and ban

2014-06-06T04:30:00.000-07:00

This is the file I use for deploying new vcls and purging stuff from my caches. I've got load balancing with haproxy set up in front of three caches so deploy or purge needs to be done on all three caches pretty much at the same time. I've set up ssh keys on the hosts so I won't have to stuff the fabfile with passwords and such.

For more on my load balancing proxy set up, see this post.

The fabfile is kept in the same directory as I keep the default.vcl for the hosts.

To purge an url: fab purgeUrl:<urlpattern>
To purge a host: fab purgeHost:<hostnamepattern>
To deploy default.vcl: fab deploy

from __future__ import with_statement

import os.path
import time
from fabric.api import *
from fabric.contrib.project import *

"""
Environments
"""

def dev():
env.hosts = ['host1','host2','host3']
env.user = 'root'
env.path = '/etc/varnish'

"Default to 'dev' environment"
dev()

"""
Tasks - Deployment
"""

def deploy():
require('path', provided_by=[dev])
with cd(env.path):
put('default.vcl','default.vcl')
programname = str(time.time())
run('varnishadm -Tlocalhost:6082 -S/etc/varnish/secret vcl.load ' + programname + ' /etc/varnish/default.vcl')
run('varnishadm -Tlocalhost:6082 -S/etc/varnish/secret vcl.use ' + programname)

def purgeHost(myHost):
require('path', provided_by=[dev])
with cd(env.path):
run('varnishadm -Tlocalhost:6082 -S/etc/varnish/secret ban req.http.host == '+ myHost)

def purgeUrl(myUrl):
require('path', provided_by=[dev])
with cd(env.path):
run('varnishadm -Tlocalhost:6082 -S/etc/varnish/secret ban "req.url ~ '+ myUrl +'"')

Bulding a custom token for url aliases (or anything else)

2014-06-06T04:07:00.000-07:00

Tokens are a great resource in Drupal when it comes to creating url aliases, replacing text in emails, creating Rules actions or pretty much anything when it comes to automation.

Here's how to create a custom token for nodes that will perform an action on a field from the node and return it as a replacement.

We start out by informing Drupal that we provide a token applicable to nodes. You can replace node with any entity bundle name to provide tokens for them.

/**
* Implements hook_token_info().
* Provides Drupal with a list of our tokens to present in the UI
*/

function example_token_info() {
$info = array();
// Define a new token for nodes.
// "node" can be replaced with other entity names
$info['tokens']['node']['reverse'] = array(
'name' => t('Reverse category term name'),
'description' => t('Outputs a reversed term name.'),
);
// Any other tokens follow here
return $info;
}

We continue by implementing the hook_tokens method to provide methods for performing the token replacements.

/**
* Implements hook_tokens() .
* Takes care of the actual replacement of the token
*/
function example_tokens($type, $tokens, array $data = array(), array $options = array()) {
$replacements = array();

if ($type == 'node') {
// Loop through the available tokens.
foreach ($tokens as $name => $original) {
// Find our custom tokens by name.
switch ($name) {
case 'reverse':
// Load field from node.
$fielditems = field_get_items('node',$data['node'],'field_category');
if ($fielditems[0]['value']) {
// Load the term.
$term = taxonomy_term_load($fielditems[0]['value']);
// Replace placeholder with reversed term name.
$replacements[$original] = strrev($term->name);
}
else {
// No category assigned, replace with ''.
$replacements[$original] = '';
}
break;
}
}
}
// All done.
return $replacements;

}

That's it. You are now replacing tokens like nobody's business.

Why Display Suite is a bad idea

2014-06-05T05:19:00.001-07:00

And why Code Fields are an even worse idea

You start out with three content types and a couple of view modes and everything is hunky-dory. Then the customer has some specific requirements and you say to your self – "Hey, that's kind of a weird thing to do but what the heck, I'll just do a code field and get it done!"
Of course, this continues throughout the project and you keep adding view modes and code fields to fulfil the reqs because once you've started down the DS + code fields road there's just no stopping it.

A year later, the Customer comes back with some new requirements. The site is going to be relaunched with multiple languages. And you're like – "Fuck yeah! i18n FTW!". Until you realize that all those sweet code fields are all like UND UND UND UND and you really should have written those umpteen gazillion tpls instead.

So, Display Suite? No.
Code Fields? No.

Apache Solr Search and FacetAPI translation

2014-06-04T12:43:00.003-07:00

To be able to translate the strings "Displaying: 1-10 of 20" and the other parts of the current search blocks you need to install facetapi_i18n. After installing it you need to visit your config for the current search block and hit Save. Then go to your facet configuration and hit Save. Last but not least you need to refresh the strings for translation.

Oh, and btw, the text "Displaying..." will be triple encoded and output escaped html if you don't patch facetapi_i18n with this patch referenced in this issue https://drupal.org/node/1741444

Drupal and Ajax, now with language negotiation...

2014-06-03T14:25:00.002-07:00

Sometimes I just want to kick myself! I was struggling to set the language for a module that gets called via Ajax and that prints out the contents of a view (with supposedly the correct field language set) but the field language always went for default language...

Well, turns out, sometimes it's Just Not That Hard.

Set up the language negotiation to use prefixes and then just prefix the url with the desired language like /lang/arg

Menu hook from the module:

function example_menu(){
$items = array();
$items['example/ajaxview'] = array(
'title' => 'Ajax View Loader',
'page callback' => 'example_loadview',
'access arguments' => array('access content'),
'type' => MENU_CALLBACK
);
$items['sv/example/ajaxview'] = array(
'title' => 'Ajax View Loader',
'page callback' => 'example_loadview',
'access arguments' => array('access content'),
'type' => MENU_CALLBACK
);

$items['en/example/ajaxview'] = array(
'title' => 'Ajax View Loader',
'page callback' => 'example_loadview',
'access arguments' => array('access content'),
'type' => MENU_CALLBACK
);

$items['de/example/ajaxview'] = array(
'title' => 'Ajax View Loader',
'page callback' => 'example_loadview',
'access arguments' => array('access content'),
'type' => MENU_CALLBACK
);

return $items;

}

The callback:

function example_loadview($arg){

// Print the output from a view that takes a contextual argument

print views_embed_view('myajaxview', 'my_display_id, $arg);

exit;

}

Calling the function from javascript (Drupal keeps a note of the path prefix used in Drupal.settings.pathPrefix)

var url = "/" + Drupal.settings.pathPrefix + "example/ajaxview/" + arg;

$.get(url, function (data) {

alert(data); // Or do something with it
});

The language negotiation will do the rest of the heavy lifting.

String just won't translate?

2014-06-03T09:36:00.000-07:00

... make sure you're not overwriting it with your fancy-pants javascript functions... :)

Getting the path of a translated node

2014-06-03T09:33:00.000-07:00

for those pesky hardcoded links in tpls

global $language;
$translations = translation_path_get_translations("node/42");
/*
Returns the paths of all translations of a node, based on its Drupal path:
array(2) {
["de"]=>string(7) "node/42"
["en"]=>string(7) "node/43"
}
*/
print l(t('Link Title'), $translations[$language->language]); //l() will return the alias to node/42

Adding new node operations for the content overview screen

2014-05-28T06:09:00.001-07:00

Extending the list of operations available for batch processing on /admin/content is quite easy. This goes in a module called example.module

function example_node_operations() {
$operations = array(
'example_magic_operation_1' => array(
'label' => t('Works magic on your nodes'),
'callback' => 'example_operation',
'callback arguments' => array('bulkupdate', array('message' => TRUE)),
),
'example_magic_operation_2' => array(
'label' => t('Works even more magic on your nodes'),
'callback' => 'example_operation_2',
'callback arguments' => array('bulkupdate', array('message' => TRUE)),
),
);
return $operations;
}

// This function gets an array of nids from what was selected
// on the /admin/content screen
function example_operation(array $nids, $op, array $options = array()) {
foreach ($nids as $nid) {
// Work your magic here
}
}

Drupal 7 translation gotchas

2014-05-21T02:58:00.001-07:00

Don't slip.

Entity Reference fields have no clue about languages. To them, everything is Language Neutral. Don't expect them to display the referenced entitys translation. Safest bet is to either translate the field or do field translation on the referenced entitys.

Content loaded (from Drupal) via Ajax will not work with the language system. You need to either pass the language code as an argument or set it globally at some other point in your process.
I've done two fixes on this, one being that I pass the nodeid as an argument and check it for language. Another is in the preprocess_node function in template.php adding the following:
drupal_add_js('jQuery(document).ready(function () { window.nodeid="'.$vars['node']->nid.'";window.lang="'.$vars['node']->language.'" });', 'inline');

Taxonomy Menu will beat you with a stick everytime you edit any term. You need to regenerate the menu or else your listings that are based on the vocabulary you're using for the taxonomy menu will fail miserably.

Don't forget to update any Display Suite Code Fields you've used as you've probably left enough UND in there to sound like a german techno parade. And this is the time when you kick yourself and wonder why you ever went down the code field route in the first place. It's a bad bad idea that starts with that one odd field that just needs some special treatment since the client requires it. Then you end up with 30 code fields with hardcoded UND UND UND UND...

Using Drupal.t()? Don't forget that Drupal can't build it's string index for this function unless everything you're using the function in gets added as files with drupal_add_js()

One basic gotcha that I ran across is that t() expects english as the input language regardless of you sites source language. Which is kind of backwards since I'm Swedish and my source language and default language is Swedish. Well well.

Here are some links I've found useful:

HAProxy and Keepalived on Debian Squeeze for failover and loadbalancing

2012-09-28T06:16:00.000-07:00

Building a failover load balancing cluster on four machines with HAProxy and Keepalived in Debian Squeeze

So you've got a big-ass VMWare machine with some servers to spare? Lets put them to work creating that redundancy your boss always nags you about whenever there is a split-second of downtime. In this post, I'll dive into how you can build a basic load-balancing high availability cluster either on VMWare or with separate bare-metal servers.

I've made a few assumptions about network topography and services used based on my own server environment and the services I work with which are mainly Drupal servers and Varnish cache servers. I usually run my Drupal backends on their own servers fronted by a Varnish server on its own box. I run Debian 5 (Lenny) and Debian 6 (Squeeze). Other than that, it's all pretty basic stuff.

What I'll build is a solution that will project one server to the outside but on the inside it will consist of four servers, in essence a small high-availability cluster. The cluster can be extended infinitely in all levels should the need arise.

High availability cluster

The pros of this kind of setup are redundancy, load balancing, ease of maintenance and the possibility to do proxying for all kinds of tcp connections, not just http but for basically any service over TCP.

The cons are that it can take quite a bit of debugging if anything starts behaving funky and, obviously, it requires a few spare servers and IPs.

Outline

There are a few steps we need to go through to get this up and running:

Install HAProxy and Keepalived
Configure HAProxy and Keepalived
Configure sysctl
Configure cache servers/backends
Start services
Verify that the system is up
Verify failover function

The tools

To provide redundancy I'll be using Keepalived, a simple and robust linux routing software written in C that provides failover functionality via the virtual routing protocol VRRP
Keepalived also provides layer 4 load balancing. Keepalived is responsible for maintaining the shared public IP and determining which server is alive.

To provide true layer 7 load balancing I will be using HAProxy. HAProxy is a fast, free and reliable TCP load balancing, proxying and high availability software that provides us with the parts needed to finish our cluster. HAProxy determines the health of the backends - removing any one that fails - and distributes the load between them. HAProxy also provides sticky sessions through cookies that pin each visitor to it's own backend.
HAProxy is also light on resources, easily handling thousands of connections on cheap hardware

To make it all happen, I'm using four servers. Two will be serving as load balancers and two as cache servers. The load balancers run HAProxy and Keepalived and the cache servers run Varnish and Apache (which you could replace with Nginx or whatever). The reason the cache servers need to run Apache as well as Varnish is because they need to be able to serve a file on their default IP and Varnish doesn't handle that (yet). Of course, many are running backend and cache server on the same box so in that case, this is not a problem.

The idea of it all is to have the load balancers sharing one IP in a master/backup setup. Since they share one IP there will only be one server visible to the outside at any one time. Should one of them fail, the other instantly picks up the IP and resumes business. For this to work, we need to use the Keepalived daemon.

For the load balancing part, we use HAProxy which works in tandem with Keepalived to assure that the backup server takes over if HAProxy should fail. HAProxy provides another layer of failover by monitoring our cache servers and removing them if they fail to respond.

The servers are:

proxy1 haproxy, keepalived, ip 11.22.33.42 (sharing ip 11.22.33.44)

proxy2 haproxy, keepalived, ip 11.22.33.43 (sharing ip 11.22.33.44)

cache1 varnish, apache2, ip 11.22.33.45

cache2 varnish, apache2, ip 11.22.33.46

Install HAProxy and Keepalived

I won't cover installation of web servers and caching servers.

For the load balancers you need to install HAProxy and Keepalived which are available in the usual deb repositiories.

root@proxy1: # apt-get install haproxy keepalived

root@proxy2: # apt-get install haproxy keepalived

On Squeeze, this will also install ipvsadm and ask you to run dpkg-reconfigure to enable it but this can safely be ignored since Keepalived will load necessary parts from it.

Configure HAProxy and Keepalived

UPDATE: Adjusted HAProxy config as per input from Willy Tarreau, resulting in a nice extra 1000 req/s throughput increase
This is a very basic setup to get the load balancing and failover running

/etc/keepalived/keepalived.conf on proxy1:

vrrp_script chk_haproxy { # Requires keepalived-1.1.13

script "killall -0 haproxy" # cheaper than pidof

interval 2 # check every 2 seconds

weight 2 # add 2 points of prio if OK

}

vrrp_instance VI_1 {

interface eth0

state MASTER

virtual_router_id 51

priority 101 # 101 on master, 100 on backup

virtual_ipaddress {

11.22.33.44 # supply your own spare public ip

}

track_script {

chk_haproxy

}

/etc/keepalived/keepalived.conf on proxy2:

vrrp_script chk_haproxy { # Requires keepalived-1.1.13

script "killall -0 haproxy" # cheaper than pidof

interval 2 # check every 2 seconds

weight 2 # add 2 points of prio if OK

}

vrrp_instance VI_1 {

interface eth0

state MASTER

virtual_router_id 51

priority 100 # 101 on master, 100 on backup

virtual_ipaddress {

11.22.33.44 # supply your own spare public ip

}

track_script {

chk_haproxy

}

/etc/default/haproxy on proxy1 and proxy2:

# Set ENABLED to 1 if you want the init script to start haproxy.

ENABLED=1

# Add extra flags here.

#EXTRAOPTS="-de -m 16"

/etc/haproxy/haproxy.cfg on proxy1 and proxy2:

global

log 127.0.0.1 local0

log 127.0.0.1 local1 notice

#log loghost local0 info

maxconn 4096

#debug

#quiet

user haproxy

group haproxy

daemon

defaults

log global

mode http

option httplog

option dontlognull

retries 3

option redispatch

maxconn 2000

contimeout 5000

clitimeout 50000

srvtimeout 50000

listen webfarm *:80

mode http

stats enable

stats auth user:pass

balance roundrobin

cookie SERVERID insert # pin visitor to server

option http-server-close # Thanks Willy!

option forwardfor

option httpchk HEAD /check.txt HTTP/1.0

# change IP to your cacheservers public IP

server webA 11.22.33.45:80 cookie A check

# change IP to your cacheservers public IP

server webB 11.22.33.46:80 cookie B check

Configure sysctl

Edit /etc/sysctl.conf on proxy1:

root@proxy1: # echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf

root@proxy1: # echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf

root@proxy1: # sysctl -p

net.ipv4.ip_forward = 1

net.ipv4.ip_nonlocal_bind = 1

root@proxy1: #

Edit /etc/sysctl.conf on proxy2:

root@proxy2: # echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf

root@proxy2: # echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf

root@proxy2: # sysctl -p

net.ipv4.ip_forward = 1

net.ipv4.ip_nonlocal_bind = 1

root@proxy2: #

Configure cache servers / backends

Make sure that your cache servers respond on port 80 of their IP as assigned in your haproxy config on the load balancers.

Make sure they serve a file named check.txt on for example 11.22.33.45:80/check.txt . The contents are not important but since they will be serving it every 2 seconds, just put "check ok" inside it or a simple "1" if you're really anal about optimizations. Which you should be :)

Start services

Start services on proxy1

root@proxy1: # service haproxy start

root@proxy1: # service keepalived start

Start services on proxy2

root@proxy2: # service haproxy start

root@proxy2: # service keepalived start

Verify that the system works

Check your network on proxy1

root@proxy1: # ip addr sh eth0

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:11:22:33:44:55 brd ff:ff:ff:ff:ff:ff
inet 11.22.33.43/27 brd 11.22.33.63 scope global eth0
inet 11.22.33.44/32 scope global eth0
valid_lft forever preferred_lft forever

Check your network on proxy2

root@proxy2: # ip addr sh eth0

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:11:22:33:44:54 brd ff:ff:ff:ff:ff:ff
inet 11.22.33.42/27 brd 11.22.33.63 scope global eth0
valid_lft forever preferred_lft forever

It's working! Proxy1 is now master and has the correct IP assigned as an alias. It assumes the master role by default since it has a higher priority defined in keepalived.conf

To verify that the connection to your backend / cache servers work, you'll have to edit the hosts file on your local machine, adding an entry for any of your backend domains that points to 11.22.33.44. After flushing your DNS cache, you should now be able to browse your site. If it doesn't work, check that haproxy.cfg points to the correct cache/backend and that your backend server is up. Also check that your dns entry was activated.

Verify the failover function

Watch /var/log/messages on proxy2 while stopping the network on proxy1 and you should see something along the lines of this:

root@proxy2: # tail -f /var/log/messages

Sep 28 15:12:28 proxy-02 Keepalived_vrrp: VRRP_Instance(VI_1) Received lower prio advert, forcing new election

Sep 28 15:12:56 proxy-02 Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert

Sep 28 15:12:56 proxy-02 Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE

Sep 28 16:29:45 proxy-02 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE

Sep 28 16:29:46 proxy-02 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE

Check your network

root@proxy2: # ip addr sh eth0

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether 00:11:22:33:44:54 brd ff:ff:ff:ff:ff:ff
inet 11.22.33.42/27 brd 11.22.33.63 scope global eth0
inet 11.22.33.44/32 scope global eth0
valid_lft forever preferred_lft forever

It's working. Proxy2 has now taken over the role of master and activated the IP.

Now watch the messages log again and reenable networking on proxy1

root@proxy2: # tail -f /var/log/messages
Sep 28 16:29:45 proxy-02 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE

Sep 28 16:29:46 proxy-02 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Sep 28 16:29:55 proxy-02 mpt-statusd: detected non-optimal RAID status
Sep 28 16:32:52 proxy-02 Keepalived_vrrp: VRRP_Instance(VI_1) Received lower prio advert, forcing new election
Sep 28 16:32:53 proxy-02 Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert
Sep 28 16:32:53 proxy-02 Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE

Check your network

root@proxy2: # ip addr sh eth0

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether 00:11:22:33:44:54 brd ff:ff:ff:ff:ff:ff
inet 11.22.33.42/27 brd 11.22.33.63 scope global eth0
valid_lft forever preferred_lft forever

It's working. Proxy1 has now taken over the role of master and activated the IP.

Benchmarking

testserver1:~# ab -k -n 10000 -c 1000 http://www.mydomain.com/favicon.ico

This is ApacheBench, Version 2.3 <$Revision: 655654 $>

Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking www.mydomain.com (be patient)

Completed 1000 requests

Completed 2000 requests

Completed 3000 requests

Completed 4000 requests

Completed 5000 requests

Completed 6000 requests

Completed 7000 requests

Completed 8000 requests

Completed 9000 requests

Completed 10000 requests

Finished 10000 requests

Server Software: Apache/2.2.16

Server Hostname: www.mydomain.com

Server Port: 80

Document Path: /favicon.ico

Document Length: 894 bytes

Concurrency Level: 1000
Time taken for tests: 1.474 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 10000
Total transferred: 14118199 bytes
HTML transferred: 8940000 bytes
Requests per second: 6783.93 [#/sec] (mean)
Time per request: 147.407 [ms] (mean)
Time per request: 0.147 [ms] (mean, across all concurrent requests)
Transfer rate: 9353.22 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 5.3 0 30
Processing: 23 79 88.0 29 706
Waiting: 23 79 88.0 29 706
Total: 23 80 88.6 29 706

Percentage of the requests served within a certain time (ms)
50% 29
66% 93
75% 119
80% 127
90% 204
95% 244
98% 266
99% 295
100% 706 (longest request)

Deploying

To put this all to work, all you need to do is to alter your dns records and point them towards your shared proxy IP where you normally would point to your backend / cache server.

Monitoring

To get an overview of HAProxy current statistics, log in to your stats page at 11.22.33.44:1936 as configured in haproxy.cfg.

HAProxy statistics overview

Troubleshooting

Replace all the example IPs with real addresses
Run keepalived with the -d flag
Check your network
Check that the cache/backend servers listen on the IP:s you assigned in haproxy.conf
Verify that you can reach the shared IP from the outside
Tail those logs.. /var/log/daemon.log /var/log/messages
Read the documentation!
Low performance? Check that your virtualization network drivers can handle the load. See this blog post - http://www.networkredux.com/blog/view/1346

Gotchas

If all else fails and that shared IP won't respond to your pings from the outside - check with your hosting admin and see if he has configured your vlan correctly. I spent a few hours debugging everything without result and the day after, our provider checked and lo and behold - the vlan was misconfigured. Click-clickety-clack, ten seconds later everything was working as intented.

Using ACLs in Varnish? Fail much? client.ip will now always be the same as your proxy ip.
Use req.http.x-forward-for and match them line for line instead. Or write a VMOD.
Check this blog post for more info: http://zcentric.com/2012/03/16/varnish-acl-with-x-forwarded-for-header/

Resources

Rendering fields correctly in Drupal 7

2012-03-28T02:29:00.000-07:00

While searching for some field api stuff I stumbled on this rather good introduction to the Field API in Drupal 7 and specifically on how to read field contents in a safe manner.

If you (as I did) read the below excerpt and feel a little guilty then you should most definitely read the full article.

From http://www.computerminds.co.uk/articles/rendering-drupal-7-fields-right-way by Stephen Tweeddale:

You may well have seen (or written!) code that looks something like this:

// This is WRONG example.

$block['content'] = $node->field_name['und'][0]['safe_value'];

Poking around the node object for the value you wanted to print was fairly common in Drupal 6, and the 'safe_value' sounds like it's been sanitised, right? What's wrong with that? Oh, Let me count the ways.

Firstly, the ['und'] element is part of the field localisation in Drupal 7 (see this article from Gábor Hojtsy for more on that); directly accessing that value will cause issues in any kind of multi-lingual environment. Boo.

By accessing the field value directly you miss out on any theming that might come courtesy of the normal field markup.

The [0][safe_value] explicitly accesses the first value of the field - if you wanted every value from a multi-value field you'd need to do some sort of loop.

Some fields (such as node references) won't have a safe_value element, only a value - which can easily be printed without thought for sanitisation. This is dangerous, not because node reference fields contain dangerous data (they're just a nid), but because it's not a helpful habit to get into, especially for new developers. Other fields types 'value' may well be highly dangerous.

Thanks, Stephen. I hearby wow never to repeat my sins against Drupal. Honestly.

Site performance - analyze and improve

2012-02-18T07:32:00.002-08:00

Looking back at the Drupal site I built two years ago for my current employer it's starting to look a little old and worn. Sure, we're running Varnish, we have purge rules, we have memcached for the backend and we've shut off basically all cpu hogs. But what about the visitors side - apart from getting a speedy response from our cache servers, there has to be more we can do? Well - load times, page rendering speed and total data size seem like good targets for improvement since they also provide (via Firebug, YSlow and NewRelic) great metrics.

Record, improve, measure, profit?

We start initially by collecting metrics on the site usage so we can set goals for the improvements. We use a mix-and-match approach to this, utilizing tools like Yahoos YSlow, Pingdoms Pagetest, New Relic and Firebug. Mind you, we had to disable a few rules in YSlow since they're really not applicable for us - the CDN rules (hey, we don't have a CDN), ETags (we have them but they're default Apache - YSlow hates them), Expires headers (yeah we don't want to set them 3 years into the future) and DNS Lookups (Friggin external services are really DNS-expensive)

The pages we're measuring typically show one full article, 3 banners (flash/jpg/whatever) and up to 150 teasers consisting of an image and up to 200 characters text. Insane amount of teasers.

Initial metrics
Total load time: 8sec

Waiting for external services: 3-12 sec

Data size: 2.5mb

DOM Elements: 1377

YSlow score: D (67)

Changes

1) GZIP GZIP GZIP

Apparently, somewhere in the past we had to turn the mod_deflate off and never brought it back online. Bad robot! We turned it back on and are now gzipping css, js, html and anything text-based which resulted in the total data size dropping to about 2.1 megs. Still a shitload of data though.

2) Lazy-loading of images

Instead of loading 150+ images from the get-go, we only load what's visible in the visitors browser. Anything else is loaded when it scrolls into view. This was the major improvement, bringing us down to about 1.5 megs. Still not slim enough!

3) Reduction of DOM elements

As we've progressed through the last two years, the site has been extended and fixed and extended again - resulting in a patchwork of DIVs and css classes. Some parts were done through Semantic Views and other parts through their own .tpl files. We went through and made sure all parts are rendered from their own .tpl so we have full control over the output. Then we got started reducing and simplifying the files, swapping out wrapper DIVs and extraneous containers. This brought us down to 1044 DOM elements and 1.1mb in size

4) Minimizing external services

We used to have a GooglePlus button on the site. It typically took anywhere between 3 and 10 seconds to load all of it's resources. Need I say it's gone? We also used to have the Facebook activity stream with Faces enabled. It required between 50 and 70 http requests to load all it's resources and took up to 8 seconds to load. No more. We also used to load the ever-ubiquitous JQuery from Googles ajax-cdn to be sure we always had the latest version on the site. I guess we can live with having it lag a version or two if that's the price we have to pay to have a faster site so we cached it on our own servers instead.

We still have a Twitter button and a few Facebook boxes on the page but we'll have to live with them until we develop a better method to load them (preferrably they will be loaded when the user hovers over the boxes)

Result

Final metrics
Total load time: 2 sec (-75%)

Waiting for external services: 2-3 sec

Data size: 1.1mb (-56%)

DOM Elements 1044 (-24%)

YSlow score: B (89)

So here we are. The new site is now a lot slimmer, nearly ready for Beach 2012. Stuff that's left to consider is trimming of the master CSS, adding the responsive parts to the CSS and trimming away some of the extraneous Drupal CSS classes which it so loves to add. All in all, I think the result is quite good but surely there are a some more optimizations left to do. Or at least I hope so, this performance hunting is addictive!

Other possible optimizations

A couple of points that could be worth researching are

Progressive loading of all content, not just images
Dividing the posts into categories, thereby reducing the number of posts on any page
Minifying *everything*
Writing our own sharing functions that don't require hefty scripts from external sources
Spreading the downloads over multiple domains to reduce browser blocking

Tools used

FireBug, a Mozilla extension
Yahoo's YSlow, another Mozilla extension
New Relic, performance analysis tools for php
Coffee and heavy metal

VCL Random url part

2012-02-14T06:51:00.001-08:00

Random is random, usually. Unless it's being fetched every 20th minute and included as a part of a site with heavy traffic. That's a lesson I learned the hard way by relying on an ad networks RSS aggregator that included one of our RSS feeds on it's pages. The feed is randomized on our end with a list of nodes that are straight out of Views via some custom theming. The nodes are grouped in threes from a larger set.

What I noticed was that over 24 hours, what should have been an even distribution turned out to be heavily weighted towards one specific set. The ad networks RSS aggregation ran on a schedule and as bad luck would have it, Views liked to display the same set every time the RSS aggregator made a visit. Probably just bad luck but to the customer it looked like one set of nodes were favored over the others.

As I run Varnish for caching I started thinking about if I could do this in another way. I thought about faux round-robin directors and complicated solutions involving dns aliases and htaccess rewrites.

Then I stumbled upon this blogpost by Chris Davies who is just generally speaking one of the most knowledgeable techies I know of. So enter inline-C for my VCL.

sub vcl_recv{

    if (req.url == "/randomizeme") {

C{

char buff[5];

sprintf(buff, "%d", rand()%4+1);

VRT_SetHdr(sp, HDR_REQ, "\017X-Distribution:", buff, vrt_magic_string_end);

}C

        set req.url = "/randomizeme/" req.http.X-Distribution;

    }

 }

It works like this:
- Varnish picks up the request on url /randomizeme
- Varnish executes the inline C
- Varnish sets a special header with the randomized integer (the header works as a variable)
- Varnish then rewrites the url to /randomizeme/1-4
- Varnish finally fetches the above url from the backend which is rigged to display different content depending on which "slot" is chosen.
- Varnish delivers the randomized content to the visitor/aggregator/whatever

A little warning: If you modify the header name "X-Distribution", note that the "\017" must be updated, it's octal for the length of the string. I forget if it should include the ":" or not. Trial and error, watch that log for segfaults!

Swedish browser statistics Q4 2011

2011-11-02T08:17:00.001-07:00

It's been a while since I did this last but it's time for another rundown of who uses what in the Swedish (and nordic) marketspace.

These figures are extracted from a 24-hr statistics collection period and represent around five million pageviews of which roughly 350K were mobile views. Total unique visitors around one million. The sources are mainstream sites that serve mainly swedish content and targets a wide array of audiences.

Desktop browsers:

	Browser make	Now	Feb2011	Change
1.	Firefox	43,42 %	(51,92%)	-8,5%
2.	Chrome	42,07 %	(31,57%)	+10,5%
3.	Internet Explorer	9,25 %	(12,12%)	-3,13%
4.	Safari	3,23 %	(2,16%)	+1,08%
5.	Opera	1,60 %	(1,81%)	-0,21%
6.	Mozilla Compatible Agent	0,20 %	(0,01%)	+0,19%
7.	RockMelt	0,08 %	(0,11%)	-0,03%
8.	BlackBerry9700	0,04 %	---	+0,04%
9.	IE with Chrome Frame	0,02 %	---	+0,02%
10.	Mozilla	0,01 %	(0,03%)	-0,02%

Mobile browsers:

iPhone	396,326	70.9%
Android	153,449	27.4%
Symbian	8,076	1.4%
Blackberry	992	0.2%
Windows phone	351	0.1%
Windows CE	21	0%
Unknown mobile	17	0%
WebOS	2	0%

Conclusion:
IE steadily declining. Firefox and Chrome dominate desktop market. Android dominates mobile market allthough browser brand and versions seems obscured.

Mobile continues to grow as a platform, now at 14.7% of total visitors. iPad stands for 1.5% of that figure while Android tablets are hard to identify (and GA won't group very good on screen resolutions) so I'll leave that figure dangling until I get better data.

For us CSS and HTML jockeys it sure looks like we get to play with the nice toys a bit more next year since Chrome and Firefox totally dominate the market with a whooping 85.47%. Crossbrowser hell just became a little cooler.

Node.JS-based real-time web tracking

2011-10-05T05:36:00.000-07:00

This all started out of a need. A need for a statistics service that did not suck and that could handle our quite intense traffic (we run at 1500 req/sec max). Google Analytics failed miserably, GetClicky failed a little less miserably and other options were quite simply to expensive for our taste.

Enter Node.JS and a little bit of pre-alpha code known as Hummingbird. I'll let Hummingbird introduce it self:

"Hummingbird lets you see how visitors are interacting with your website in real time. And by “real time” we don’t mean it refreshes every 5 minutes—WebSockets enable Hummingbird to update 20 times per second. Hummingbird is built on top of Node.js, a new javascript web toolkit that can handle large amounts of traffic and many concurrent users."

Sounds good to me, where do I sign up? Well, as is customary these days, this project is on github and is installed by first cloning the repo and then running "npm install" in the hummingbird subdir. Of course, you need to have a recent node.js core, mongodb and libgeoip-dev 1.4.7+ if you want to get ip-location working (hummingbird has a nice little map display in the demo setup). Unfortunately there is not much documentation written (yet) so be prepared for a lot of debugging and guesswork if you want it to do anything besides the graph / total / map that's included in the demo.
Note: On Debian Lenny, libgeoip is at 1.4.4 which means you'll have to load it from lenny-backports to get the correct version (1.4.7+). Instructions for installing from backports can be found here: http://backports-master.debian.org/Instructions/

So how does it work? Well, the core is really a pixeltracker - including a 1x1 pixel from the system in your content will register a view with hummingbird, in effect saving a little bit of data to our mongodb collection. Hummingbird also runs a dashboard (if enabled) that can be viewed via http. The dashboard utilizes Web Sockets for rapid updates and binds the updates to jQuery elements on the dashboard page.

This is about as far as I have gotten right now. The lack of documentation makes it a bit hard to develop more widgets for the dashboard but that's nothing a little trial-and-error won't fix :)

Update:
Managed to get another piece of the puzzle working - a custom widget for the dashboard page. I started with the backend connection by combining the code for cart_adds.js and total_views.js to get a starting point for an event-driven counter widget. Followed up by creating a very rudimentary widget endpoint that picks up on whatever is processed on the backend by hooking into the event system. So any tracking that has a specific event attached to it (basically parts of the query string for the tracking pixel) will result in a display on the dashboard that reacts to new hits by incrementing a value or otherwise adjusting the style or layout of the widget. Nothing new really but still a valuable tool for our editors who are constantly chasing those extra page views.

I'll update again once I have a better implementation of it all.

How to ip-migrate web sites using Varnish

2011-08-18T04:18:00.001-07:00

This is a quick guide on how to migrate a bunch of web sites by using Varnish as a intermediary to allow for low-latency zero downtime migration between ip addresses or domain names.

Requirements: Varnish 2.x up and running, access to apache config, access to dns config.

1) Configure Varnish

Start out by defining your existing web sites as backends in the varnish config (default.vcl) by adding them in this format to the beginning of your vcl:

backend website1{
// NO CACHING
// website1.company.com
.host = "10.0.0.1";
.port = "80";
.connect_timeout = 160s;
.first_byte_timeout = 55s;
.between_bytes_timeout = 25s;
}

backend website2{
...
}

backend website3{
...
}

Then add some clever host/backend switching logic to your vcl:

sub vcl_recv {
# Find out which backend to use for the request
if (req.http.Host == "website1.company.com"){
set req.backend = website1;
return(pass);
}
else if (req.http.Host == "website2.company.com"){
set req.backend = website2;
return(pass);
}
else {
# maybe switch to your default backend here if you're using varnish in production
}
.....

You're now ready to start using varnish as your intermediary frontend router. Varnish will PASS any requests to these sites and you are now free to change the IP:s of the sites without the lag of DNS. You remembered to load and use your varnish config too, right?

2) Alter DNS for sites
Update your dns records, all www pointers should be changed to the ip address of your Varnish server.

3) Migrate!
When the changes have propagated (and you've verified that it works as intended) you are free to change ip:s or whatever you need on your sites, Varnish will maintain the dnsname -> backend connection as long as you remember to update your backend config in vcl. And, changes are instant - allowing fast rollbacks if needed.

How to use bash to find all used IP:s in your apache configs (following the debian config model)

2011-08-17T06:49:00.000-07:00

for i in `ls /etc/apache2/sites-enabled/*.vhost`; do echo "Working file:$i"; echo "Servername:";grep "ServerName" $i | cut -c 16-40; echo "Ipaddress:";grep "<VirtualHost" $i|cut -c 14-27; done

Top 5 Varnish commands

2011-03-22T07:08:00.000-07:00

Here's a brief list of my top 5 varnish commands

varnishstat
This is what I use to get a birds-eye view of what we have cooking. Provides all the info you need to spot cache misses and errors.
Usage: varnishstat for continuous display or varnishstat -i 1 for a quick one-off. More on this command: http://www.varnish-cache.org/docs/2.1/reference/varnishstat.html

varnishhist
Provides a histogram view of cache hits/misses
Usage: varnishhist More on this command: http://www.varnish-cache.org/docs/2.1/reference/varnishhist.html

varnishlog
Provides detailed information on requests. I use it to debug cache operations by for example issuing the command varnishlog -c -o ReqStart [insert ip here] which allows me to see what happens when I visit a cache object in my browser.
Usage: varnishlog -c -o ReqStart 192.168.0.1 More on this command: http://www.varnish-cache.org/docs/2.1/reference/varnishlog.html

varnishtop
I use this mainly to get lists of things. Like this one that gives me a list of the top urls hitting the backend (pass).
Usage: varnishtop -b -i TxURL More on this command: http://www.varnish-cache.org/docs/2.1/reference/varnishtop.html

varnishadm
Command-line varnish administration. I mainly use this to reload vcl and purge urls.
Usage: varnishadm -t host:port command
Reloading a vcl from the command line can be done like this:
Load (your changed vcl): varnishadm -t localhost:6082 vcl.load load01 /etc/varnish/default.vcl
Use (apply) the vcl: varnishadm -t localhost:6082 vcl.use load01
More on this command: http://www.varnish-cache.org/docs/2.1/reference/varnishadm.html

Facebook iframe like box tracking with Clicky (or Google Analytics)

2011-03-15T14:29:00.000-07:00

This is how to do GetClicky tracking for facebooks fbxml iframe like box and the fbxml like button.

Without going into details about fb app id and such, this is a working method to implement the GetClicky tracking for facebook like boxes and buttons.

Standard fb script:

<div id="fb-root"></div>
<script type="text/javascript">
//<![CDATA[
window.fbAsyncInit = function() {
 FB.init({appId: '157019890979609', status: true, cookie: true,
 xfbml: true});
});
(function() {
var e = document.createElement('script'); 
e.async = true;
e.src = document.location.protocol +
'//connect.facebook.net/sv_SE/all.js';
document.getElementById('fb-root').appendChild(e);
}());
//]]>
</script>

The facebook "like" event hook:

FB.Event.subscribe('edge.create', function(href, widget) {
// do something here
});

The clicky call:

if(typeof(clicky) != "undefined"){
clicky.log('page url','facebook like clicked','click');
}

Combined:

<div id="fb-root"></div>
<script type="text/javascript">
//<![CDATA[
window.fbAsyncInit = function() {
FB.init({appId: '157019890979609', status: true, cookie: true,
xfbml: true});
FB.Event.subscribe('edge.create', function(href, widget) {
if(typeof(clicky) != "undefined"){
clicky.log('page url','facebook like clicked','click');
}
});
};
(function() {
var e = document.createElement('script'); 
e.async = true;
e.src = document.location.protocol +
'//connect.facebook.net/sv_SE/all.js';
document.getElementById('fb-root').appendChild(e);
}());
//]]>
</script>

Of course the same works for Google Analytics, just swap clicky.log for _gaq.push

Custom Drupal RSS Feed with CCK

2011-03-14T15:29:00.000-07:00

Ever needed to do a quick RSS feed one-off out of your Drupal site? This is how I do it.

Setup: Drupal 6, CCK, Views enabled.

First, create a new content type and name it rssarticle. Add three text fields named heading, text and link. Save the content type.

Now, we'll use Views as a query builder that will provide the SQL you will run to build the feed. Start out by setting up a Node View, set Style to unformatted and Row style to Fields. Select the fields you want in your feed and move on to add a node type filter to restrict the view to our newly added rssarticle content type. Keep the rest of the settings untouched.

Now, hit the Preview button. You should now see some nodes rendered and a Query field that displays the query like so:

SELECT node.nid AS nid,
   node_data_field_heading.field_heading_value AS 
   node_data_field_heading_field_heading_value,
   node.type AS node_type,
   node.vid AS node_vid,
   node_data_field_heading.field_link_value AS 
   node_data_field_heading_field_link_value,
   node_data_field_heading.field_text_value AS 
   node_data_field_heading_field_text_value
 FROM node node 
 LEFT JOIN content_type_rssarticle node_data_field_heading ON 
   node.vid = node_data_field_heading.vid
 WHERE node.type in ('rssarticle')

Copy that SQL query and fire up your favorite php editor. Create a new php file by the name of mycustomfeed.php. Start by adding the following lines.

<?php
// Load necessary Drupal parts
require_once './includes/bootstrap.inc';
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);

// XML header
echo('<?xml version="1.0" encoding="utf-8" ?>');

// Get nodes from db
$results = db_query("");

// Loop over the result
while ($result = db_fetch_object($results)) {
 print $result->nid;
}
?>

Now this won't do much on it's own, we need the SQL you just copied to go in here too. Complete the code with this, placing it within the db_query("") statement.

<?php
// Load necessary Drupal parts
require_once './includes/bootstrap.inc';
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);

// XML header
echo('<?xml version="1.0" encoding="utf-8" ?>');

// Get nodes from db
$results = db_query("SELECT node.nid AS nid, 
node_data_field_heading.field_heading_value AS 
node_data_field_heading_field_heading_value, node.type AS 
node_type, node.vid AS node_vid, 
node_data_field_heading.field_link_value AS 
node_data_field_heading_field_link_value, 
node_data_field_heading.field_text_value AS 
node_data_field_heading_field_text_value FROM node node 
LEFT JOIN content_type_rssarticle node_data_field_heading 
ON node.vid = node_data_field_heading.vid WHERE 
node.type in ('rssarticle')");

// Loop over the result
while ($result = db_fetch_object($results)) {
 print $result->nid . "\n";
}
?>

Upload the file to your Drupal root and run either from a browser or from the command line.

It should output an xml header and list of NIDs. Check error logs for any warnings or sql errors and correct your code accordingly.

Now that we have the query working it's time to finish the job, adding the xml parts required for a valid RSS 2.0 feed as well as adding a LIMIT clause and performing some ISO date conversion magic.

<?php
// Load necessary Drupal parts
require_once './includes/bootstrap.inc';
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);

// XML header
echo('<?xml version="1.0" encoding="utf-8" ?>');
?>

<rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>My Custom Feed</title>
    <description>This is My Custom Feed</description>
    <link>http://www.domain.com/mycustomfeed</link>

<?php

// Get nodes from db
$results = db_query("SELECT node.nid AS nid,
node_data_field_heading.field_heading_value AS 
node_data_field_heading_field_heading_value, 
node.type AS node_type, node.vid AS node_vid, 
node_data_field_heading.field_link_value AS 
node_data_field_heading_field_link_value, 
node_data_field_heading.field_text_value AS 
node_data_field_heading_field_text_value,
node.changed AS node_changed FROM node node 
LEFT JOIN content_type_rssarticle node_data_field_heading 
ON node.vid = node_data_field_heading.vid
WHERE node.type in ('rssarticle') LIMIT 0,10");
// Execute query and loop over the result
while ($result = db_fetch_object($results)) {
?>
<item>
<title>
<?php echo $result->node_data_field_heading_field_heading_value?>
</title>
<description>
<?php echo $result->node_data_field_heading_field_text_value?>
</description>
<link>
<?php echo $result->node_data_field_heading_field_link_value?>
</link>
<author>noreply@domain.com (Corporate Inc)</author>
<dc:creator>Head.Honcho</dc:creator>
<category>CustomCategory</category>
<guid isPermaLink="false"><?php echo $node->nid?></guid>
<pubDate>
<?php echo date('D, d M Y H:i:s T',$result->node_changed)?>
</pubDate>
</item>
<?php } // End while loop ?>
</channel>
</rss>

For more on custom rss from drupal, check out my post on rss with imagecache images and media enclosures

Drupal RSS 2 with imagecache images in media enclosures

2011-03-03T07:35:00.000-08:00

I got a new project to do. It involved publishing news clips to various sites via valid rss 2.0. The feed includes a headline, a link, a short text and an image. But, the image has to go in an enclosure like the following

<enclosure url="http://domain.com/file.jpg" 
length="3737" 
type="image/jpeg"/>

The way I chose to do this was by setting up a view to handle the listings and then doing a custom template based on the dead simple Basic. I removed all trim and just added the outline of a valid rss 2.0 feed structure by creating my page.tpl.php, node-view-viewname.tpl.php and views-view.tpl.php where node-view-viewname is the template file that will print the actual item. Easy.

Now here comes a problem... To make up for the never-ending creativity of our dear editors, we use ImageCache. Without it we would be lost in a world of image sizes and formats. If you are at all familiar with ImageCache you know that it's lazy and only generates an image if a request for an image fires a 404 error. This is fine for most purposes but since rss media enclosures require certain metadata about the file and it doesn't exist yet we need to force ImageCache to generate the image before we can fill the enclosure tag with the correct values.
We can do this in template.php by adding the following function:

/*
* This function pregenerates ImageCache images
* to give access to the file metadata
* @param $vars
*/
function basic_attach_ic_metadata(&$vars) {
$presetname = "100x100";
$filepath = $vars['field_bild'][0]['filepath'];
$preset = imagecache_preset_by_name($presetname);
$dst = imagecache_create_path($presetname, $filepath);
if (!file_exists($dst)) {
imagecache_build_derivative($preset['actions'], $filepath, $dst);
}
$vars['genimage'] = $dst;
}

Which in it's turn gets called from template.php like so:

<?php function basic_preprocess_node(&$vars, $hook) {
  basic_attach_ic_metadata(&$vars);
...
}>

Now, when the node is prepared for display the variable $genimage will be set with the path to a generated image. This now available for your node template to consume and we can print the enclosure tag.

<enclosure url="http://domain.com/<?php echo $genimage;?>"
length="<?php echo filesize($genimage);?>" 
type="image/jpeg"/>

All done!

External ads in Drupal

2011-03-01T12:49:00.000-08:00

Allrighty. So someone is twisting your arm and trying to make you add blocks of ad network script tags on your perfect Drupal site? Polluting your DOM? No way, not on my shift! Here's how to keep it safe.

First you need a basic understanding of a few concepts. I'll use Taxonomy, CCK, Views and some funky template preprocess magic all bound together by our beloved page.tpl.php. The external ads are presumably a bunch of script tags.

Start out by creating a .php or .html file that hold each ad, place these somewhere within your drupal folder in a folder, why not name it "ads". Any cache-busting techniques should be done with php. Note the full URL:s.

Now for some Drupal hands-on: Create a Vocabulary that will hold the tags that will tell Drupal which nodes get what ads. Name it "Customers". It should be a flat vocabulary.

Next, create a content-type that will serve as the holder for the dreaded external ads. Name it "Customer". Add as many text fields as the total of your ad slots. The text fields will only hold URLs and should be plain text only. Save when done. (You can easily add more custom fields here that can be used for your nodes)

Move on to Taxonomy->Vocabularies. Configure the previously added vocab so that it is selected for your "regular" content-type so you can relate it with its Customer. Now, add as many terms to the vocabulary as you have customer sections, i.e. companya, companyb, companyc.

Create a Customer node. Populate the fields with the URL:s from the first step. Fill the title field with one of your terms from Customers, for example "companya". Note: This needs to be unique.

Now, edit one of your "regular" nodes. Add the term "companya" to the node and save it.

Set up a view to load your Customer data by creating a new View, naming it "customerdata". The View should be Unformatted and row style should be Node. Add Node:Title as an argument, accepting the default 404/hide view settings. Add a filter on node type and set it to Customer. Save it.

The time has come for us to bring it all together and display the ads. This is when your template.php hacking skills come into play. Fire up an editor and open template.php for the theme you'll be using. You now need to create a function that will load the data from the Customer node and attach it to the node before rendering.
You also need to make sure this gets called when the page is constructed. You might need it on the page level or the node level, it depends.

template.php:

The last part is your page.tpl.php. To print your now safer external ads, construct something along the lines of

echo "iframe src=" . $ad1 . "/iframe";

The same can be repeated for any custom field. The process is the same, add it to the Customer content-type, fill out the Customer node, tag your node(s) with the appropriate term and output it in your page.tpl.php.

Upload template.php and page.tpl.php, flush your caches and load one of your tagged nodes. You should now see the ad being displayed. If not, check the html source for your iframe tag or flush your cache.

This example is just a basic example and can be improved upon endlessly. A few of those being Rules integration, error checking and Views caching.

Christopher Cato - my old blog posts on tech

Translating the Site Name in Drupal 7

Script ads and their inherent security flaws

Fabric fabfile for Varnish deploy and ban

Bulding a custom token for url aliases (or anything else)

Why Display Suite is a bad idea

And why Code Fields are an even worse idea

Apache Solr Search and FacetAPI translation

Drupal and Ajax, now with language negotiation...

String just won't translate?

Getting the path of a translated node

Adding new node operations for the content overview screen

Drupal 7 translation gotchas

Don't slip.

HAProxy and Keepalived on Debian Squeeze for failover and loadbalancing

Building a failover load balancing cluster on four machines with HAProxy and Keepalived in Debian Squeeze

Outline

The tools

The servers are: proxy1 haproxy, keepalived, ip 11.22.33.42 (sharing ip 11.22.33.44) proxy2 haproxy, keepalived, ip 11.22.33.43 (sharing ip 11.22.33.44) cache1 varnish, apache2, ip 11.22.33.45 cache2 varnish, apache2, ip 11.22.33.46

Install HAProxy and Keepalived

Configure HAProxy and Keepalived

Configure sysctl

Configure cache servers / backends

Start services

Verify that the system works

Verify the failover function

Benchmarking

Deploying

Monitoring

Troubleshooting

Gotchas

Resources

Rendering fields correctly in Drupal 7

Site performance - analyze and improve

VCL Random url part

Swedish browser statistics Q4 2011

iPhone

396,326

Android

153,449

Symbian

8,076

Blackberry

992

Windows phone

351

Windows CE

21

Unknown mobile

17

WebOS

2

Node.JS-based real-time web tracking

How to ip-migrate web sites using Varnish

How to use bash to find all used IP:s in your apache configs (following the debian config model)

Top 5 Varnish commands

Facebook iframe like box tracking with Clicky (or Google Analytics)

Custom Drupal RSS Feed with CCK

Drupal RSS 2 with imagecache images in media enclosures

External ads in Drupal

The servers are:

proxy1 haproxy, keepalived, ip 11.22.33.42 (sharing ip 11.22.33.44)

proxy2 haproxy, keepalived, ip 11.22.33.43 (sharing ip 11.22.33.44)

cache1 varnish, apache2, ip 11.22.33.45

cache2 varnish, apache2, ip 11.22.33.46