Problem

If you've been reading this blog you might have noticed I published some wordpress plugins, patches and an xml sitemap module for pligg. Sometimes other people write posts about my patches but instead of linking to my posts they link directly to the download. This creates a series of problems: people might miss important information about the download, page rank is uselessly transferred to a zip, tar.gz, or .patch file, and you're basically serving content for other people's posts while they rip all the benefits.

Solution

So here's an easy way to avoid it. Well you can't really avoid it but you can benefit from it. All you have to do is set up a .htaccess file in your wp-content/upload directory ( that's where downloads are storred by default, feel free to change the location if you're using something else.

This .htaccess file will check the referer of every request on any file in that folder and if the referer doesn't match your domain it will redirect the visitor the search page on your blog with the search term set to the name of the file they wanted to download. most of the time this search will show as the first result the post where you published.

Here's how the file looks on my blog:

RewriteEngine On
RewriteBase /wp-content/uploads/
RewriteCond %{HTTP_REFERER} !(www\.)?patchlog.com [NC]
RewriteCond %{HTTP_REFERER} !^$
RewriteRule ([^\/]+)$ http://patchlog.com/index.php?s=$1 [R=permanent,L]

Pretty simple huh?
What about the SEO profits?
As you can see the last rule redirects using a permanent redirect and this means google will transfer page rank to the search page.

Other solutions?

This method is good because it's really easy to implement, but I would like a method that would redirect the visitor directly to the post page instead of the search page.  I'm hoping the wordpress download monitor plugin that I'm using will at some point implement an advanced hotlink protection method but until then or until I find time to do it myself this is good enough.

This is a post from Mihai Secasiu's PatchLog - Patches 'n Hacks.

Simple Hotlink protection for SEO profits

The previous post shows you how you can configure the outgoing ip in antinat but if you have multiple ips and you want to use all of them and you want to be able to control which one to be used for certain things that patch doesn't do enough for you.

Solution

Antinat should bind the ougoing connection on the same ip on which it receives the connection from the client.

So if you want to use a different ip just set your configure your socks settings in the browser or proxifier to the ip you want antinat to use.

And here's the patch to let you do that ...

Antinat outgoing ip same as Incomming-1.0 (10.1 KB)

This patch is incompatible with the one on the previous post, you can either have that one or  this one so make sure you apply it on the original antinat source.

Questions or suggestions are welcome as always ...

This is a post from Mihai Secasiu's PatchLog - Patches 'n Hacks.

Antinat outgoing ip same as incoming

Problem

Antinat creates outgoing connection from the primary ip defined on the machine where it's running. There's a config option to make antinat listen on a specific ip but no config option to make it use a specific ip for outgoing connections.

Solution

The attached patch will make antinat use the same ip that it's listening on for outgoing connections. You specify the listening ip with the "interface" config option and now that ip will also be used for outgoing connections.

Antinat outgoing ip-1.0 (3.57 KB)

I assume you already know how to patch ... if not ... just ask in the comments or hire me to patch it for you

This is a post from Mihai Secasiu's PatchLog - Patches 'n Hacks.

Antinat outgoing ip

In my last post about qmail I said that once you solve the big concurrency problem you'll end up with another problem because your mail server would create too many outgoing connections to some domains and you risk having your ips banned by those servers.

Solution

The solution is to have a way of limiting the maximum concurrency rate by domains. To do that you'll need the   qmail channels patch or write your own patch like I did ( mostly because I was unaware of the existence of the qmail channels patch )

The home page of the qmail channels patch will explain how to setup and configure qmail to limit the concurrency by a domain or group of domains.

What I like about this patch is that it allows you to set a concurrency limit for a group of domains like set 100 for all yahoo.com, yahoo.co.uk, yahoo.ca, etc .

What I don't like is that it doesn't seem to be able to set a default concurrency level for any domain. If I'm wrong please correct me, but if I'm right then this seems like a major problem for an email server that sends to a large number of addresses distributed over a large number of domains because you would have to configure concurrency limits for a lot of domains.

The ideal solution would allow you to specify a default per domain concurrency and this would apply to any domain that doesn't have a specific concurrency. For example most email servers would be ok with 5 concurrency connections from the same ip but no way for AOL (unless you're white listed and maybe not even then ) .

Another feature I would like is to be able to specify concurrency by domain's MX records or ips/group of ips assigned to the MX servers instead of the actual domain. This would ease the configuration for ISPs that host a lot of domains like rr or yahoo.

This is a post from Mihai Secasiu's PatchLog - Patches 'n Hacks.

Qmail per domain concurrency

By fetching as little as possible your server gets less hits so it minimizes the load and the need to acquire new hardware and complicated setups but this also improves user's experience a lot because the web application will load a lot faster since most files ( scripts, css, images ) are already on her/his disk.

The idea is to set such a high cache expiry time or ( max-age and other parameters ) that they ( browsers ) would not even want to look for newer versions for a long time ( like a year or more)

Here's what I learned recently when trying to optimize a big web application built on javascript  and php:

0) Page analysis

Before you get started get Page Speed or Yslow and do an analysis on your app/site then come back here and see how you can solve the caching problems listed there.

1) High cache age is good but what do you do when your site changes?

You definitely want to force the changes to your users right ?

Answer: version everything.

You may have noticed the way a lot of sites include scripts and css files with a version at the end like : jquery.js?ver=1232442

Here's how this works: The main page that includes this script is not cached so the visitor will load it every time but the browser caches the jquery.js?ver=1232442 url ( because you said so in your web server config ).

Now if you update jquery to a new version all you have to do is modify the url like jquery.js?ver=1232443 in the main page and the browser will know it has to fetch the jquery.js file again because from it's point of view it's a totally different file.

If you can use php in the template that outputs the page you could even do something like:
<script src="jquery.js?ver=<?=filemtime('jquery.js')?> . By doing this you don't have to worry to update the main page when you update jquery.js.

2) CSS/HTML rewriting.

So you do this versioning thing for javascripts and maybe css files but what do you do about images? How do you cache them and still make sure your visitors will always see the latest version?

Your images are referenced from the css files or HTML content. You probably already serve your HTML content through a script ( cms ? ) so you'll have to modify this script to automatically add the versioning string to each image or other static file you want cached.

For css do the same, serve it through a script and before you output modify the image paths or even better, especially if you use multiple css files, write a script that generates one file from all ( it loads faster this way ), does the rewrite, then minifies it, save it, then it compresses it and stores the compressed version so you can serve that when possible. You would have to run this script every time you change something to your css code.

3) HTTP proxies cache differently.

It is believed that most will not cache URLs with query strings in them like jquery.js?ver=122323

A HTTP proxy can minimize the hits on your server by fetching only once and distributing to more then one user but if you want to take advantage of that you have to use a different versioning scheme.

An idea is to insert the version before the file extension like: jquery-122323.js so the URLs would no look "dynamic" anymore.

If you do this and you don't actually want to rename all the files you could use some mod_rewrite rules to redirect anything matching that pattern to the actual files.

4) HTTPS is a different animal

Yeah browsers will not cache content that comes over https because it's considered a security issue. Imagine your app generates a pdf or image with sensitive user info and "says" yeah you can cache it for a year, and the user downloads it in a publicly available computer. The next user will get the same file. Of course this would happen over HTTP too so be careful with what you allow to be cached. The only difference with HTTPS is that the browser will disregard normal caching instructions if the file is served over HTTPS.

Now you would say "why would you even want to send generic scripts, css or images over https?" ...right ... Well you do because if you allow HTTPS access to your app and you don't send everything over HTTPS then the browsers would warn the user that not everything on the page is encrypted. Now some users wouldn't care especially if they know what the warning means or how to check what's not encrypted, but other's might freak out about it.

So if you want to send everything over HTTPS and you want the browser to cache the files you have to set  the header "Cache-control: public" but again....make sure you only set this for static files that are generic for all users.

And if you set Cache-control add the max-age to it otherwise if you only set "public" it might invalidate any other "max-age" set in other headers like "Expires". So the header should look like: Cache-control: public, max-age=31536000 ( cache even for HTTPS and authenticated (HTTP authentication) users for a year )

5) Gzip caching

If you're using apache then it's probably already using mod_deflate to compress static files when talking to browsers that accept deflate as the Content-encoding. This is good as it speeds up page loading a bit but this means that apache it's compressing the same content over and over for each visitor consuming your CPU time. And even if you do caching as mentioned above it will still compress for new visitors. So why not cache the compressed content once and server it to everybody?

To do that you'll have to use mod_gzip . This apache module will negotiate Content-encoding with browsers and if the browser supports it then it will send the compressed file instead of the non compressed one. mod_gzip will do even more, it will pre-compress the files so you don't have to do it yourself and it can figure out by itself when you updated the original file and it will regenerate the compressed version. mod_gzip can really save a lot of cpu time for your server.

6) Caching Dynamic content

This basically means generate static content from your dynamic one and save it on disk ( plain and compressed ... see #5 ) so apache or a script can serve it directly without having to go to the database or compute the results . Wp-super-cache does something like this for wordpress.

Since dynamic content is more likely to change often and it's most likely not referenced from other non cachable pages like images,css and JavaScript you can't set a high cache max age for it so you can't reduce the hits so much.

But if you serve it through a script that can easily ( cheaply ) determine that the content has not changed then that script can issue a "304 Not Modified" response and the browser will know that it already has the content. This may be a lot faster then actually regenerating the dynamic content and sending it to the client.

Here's how to do dynamic content caching in PHP

There's also a lot of caching that can be done at the database server level or before/after talking to the database server ( memcached ) but this is totally different topic.

What else ?

Did I miss anything ? If you know other techniques I'd love to read about them so feel free to hit the comments but not too hard as this blog doesn't do much of the caching discussed here

BTW: that big web app I mentioned at the beginning of this post is an email marketing service that I just launched in beta. If you run a blog and you think about sending a newsletter you might want to try it. Beta testers get some nice benefits.

This is a post from Mihai Secasiu's PatchLog - Patches 'n Hacks.

7 Methods to cache web applications

Powered by Twitter Tools

This is a post from Mihai Secasiu's PatchLog - Patches 'n Hacks.

This week on twitter 2010-02-21

Mod rewrite does NOT match your pattern on the query string but only on the path part of the URL.
To match the query string you must use the RewriteCond rule.

From mod_rewrite documentation:

Note: Query String

The Pattern will not be matched against the query string. Instead, you must use a RewriteCond with the %{QUERY_STRING} variable. You can, however, create URLs in the substitution string, containing a query string part. Simply use a question mark inside the substitution string, to indicate that the following text should be re-injected into the query string. When you want to erase an existing query string, end the substitution string with just a question mark. To combine a new query string with an old one, use the [QSA] flag.

That last  part about QSA was the one that made me rediscover this

This is a post from Mihai Secasiu's PatchLog - Patches 'n Hacks.

Mod_rewrite quick tip

@MarcPerel I guess that means the cellphone company is trying to screw you more then your bank in reply to MarcPerel #

Powered by Twitter Tools

This is a post from Mihai Secasiu's PatchLog - Patches 'n Hacks.

This week on twitter 2010-02-14

@bensonk42 firebug, working session save/restore, all those saved passwords #chrome seems like just an experiment good to play with in reply to bensonk42 #

Powered by Twitter Tools

This is a post from Mihai Secasiu's PatchLog - Patches 'n Hacks.

This week on twitter 2010-02-07

I can finally use #thunderbird 3.0 now that I managed to exclude some folders from indexing. I like how fast it can delete messages #

Powered by Twitter Tools

This is a post from Mihai Secasiu's PatchLog - Patches 'n Hacks.

This week on twitter 2010-01-31

Powered by Twitter Tools

This is a post from Mihai Secasiu's PatchLog - Patches 'n Hacks.

This week on twitter 2010-01-24

@andreisavu if it would make it faster then it wouldn't be bad code anymore ...but I may be wrong, sometimes bad code means faster dev #

Powered by Twitter Tools

This is a post from Mihai Secasiu's PatchLog - Patches 'n Hacks.

This week on twitter 2010-01-17

@alicam it's also an example of "ugly sites can still be popular" in reply to alicam #

Powered by Twitter Tools

This is a post from Mihai Secasiu's PatchLog - Patches 'n Hacks.

This week on twitter 2010-01-10

The problem with this feature is that it only supports the DynDNS service and not everyone is using this service to update their host when their ip changes.

I'm using my own dynamic dns service and the good news is that this service is using the same protocol as DynDNS so virtualmin needs little change in order to support it.

All I had to do was add a way of specifying the hostname and port of the dynip update server.

Here's the patch for virtualmin ( tested with 3.75 and 3.76 ) :

This is a post from Mihai Secasiu's PatchLog - Patches 'n Hacks.

Virtualmin custom dynip

A: Because you have your own domains that you want to use as the parent domains for your hosts and you already have all required ingredients ( a linux server with a static ip that's always up and runs bind and apache with php )

Q: How do you run your own dynamic DNS service?

A: get DDNS Server, read the README file and set it up. The nice thing about this script is that it uses the same protocol as the popular DynDNS service so any dynamic ip update client like ddclient would work with this script. Of course the client would have to allow you to specify the hostname or ip and port of the dynamic dns server.

Oh and one more thing, if you're running bind you're probably running it in a chroot environment so you'll have to apply the following patch to DDNS Server ( current version 1.0.0 ) to make it work with the chrooted bind :

This patch adds a new configuration option named BIND_CHROOT ( in config.php ) that has to be set to the path of the chroot environment where bind runs. Eg. on centos 5.3 this is /var/named/run-root .

For any questions or suggestions hit the comments.

This is a post from Mihai Secasiu's PatchLog - Patches 'n Hacks.

Build your own dynamic DNS service