EZAPPS.CA Development Journal

Zoom: The Magento Full-Page Cache goes Open Source

2012-03-30T14:29:00.000-07:00

Support the Zoom FPC Project:

Since releasing the Zoom Full-Page Cache for Magento, things have been exciting around here, but we are only getting started! From this point forward, Magento no longer needs to be saddled with the reputation of being a slow resource-hog. And no longer will businesses need special hosting providers or rooms full of servers to run a competent store. Zoom is now open-source at GitHub and will be coming to Magento Connect soon.

We jumped into this thinking that a small, quick project like the Zoom FPC woud be a great initial offering of our commercial services. But after some long thought and recent interaction with many community members, we've considered the following:

Magento really needs ZoomI have experienced everything from 4/10th of a second to 4 second page loads with Magento CE on highly optimized systems. This expansive code base--weighed down with magic gets, heavy-abstraction and resource-insensitive design--can buckle under its own weight, rendering it useless in many scenarios. The Zoom Full-Page Cache really does unlock a large portion of Magento's potential by delivering the pages as fast as static content. Implemented properly, your webserver becomes your bottle-neck and Magento Community performs like enterprise level software.
There are already enough commercial products fixing basic deficienciesI love open source software as much as the next guy, but I am apprehensive of the Open Source as Crack business model. Lipstick and potential can lure a "customer" into adopting a free platform, only to find themselves in a corner faced with a decision: "should I cough up thousands for an actual usable product or should I migrate yet again?" Many free products are simply unsupported demos, and entire industries exists to "fix" these products for distressed users--I don't want Ezapps to be a part of that model.

At this point, Zoom will never have a commercial version. What you see is what you get. When I have time, I will fix bugs found in the system. I will also accept community offered bug-fixes and incorporate them into the main code.

With that said, I have been very busy altering Zoom to work with highly customized Magento stores, as well as expanding it. If you need any help in that area, contact us and we will be happy to provide a quote.
This allows us focus on more important offeringsThe Zoom FPC was never originally on our radar (most of our active projects are platform independent). It was a quick idea, rush-coded in the midnight hours for Magento mass-consumption. Open-sourcing this product lets us get back to the basics. If the response to Zoom continues to be positive, we will probably be bringing these other projects out commercially.

So, to finish up, if you download Zoom and find it useful, please realize that we are a small company that will only thrive if you:

Tell your friends
Report bugs and contribute
Support the project

If you need installation help or any custom work done, such as expanding Zoom's hole-punching to custom blocks or resolving conflicts with other custom modules, please let us know. Thanks!

Zoom Full-Page Cache and Customized Magento, Part 1

2012-03-28T17:49:00.000-07:00

Resolving Conflicts

The most common question I have been asked about the Zoom Full-Page Cache is: will it work with my custom Magento modules? In most cases Zoom has proven to be versatile, but if you have a module installed that is trying to override the same core block that Zoom is, there is some leg-work required.

When these conflicts appear either 1) a block will not be punched, or 2) it will be punched, but it will lack the functionality of your custom module. In one scenario, your block is being used, while in the other, the Zoom block has been chosen by Magento. There is also the case of a model being in conflict, and that is resolved in a similar way to what I describe below.

When there is a conflict, you will normally have a class that looks like:

class Custom_Module_Block_Template_Links extends Mage_Page_Block_Template_Links

And a Zoom class that looks like:

class Ezapps_Zoom_Block_Wrapper_Links extends Mage_Page_Block_Template_Links

If you do not feel comfortable altering your store source, or even the source of your extensions, you may not want to proceed without professional help. When in doubt, please feel free to contact us and we can provide a quote. And remember, always backup your source and database before making evasive changes.

Method 1: Using your block

Zoom overrides very little core functionality, and is normally just trying to wrap the HTML content to mark a "hole" (an area to remove and fill in later with dynamic content). So the easy answer is to simply copy the Zoom source into your custom class.

First, copy the following variables/code from the Zoom block to yours. You may have to merge the two versions if they share methods:

private $_key = 'holename';
private $_cache_tag = true;

public function __construct() {  // COPY ZOOM CLASS CODE  }

public function _afterToHtml($html) {  

    // ONLY CALL ZOOM FROM CUSTOM MODULE IF IT EXISTS AND IS ACTIVE
    $modules = (array)Mage::getConfig()->getNode('modules')->children();
    if (array_key_exists('Ezapps_Zoom', $modules) && 
        $modules['Ezapps_Zoom']->is('active') ) {

           // COPY ZOOM CLASS CODE

    } else return parent::_afterToHtml($html);
}

public function setCacheTag($status) {  // COPY ZOOM CLASS CODE  }

public function getCacheTag() {  // COPY ZOOM CLASS CODE  }

Next, remove the rewrite statement for this block in Zoom's "config.xml". Your custom block will now create its own hole.

Method 2: Using Zoom's Block

You may not want to alter your own custom block, because this may prevent future upgrades of that module. A more brute-force method of over-ridding a previously over-ridden core block is as follows...

First, change:

class Ezapps_Zoom_Block_Wrapper_Links extends Mage_Page_Block_Template_Links

To:

// PATH TO CONFLICTING BLOCK
require_once("Custom/Module/Block/Template/Links.php");

class Ezapps_Zoom_Block_Wrapper_Links extends Custom_Module_Block_Template_Links

Next, remove the rewrite statement for this block in the custom module's "config.xml". The Zoom block will inherit the other blocks changes, and continue to create the hole.

Method 3: Give EZAPPS a Call

By now, we have a plethora of experience when it comes to identifying and resolving these conflicts, and we have made Zoom cache very difficult content without dramatic effort. What might be hours for you could be a few minutes for us (though we have faced some very customized Magento stores that required a great deal of effort). We have some popular extensions already modified to play nice with the Zoom Full-Page Cache.

With that, I hope enough people in the community learned how to resolve conflicts when you have two modules competing for the same core Magento block. This principle also applies to any situation where this may occur (even outside the usage of Zoom).

We have some very exciting news on the horizon and soon I will also cover how to hole-punch custom modules that are not in the original Zoom package--so stay tuned.

So you want to GZIP your Magento Full-Page Cache?

2012-03-21T11:08:00.002-07:00

Compressing dynamic pages on the fly, to a high degree, can be resource-prohibited. Most, if not all, modern webservers have the ability to compress a page before serving (as does PHP). But for optimal compression, whatever you may save in bandwidth, you may be giving up in performance. And if performance is always in the back of your mind from running a bloated piece of architecture (if you run Magento) then it probably isn't a great idea to churn out pages on a high traffic site with maximum GZIP compression.

The Good News

Now if you are running a full-page cache for Magento that supports GZIP (like my product Zoom), not only do you have to generate a page just a single time, you really only have to compress it once! As the number hits begins to dwarf your new page generation/compression rate, the pendulum quickly swings to favor maximum compression; in a short time, the effort can save a merchant a great deal on both hardware (from using less CPU cycles) and bandwidth (from serving pages with up to 90% compression)--and further, your customers get the pages even faster! The only real cost is a little more hard-drive space to hold the copies.

This is why I opted to support GZIP compression with the Zoom Full-Page Cache. It only makes sense to give performance-concerned stores the ability to maintain a GZIPed version of their pages to serve in place of the uncompressed cached file. This feature is straight-forward, out-of-the-box, and can be employed within minutes of installation.

To give you a brief illustration, here is a listing of my cache at http://www.ezapps.ca/:

[root@ezapps magento]# ls -l var/zoom/default
-rw-rw-rw- 1 php-fpm php-fpm 18722 Mar 19 04:00 about-ezapps.html
-rw-rw-rw- 1 php-fpm php-fpm  6560 Mar 19 04:00 about-ezapps.html.gz
-rw-rw-rw- 1 php-fpm php-fpm 19061 Mar 19 04:00 privacy-policy
-rw-rw-rw- 1 php-fpm php-fpm  6608 Mar 19 04:00 privacy-policy.gz
drwxrwxrwx 3 php-fpm php-fpm  4096 Mar 19 20:58 review
-rw-rw-rw- 1 php-fpm php-fpm 22598 Mar 19 04:00 terms-and-conditions
-rw-rw-rw- 1 php-fpm php-fpm  8018 Mar 19 04:00 terms-and-conditions.gz
-rw-rw-rw- 1 php-fpm php-fpm 17350 Mar 19 16:27 zoom-free.html
-rw-rw-rw- 1 php-fpm php-fpm  5992 Mar 19 16:27 zoom-free.html.gz
-rw-rw-rw- 1 php-fpm php-fpm 39406 Mar 19 14:34 ZOOM_INDEX
-rw-rw-rw- 1 php-fpm php-fpm  7452 Mar 19 04:00 ZOOM_INDEX.gz
-rw-rw-rw- 1 php-fpm php-fpm 70736 Mar 20 02:05 zoom-magento-full-page-cache.html
-rw-rw-rw- 1 php-fpm php-fpm 15609 Mar 20 02:05 zoom-magento-full-page-cache.html.gz

My pages are small and few, but if you are running a store with thousands of products (and categories that have numerous states), there can be immense CPU and bandwidth savings here.

Using the Compressed Cache with Zoom FPC

Just speaking for my own cache, usage is simple: The Zoom PHP retriever will automatically attempt to find the GZIPed page, set the headers correctly and serve it (unless your customers are using MSIE <= 6). No further work is needed for those sending requests straight to PHP from the webserver.

But if you've taken my earlier advice and setup NGINX to serve the Magento full-page cache files directly, there is some leg work required.

Introducing GZIP static for NGINX users

If you did not yet know, there is an NGINX module called HttpGzipStatic that can make the process just as automatic as passing the request to my handler. With this module active, whenever you issue a "try_files" directive, NGINX will look for a GZIP version to send instead. In other words, when you issue:

try_files $zoomuri $zoomg $uri $uri/ @handler;

NGINX does this:

try_files $zoomuri.gz $zoomuri $zoomg.gz $zoomg $uri.gz $uri $uri/ @handler;

From what I can ascertain, the overhead is not significant.

Using GZIP static

This module is not built into many standard NGINX distributions, which means that you will need to compile NGINX by hand. This is not a difficult process:

wget http://nginx.org/download/nginx-1.0.14.tar.gz
tar -zxvf nginx-1.0.14.tar.gz
cd nginx-1.0.14

# Your compile line may differ
./configure --without-http_autoindex_module --with-http_ssl_module --without-http_geo_module --with-http_gzip_static_module --without-http_empty_gif_module
make
make install
cd ..
rm -rf nginx-1.0.14
service nginx restart

Now we have to tell NGINX to use this module inside our server configuration files. You can try a configuration simular to:

gzip  on;
gzip_disable        "MSIE [1-6]\.";
gzip_vary           on;
gzip_comp_level 2;
gzip_min_length 1000;
gzip_proxied any;
gzip_types      text/plain  text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript;
charset  utf-8;
gzip_static on;

After a server restart/reload, you should can now be serving pre-gzipped files directly from NGINX! Paired with my earlier tutorial on serving Zoom from NGINX, you can be running one of the fastest and efficient Magento stores possible.

Taking it one step further

Why stop here? We are already checking from NGINX for GZIP copies of every file, so why not give it to them? If you want to take this one step further, extend Magento's core thumbnailing libraries to produce a GZIP copy as well so your product images are zooming on over to the user (for compressed images, this won't save much). CSS and javascript and other text files are also prime subjects.

If you are interested in any custom work related to anything I've discussed in this article, please contact me.

Setting up Zoom Full-Page Cache with Magento & NGINX

2012-03-19T08:45:00.000-07:00

(Please Note: The following is not necessary for using EZAPPS Zoom, nor is NGINX even required for this FPC)

While Zoom: The Magento Full-Page Cache is quite fast on its own, I've found that NGINX can retrieve files in half the time of Zoom alone. NGINX also scales much better and has a lower memory footprint--this becomes immensely important if you need to keep your store functioning in the midst of a DDoS attack or heavy traffic spikes. Therefore, I would encourage all Zoom users to go the extra step and configure it to work with NGINX as well.

For the scope of this article, I will assume that:

The reader has a working Magento ecommerce store
This store runs uses the NGINX webserver
EZAPPS Zoom has been purchased & installed successfully
You have basic knowledge of server management
You have a development environment to test this in before using it live

If any of these assumptions are incorrect, I encourage you to catch yourself up to speed before proceeding.

Background

Zoom caches Magento pages using a very simple idea: save the pages as files to an alternative directory where the webserver (or the Zoom handler) can access them. Further, do it in such a way that it is not difficult for the server to find the files (hence, no complicated hashing).

For a quick overview of how these files are created, reference the following (assuming "example" is the store code):

/product1.html becomes var/zoom/example/product1.html
/category1/product2.html becomes var/zoom/example/category1/product2.html
/somepagename/ becomes var/zoom/example/somepagename/ZOOM_INDEX

What becomes more difficult is categories, because each view has a state attached to it--therefore, the file names must preserve this state. A category url like "cat1.html" will be cached as:

var/zoom/example/cat1.html/zget/p/1/dir/asc/limit/9/......../ZOOM_INDEX
var/zoom/example/cat1.html/zget/p/1/ZOOM_INDEX (possible symlink)

Now lets put it together

With NGINX, you can use the "try files" directive to try multiple files (they are tried in order from left to right, and if they do not exist, the request gets pushed to PHP in the typical Magento setup). In your ".conf" file, there should be a line that looks like:

try_files $uri $uri/ @handler;

We will be changing this line to:

try_files $zoomuri $zoomg $uri $uri/ @handler;

Now, outside any location directive, all we have to do is define $zoomuri and $zoomg.

set $ua '';
set $zget /zget;

# User Agent theme overrides will be cached to their own sub-directories
# if ( $http_user_agent ~ iphone ){
# set $ua '/default/default/iphone';
# }

set $mzi /ZOOM_INDEX;
set $code default; # This should match your MAGE_RUN_CODE

# Multi-language stores use different codes on the same domain. Use the 'store' cookie to detect this
# if ($cookie_store) { set $code $cookie_store; }

set $zoombase /var/zoom/$code$ua;
set $zoomuri $zoombase$uri; # This will be pages without toolbars (products, CMS, etc)
set $zoomuriget $zoomuri$zget; # Here is our initial category variable

# By default, we will always check for the first page of a potential category page
set $p 1;

# Initialize toolbar state according to Zoom cookies.
set $dir $cookie_zoom_dir;
set $limit $cookie_zoom_limit;
set $mode $cookie_zoom_mode;
set $order $cookie_zoom_order;

set $zoomg $zoomuriget;

# Initialize layered navigation variables
set $cat "";
set $price "";

# Over-ride toolbar state with GET variables (if they exist)
if ($arg_p) { set $p $arg_p; }
if ($arg_dir) { set $dir $arg_dir; }
if ($arg_limit) { set $limit $arg_limit; }
if ($arg_mode) { set $mode $arg_mode; }
if ($arg_order) { set $order $arg_order; }

# Layered navigation will be appended in alphabetical order to the final string
if ($arg_cat) { set $cat /cat/$arg_cat; }
if ($arg_price) { set $price /price/$arg_price; }

set $control &dir=$dir&limit=$limit&mode=$mode&order=$order;

# If there is no given toolbar state, look for symlink of the default state (at the given page)
if ($control = "&dir=&limit=&mode=&order=") { set $zoomg $zoomg/p/$p; }

# Otherwise, construct complete cache filename (this requires normalized URL for consistent hits)
if ($zoomg = $zoomuriget) { set $zoomg $zoomg/p/$p/dir/$dir/limit/$limit/mode/$mode/order/$order$cat$price; }

# A fix to redirect the index page to the proper file
if ($request_uri = /) { set $zoomuri $zoomuri$mzi; }

# Append the "ZOOM_INDEX" to the file category path
set $zoomg $zoomg$mzi;

# Now clean the filename of irregular characters
if ($zoomg ~ (.*)/price/([0-9]+),([0-9]+)(.*)) { set $zoomg $1/price/$2%2C$3$4; }

I realize that for many unfamiliar with NGINX, this could be intimidating. Hopefully the comments will guide you in making customizations that your own Magento installation may require.

This code belongs above "location / {" and should be sufficient to locate the proper Zoom full-page cache file that corresponds to the Magento page. If this page is not found, NGINX will seamlessly push the request to the PHP handler (and the page will be created for next time).

Using Zoom's category URL rewrite option

Zoom FPC gives you the additional option of creating clean Magento navigation URLs (but you need to block redundant content with "robots.txt"). The following URL:

http://www.example.com/cat1.html?p=1&dir=asc&limit=9&mode=grid&order=price&price=2,10

Becomes

http://www.example.com/cat1.html/zget/p/1/dir/asc/limit/9/mode/grid/order/price/price/2,10

Handling these works in much the same manner as before. Replace the two blue lines given above with the following block:

set $match "";
set $matchbool 0;

# Is the URL a rewritten category URL?
if ($request_uri ~ (.+)/zget/p/([0-9]+)(.*) ) { set $zoomg $zoombase$1$zget/p/$2; set $match $3; }

# Are we looking for a specific tool bar state?
if ($match ~ /dir/([a-z]+)/limit/([0-9]+)/mode/([a-z]+)/order/([a-z]+)(.*)) { set $zoomg $zoomg$match$mzi; set $matchbool 1; }

# Otherwise, return the default state for that page
if ($matchbool = 0) { set $zoomg $zoomg$mzi; }

# Now clean the filename of irregular characters
if ($zoomg ~ (.*)/price/([0-9]+),([0-9]+)(.*)) { set $zoomg $1/price/$2%2C$3$4; }

Simple enough? Maybe not, but by letting NGINX serve up your Zoom cache pages directly, there is little doubt that you will have one of the fastest stores around.

Finishing up

You may notice that when NGINX serves a cached page, it is downloaded directly to your browser. If you come across this situation, the following line in your location directive seems to remedy this:

default_type text/html;

Another aspect to explore is debugging and determining which pages are being served by the webserver, and which ones are being served by PHP. One simple approach, after PHP generates the cache, is to run phpinfo(); at the start of index.php and exit the program prior to Zoom's retrieval mechanism. So if you see the store page, NGINX is pulling from the cache. If you do not, there is something wrong with your configuration.

I realize that I may have taken a few of you down the rabbit-hole with this DIY course in NGINX configuration, but I strongly feel that it is important to teach my customers how to get the most of Zoom (so they can get the most of their Magento store). If you need further clarification or direct aid, please contact us.

Introducing ZOOM: The Magento Full-Page Cache

2012-03-18T06:14:00.001-07:00

EZAPPS Zoom is a feature-rich full-page cache mechanism for Magento CE, designed for webserver integration. It includes block-level "hole-punching", versatile cache controls, event & time-driven invalidation, GZIP compression, multi-currency & language support and much more. Best of all, it is now open-source and completely free.

Creating Zoom

Almost anyone who has worked with Magento has experienced that the power of this platform is unfortunately offset by its speed. While facing this dilemma a few weeks ago, a very simple thought occurred to me: what if I could save a rendered Magento page to a basic file location that a webserver can access and serve? And within a couple of minutes, I had NGINX spitting out Magento pages in thousands of a second--almost immediately, I knew I was on to something big.

But there was a long path before me on this project. First, I had to remove personalized content from these pages like cart information (aka "Hole-Punching"). Simply creating edited template files would never accomplish this competently--this approach would leave a mess for anyone implementing the cache on a custom design. So I looked beyond the template files and decided to punch at block level, targeting a plethora of personalized blocks. And when I implemented this on my custom design, I couldn't have been more pleased with the time it saved me in the long run--I never had to touch a line of code in the layout and template files.

Next, there was the matter of controlling which pages to cache and which to ignore. My first thought: why build a comprehensive full-page cache mechanism and only allow a few different modules to interact with it? So as a developer, along with standard controls for Categories/Products/Reviews/CMS pages, I included regular expression matching at both the URI and module level. This allows you to target pages for inclusion and exclusion at a very micro level, letting you cache or ignore virtually any page. I rounded this component off with time and event-driven invalidation so store owners can focus on running their stores instead of managing more infrastructure.

On our last pass, we made sure it can play well with multi-currency and multi-language sites, and handle almost any situation that we can predict. As a bonus, to reduce your miss rates we've included an optional feature to clean up and normalize your category URLs. To top it off, we've also added the ability to GZIP the cache, which can save you a dramatic amount of bandwidth that you don't have to pay for anymore. Let us know if you think there is anything we missed!

In the end, there was Zoom

EZAPPS is proud of our first commercial extension, and we are very happy with how it turned out. And when we deployed Apache Bench this weekend, we just became that much happier. We feel that the results speak for themselves:

Remember as a new developer, we are going to live or die based on how many people can hear about us and use our products--especially at these prices. So please, if you wish to see more great extensions from us (and we have some major projects in the works), tell your friends' friends. Let's continue to build this community together and solidify Magento's place in the ecommerce world.

For a limited time, the first 50 customers who order from us using coupon "ZOOM" will get $5 dollars off!

Into the fray

2012-03-08T12:01:00.001-08:00

Here at EZAPPS, we’ve finally decided that it is time to toss our hat in the ring and become a full-fledged software provider for the Magento E-commerce Platform. This most certainly isn’t a hasty decision, as we have been designing our core offerings for almost a year now—though, our first public product is not part of that initial plan.

Frankly, it was 5 quick lines of PHP code and a simple idea that woke us from our methodical slumber two weeks ago. And 3000 lines later, this concept morphed into a product that I feel transforms Magento towards a goal of dominating e-commerce platforms (and once we have an actual store put together, you get to find out what it is).

But the best news of all is that we are practically giving it away—and not just for the publicity (but be sure to tell your friends anyway). First off, I want EZAPPS to become synonymous with “Magento”. In one way or another, every Magento store should know us by name and use at least one of our offerings—we are willing to do whatever we must to make this possible. Secondly, it took us only a week to finish the programming, working a few hours each night. Considering this, and the fact that this extension isn’t part of our “big plans”, why not be reasonable?

And lastly, we want the Magento community to strengthen and grow exponentially, and this involves helping new merchants unlock the potential of this platform without immediate and immense out-of-pocket costs. I’ve too often seen people hooked on the elegance of Magento Community, but shortly faced with the decision to turn to Enterprise or turn away entirely. We have an incredible e-commerce/cms platform here, and I feel that it is time to fill in the holes and nurture the community that has supported it thus far.

We head into this field with the mindset of offering affordable, transformative extensions that will grow the Magento community along with your sales. We know big ideas can be easy—and they are right around the corner.

If there is any lacking functionality that you feel that Magento just can’t live without, drop us a line so we will see what we can do.