Memoir of a Magento developer

Zend Queue with Magento

noreply@blogger.com (Anonymous) — Wed, 01 Aug 2012 16:53:00 +0000

Zend Queue with Magento

By using Zend Queue with magento we can create an event driven asynchronus integration system.

Just think about how much work you will offload from Magento when integrating with other systems and Magento itself.

To keep things simple, let's pretend that you send emails to your customers everytime a new product gets added. Usually these products get added during the day which also happens to be when your customers are most active buying in the site. An alert about a new product is very important but you don't want to bug down your email server. That's where Zend Queue comes to the rescue. In this example I am using mysql to store the queue. If you follow the same example remember to create the tables first, these can be found under: lib/Zend/Queue/Adapter/Db/mysql.sql

Also it will make more sense to use MemcacheQ or Apache ActiveMQ to offload mysql.

Here's an example:

In your observer class:

public function sendEmails($observer){
 Mage::helper('OfflineSync')->setEmailsOffline($observer->getData('object_container'));

}

Helper class

/*
* To change this template, choose Tools | Templates
* and open the template in the editor.
 */

/**
* Description of Data
 *
* @author letas
 */
class ZendQueue_OfflineSync_Helper_Data extends Mage_Core_Helper_Data {

    protected $_name = "general";
    protected $_registry = array();
    protected $_queue = null;

    protected function getQueue() {
        if (!isset($this->_registry[$this->_name])) {
            $db = simplexml_load_file('app' . DS . 'etc' . DS . 'local.xml');
            $db = $db->global->resources->default_setup->connection;
            $queueOptions = array(
                Zend_Queue::NAME => "{$this->_name}",
                'driverOptions' => array(
                    'host' => $db->host,
                    'port' => $db->port,
                    'username' => $db->username,
                    'password' => $db->password,
                    'dbname' => $db->dbname,
                    'type' => 'pdo_mysql',
                    Zend_Queue::TIMEOUT => 1,
                    Zend_Queue::VISIBILITY_TIMEOUT => 1
                )
            );
            //// Create a database queue
            $this->_registry[$this->_name] = new Zend_Queue('Db', $queueOptions);
        }
        return $this->_registry[$this->_name];
    }

    public function getEmailsOffline() {
        try {
            $this->_name = "offline_email";
            //cache here / singlenton
            $this->_queue = $this->getQueue();
            foreach ($this->_queue->receive(); as $i => $message) {
               //send the real mail now

    //delete this message
   $this->_queue->deleteMessage($message);
            }
        } catch (Exception $e) {
            Mage::logException($e);
            return -1;
        }
        return 1;
    }

    public function saveEmailsOffline($emails) {
        if (isset($emails)) {
            if (is_array($emails) || is_object($emails)) {
                $emails = serialize($emails);
            }
            $this->_name = "offline_email";
            //cache here / singlenton
            $this->getQueue()->send($emails);
        }
        return $this;
    }

}

Now you can also configure a crontab in your module config.xml, your code may look like this:

 Mage::helper('OfflineSync')->getEmailsOffline();

And you are done!!! Pretty simple right.

The possibilities are endless. Orders / Customer exports. Everything can be queued up.

Cloning magento modules

noreply@blogger.com (Anonymous) — Wed, 01 Aug 2012 16:37:00 +0000

Cloning magento modules will never this easy again.

Today I wanted (read was forced) to create a module based on one of Magento's core. Literally I needed to clone some of the Magento modules and while the copy and paste is simple stuff, going class by class and file by file renaming and configuring files is not joke.

So why do I need to clone it instead of just doing the usual OOP stuff Magento is so good at? Simply because the functionality is really different in most files and settings. So creating a payment method or paygate or giftcard module is easier when you have an skeleton to work with. Think about it, how different is paying with PayPal from Google Checkout? Functionality wise not that much, essentially they do the same thing but implement it differently.

I remember someone (I am talking to you Alan Storm) saying: "It's programming - come up with a canonical way of doing it, put it in a function and forget it about it".

So of course I decided that cloning magento modules was never going to be a hard task again.

So from today onward cloning magento modules is going to be pretty simple (a least to me).
Be warned there are some hard-coded paths here and little to none validation. Use at your own risk and needless to say don't use it on a production site. The script doesn't do about the app/etc/modules/module_name.xml so you have to do manually.

Copy the following code into a file and save as clone, no extensions needed:

#!/bin/bash

ORIGINAL_NAME=$1;
NEW_NAME=$2;
NAMESPACE="MyCompany";

#copy the module
`cp -R app/code/core/Mage/$ORIGINAL_NAME app/code/local/$NAMESPACE/$NEW_NAME`
#lowercase both the original name and new name
lowercase_orig=`echo $ORIGINAL_NAME | tr '[A-Z]' '[a-z]'`
lowercase_new=`echo $NEW_NAME | tr '[A-Z]' '[a-z]'`
#Rename the class declaration and stuff
`grep -lr "$ORIGINAL_NAME" "app/code/local/$NAMESPACE/$NEW_NAME/" | xargs -d "\n" sed -i "s/$ORIGINAL_NAME/$NEW_NAME/g"`
`grep -lr "Mage" "app/code/local/$NAMESPACE/$NEW_NAME/" | xargs -d "\n" sed -i "s/Mage/$NAMESPACE/g"`
#rename the  the shorcuts
`grep -lr "$lowercase_orig" "app/code/local/$NAMESPACE/$NEW_NAME/" | xargs -d "\n" sed -i "s/$lowercase_orig/$lowercase_new/g"`
#rename the files
`find "app/code/local/$NAMESPACE/$NEW_NAME" -name "*$ORIGINAL_NAME*" -exec rename "s/$ORIGINAL_NAME/$NEW_NAME/g" {} \;`

The trick here is to use find and grep to find the old module name and class shorcuts and use sed and rename to change it to the new one. Notice that tr is used to lowercase the old and new name because we are doing case sensitive searches. Also rename and sed are only replacing the portion of the text they find.

First let's make sure we can execute the file

chmod +x clone (only need to do this once)

Then in the root of your Magento installation do:
./clone module1 module2

And then you will have it a brand new module cloned from Core to use as your starting point.

Full Page cache with nginx and memcache

noreply@blogger.com (Anonymous) — Wed, 01 Aug 2012 16:32:00 +0000

Full Page cache with nginx and memcache

Since the cool kids at Google, Microsoft and Amazon researched how performance and scalability affect conversion rates, page load time has become the topic of every eCommerce store.

Magento was once a resource hog that consumated everything available to it and you had to be a magician to pull off some awesome benchmarks without using any reverse proxy or full page cache mechanism. Creating a full page cache with nginx and memcache is really simple (right after hours of research).

Words of warning first:

Don't use this instead of varnish or Magento's full page caching. This implemenation of full page cache is very simple, heck it will be even troublesome to clean the cache consistently because guess what, there is no holepunching but you could enhance the configuration file to read cookies and serve directly from the backend server instead.

Another problem is that you'll need to ensure that a TwoLevel caching is used to be able to flush specific urls.

Now that is out of the way, let's focus on the matter at hand.

I have tried this configuration file with both Magento enterprise and community and also with WordPress.

#memcache servers load balanced
upstream memcached {
        server     server_ip_1:11211 weight=5 max_fails=3  fail_timeout=30s;
        server     server_ip_2:11211 weight=3 max_fails=3  fail_timeout=30s;
        server    server_ip_3:11211;
 keepalive 1024 single;
}
#fastcgi - little load balancer
upstream phpbackend{
 server     server_ip_1:9000 weight=5 max_fails=5  fail_timeout=30s;
        server     server_ip_2:9000 weight=3 max_fails=3  fail_timeout=30s;
        server    server_ip_3:9000;
}
server {
    listen   80; ## listen for ipv4; this line is default and implied
    root /var/www/vhosts/kingletas.dev/www;
    server_name kingletas.dev;
    index index.php index.html index.htm;

    client_body_timeout  1460;
    client_header_timeout 1460;
    send_timeout 1460;
    client_max_body_size 10m;
    keepalive_timeout 1300;

    location /app/                { deny all; }
    location /includes/           { deny all; }
    location /lib/                { deny all; }
    location /media/downloadable/ { deny all; }
    location /pkginfo/            { deny all; }
    location /report/config.xml   { deny all; }
    location /var/                { deny all; }

   location ~* \.(jpg|png|gif|css|js|swf|flv|ico)$ {
     expires max;
     tcp_nodelay off;
     tcp_nopush on;
    }
    location / {
  
        try_files $uri $uri/ @handler;
        expires 30d;
    }
   location @handler {
 rewrite / /index.php;
    }

    location ~ \.php$ {
        if (!-e $request_filename) { 
            rewrite / /index.php last; 
        }  
        expires        off; ## Do not cache dynamic content
        default_type       text/html; charset utf-8;
        if ($request_method = GET) { # I know if statements are evil but don't know how else to do this
            set $memcached_key $request_uri; Catalog request modal 
            memcached_pass     memcached;
            error_page         404 502 = @cache_miss;
            add_header x-header-memcached true;
  }
  if ($request_method != GET) {
   fastcgi_pass phpbackend;
  }
    }
    location @cache_miss {
        # are we using a reverse proxy?
        proxy_set_header  X-Real-IP  $remote_addr;
        proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_max_temp_file_size 0;
        
        #configure fastcgi
        fastcgi_pass 127.0.0.1:9000;
        fastcgi_send_timeout  5m;
        fastcgi_read_timeout 5m;
        fastcgi_connect_timeout 5m;
        fastcgi_buffer_size 256k;
        fastcgi_buffers 4 512k;
        fastcgi_busy_buffers_size 768k;
        fastcgi_param GEOIP_COUNTRY_CODE $geoip_country_code; 
        fastcgi_param GEOIP_COUNTRY_NAME $geoip_country_name; 
        fastcgi_param  PHP_VALUE "memory_limit = 32M";
        fastcgi_param  PHP_VALUE "max_execution_time = 18000";
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
        include fastcgi_params;
    }
    location ~ /\. {
  deny all;
 }
}
#if you want to make it even better your own cdn
#server {
#      listen 80; 
#      server_name media.kingletas.dev;
#      root /var/www/vhosts/kingletas.dev/www;
#}
#server {
#      listen 80; 
#      server_name css.kingletas.dev;
#      root /var/www/vhosts/kingletas.dev/www;
#}
#server {
#      listen 80; 
#      server_name js.kingletas.dev;
#      root /var/www/vhosts/kingletas.dev/www;
#}

One major topic to remember is that nginx will try to read from memory not write to it. In other words you still need to write the contents to memcache. For WordPress this is what I did in the index.php

/**
* Front to the WordPress application. This file doesn't do anything, but loads
* wp-blog-header.php which does and tells WordPress to load the theme.
 *
* @package WordPress
 */

/**
* Tells WordPress to load the WordPress theme and output it.
 *
* @var bool
 */
ini_set("memcache.compress_threshold",4294967296); //2^32
ob_start();

define('WP_USE_THEMES', true);

/** Loads the WordPress Environment and Template */
require('./wp-blog-header.php');

$buffer = ob_get_contents();

ob_end_clean();

$memcache_obj = memcache_connect("localhost", 11211);
memcache_add($memcache_obj,$_SERVER['REQUEST_URI'],$buffer,0);

echo $buffer;

Notice that I had to change the memcache.compress_threshold setting to HUGE number, that is because memcache will ignore the no compress setting when this threshold is exceeded and compress the content, while this is good and dandy the results in the browser are not.

So there you have it an easy way to implement full page caching with nginx and memcache for WordPress or Magento and the rest of the framework world.

Connect to a wireless network through the command line

noreply@blogger.com (letas) — Sat, 28 Jul 2012 21:21:00 +0000

Sometimes you decide that you don't want to use any GUI to connect to your wireless network, especially when it doesn't seem to work.

One of the main benefits of Linux and any other Unix based operating system is its ability to empower users to do things on their own.
So tonight when the latest Ubuntu updates broke my laptop's wireless, I decided to write a little script to fix it.

You will need to know the following:

ifconfig
iwlist
iwconfig
dhclient

Find your network interface, unless you know it (my laptop is always wlan0)

ifconfig

ifconfig NETWORK_ID up

Find the network you want to connect to:

iwlist network_interface scan

Let' configure your network card

iwconfig NETWORK_ID essi "NAME" key s:password

iwconfig network_interface essi "NAME" key hex_password

Now all we need is to get the IP from the DHCP server:

dhclient network_interface

You could have it all in a script:

#!/bin/bash
ifconfig wlan0 up
iwconfig wlan0 essi "$1" key s:$2
dhclient wlan0

Save it as wireless_up and then chmod a+x wireless_up.
Now you just need to either do ./wireless_up network_id my_password or bash wireless_up network_id my_password in the same directory. You could also move the script to a global location or add the directory where you have it as part of your local path. The choice is yours.

Dirt cheap way to make magento fly

noreply@blogger.com (Anonymous) — Fri, 27 Jul 2012 20:47:00 +0000

There are several ways to accomplish the same task and some are more elegant than others.
This one makes any Magento community edition fly and is cheap and quick (read no Magento compliant - because it modifies the templates)
In other words while this trick can help you boost your Magento store server response I am not too proud of it.

Why is that a big deal? I am not a huge fan or changing templates, even if they are "allowed".
But let's forget about that and get down to business
Your actual cache helper class

class Performance_CacheHelper_Helper_Data extends Mage_Core_Helper_Data {

    //###########################################
    //Cache related functions and variables
    //###########################################
    /**
     * cache key
     * @var string|mixed 
     */
    protected $key;

    /**
     *
     * @var Mage_Core_Model_Cache 
     */
    protected $_cache;

    /**
     * data container
     * @var string|mixed 
     */
    protected $data;
    protected $tags = array(__CLASS__);

    public function canUseCache() {
        //verify if the cache is enabled
        return Mage::app()->useCache('block_html');
    }

    /**
     * Gets the cache object
     * @return Mage_Core_Model_Cache 
     */
    protected function _getCacheObject() {
        if (!$this->_cache) {
            $this->_cache = Mage::app()->getCache();
        }
        return $this->_cache;
    }

    public function getData() {
        return $this->data;
    }

    public function setData($data) {
        $this->data = $data;

        return $this;
    }

    public function getKey() {
        return $this->key;
    }

    public function setKey($key) {
        $this->key = $key;
        return $this;
    }

    /**
     * saves data in a serialize format to cache under the name of this class if the cache can be used
     * @return Performance_CacheHelper_Helper_Data 
     */
    public function saveDataInCache() {
        if ($this->canUseCache()) {
            $cookie = Mage::getModel('core/cookie');
            $this->_getCacheObject()->save(
                    serialize($this->data), $this->key, $this->tags, $cookie->getLifetime());
        }
        return $this;
    }

    /**
     *
     * @param string $key
     * @return mixed
     */
    public function getDataFromCache($key) {
        $this->key = $key;
        return $this->_getCachedData();
    }

    /**
     * gets the data saved in cache, if it finds it then it unserializes itF
     * @param string|mixed $key
     * @return bool | string
     */
    protected function _getCachedData($key = null) {
        if ($key !== null) {
            $this->key = $key;
        }
        /**
         * clear the data variable 
         */
        $this->data = false;
        //ensure cache can be used
        if ($data = $this->_getCacheObject()->load($this->key)) {
            $this->data = unserialize($data);
        }
        return $this->data;
    }

}

In your current template:

                
         //this code has been modified so the highligthing works right                   
        $helper = Mage::helper('cachehelper');
        $cachingEnabled = array('catalog');
        $key = md5(Mage::helper('core/url')->getCurrentUrl());
        $currentModule = Mage::app()->getRequest()->getModuleName();
        if (in_array($currentModule, $cachingEnabled)){
         if ($content = $helper->getDataFromCache($key)){
         //nothing do $content now has the data
         }else{
             $content = $this->getChildHtml('content');
             $helper->setKey($key)->setData($content)->saveDataInCache();
            
        }else{
         $content = $this->getChildHtml('content');
        }

siege -t1m -c5 -b -d -i http://mage.dev/apparel
Examples with File Based cache
Without the "dirt cheap trick"

Transactions:	324 hits
Availability:	100.00 %
Elapsed time:	59.94 secs
Data transferred:	2.13 MB
Response time:	0.92 secs
Transaction rate:	5.41 trans/sec
Throughput:	0.04 MB/sec
Concurrency:	4.95
Successful transactions:	324
Failed transactions:	0
Longest transaction:	2.32
Shortest transaction:	0.63

With the "dirt cheap trick"

Transactions:	475 hits
Availability:	100.00 %
Elapsed time:	59.56 secs
Data transferred:	3.14 MB
Response time:	0.62 secs
Transaction rate:	7.98 trans/sec
Throughput:	0.05 MB/sec
Concurrency:	4.97
Successful transactions:	475
Failed transactions:	0
Longest transaction:	1.45
Shortest transaction:	0.37

Examples with memcache
Without the "dirt cheap trick"

Transactions:	329 hits
Availability:	100.00 %
Elapsed time:	59.65 secs
Data transferred:	2.17 MB
Response time:	0.90 secs
Transaction rate:	5.52 trans/sec
Throughput:	0.04 MB/sec
Concurrency:	4.95
Successful transactions:	329
Failed transactions:	0
Longest transaction:	1.60
Shortest transaction:	0.68

With the "dirt cheap trick"

Transactions:	507 hits
Availability:	100.00 %
Elapsed time:	59.47 secs
Data transferred:	3.35 MB
Response time:	0.58 secs
Transaction rate:	8.53 trans/sec
Throughput:	0.06 MB/sec
Concurrency:	4.97
Successful transactions:	507
Failed transactions:	0
Longest transaction:	1.91
Shortest transaction:	0.40

Pretty awesome right? Don't think siege is a good testing tool and this is not a good indication? What about jmeter?
Here is the test plan I used:

      
         
            
               
               false
               false
               
                  
               
               
            
            
               
                  
                     false
                     1
                  
                  10
                  1
                  1340804136000
                  1340804136000
                  false
                  continue
                  
                  
               
               
                  
                     
                        
                     
                     mage.dev
                     
                     
                     
                     
                     
                     /apparel
                     GET
                     false
                     true
                     true
                     false
                     
                     
                     
                     false
                     
                  
                  
                     
                        
                           
                              Accept-Language
                              en-us,en;q=0.5
                           
                           
                              Accept
                              text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
                           
                           
                              User-Agent
                              Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:13.0) Gecko/20100101 Firefox/13.0.1
                           
                           
                              Accept-Encoding
                              gzip, deflate
                           
                        
                     
                     
                  
                  
                     
                        
                     
                     mage.dev
                     
                     
                     
                     
                     
                     /furniture/living-room
                     GET
                     false
                     true
                     true
                     false
                     
                     
                     
                     false
                     
                  
                  
                     
                        
                           
                              Accept-Language
                              en-us,en;q=0.5
                           
                           
                              Accept
                              text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
                           
                           
                              User-Agent
                              Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:13.0) Gecko/20100101 Firefox/13.0.1
                           
                           
                              Accept-Encoding
                              gzip, deflate
                           
                        
                     
                     
                  
                  
                     
                        
                     
                     mage.dev
                     
                     
                     
                     
                     
                     /furniture/bedroom
                     GET
                     false
                     true
                     true
                     false
                     
                     
                     
                     false
                     
                  
                  
                     
                        
                           
                              Accept-Language
                              en-us,en;q=0.5
                           
                           
                              Accept
                              text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
                           
                           
                              User-Agent
                              Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:13.0) Gecko/20100101 Firefox/13.0.1
                           
                           
                              Accept-Encoding
                              gzip, deflate
                           
                        
                     
                     
                  
               
               
                  false
                  
                     saveConfig
                     
                        true
                        true
                        true
                        true
                        true
                        true
                        true
                        true
                        true
                        false
                        true
                        true
                        false
                        false
                        true
                        false
                        false
                        false
                        false
                        false
                        0
                        true

Here are the results, sorry about the formatting:

Without the "dirt cheap trick"

sampler_label	aggregate_report_count	average	aggregate_report_min	aggregate_report_max	aggregate_report_stddev	aggregate_report_error%	aggregate_report_rate	aggregate_report_bandwidth	average_bytes
/apparel	10	1710	1274	2180	290.2423125597	0	4.5724737083	199.8358553384	44753
/furniture/living-room	10	896	618	1272	254.8664159908	0	7.5757575758	227.1173650568	30699
/furniture/bedroom	10	747	565	953	137.6	0	7.3637702504	204.8623895434	28488
TOTAL	30	1117	565	2180	484.8805133455	0	7.2904009721	246.6680589307	34646.6666666667

With the "dirt cheap trick"

sampler_label	aggregate_report_count	average	aggregate_report_min	aggregate_report_max	aggregate_report_stddev	aggregate_report_error%	aggregate_report_rate	aggregate_report_bandwidth	average_bytes
/apparel	10	675	434	904	147.3737086457	0	6.1199510404	268.3156269125	44895
/furniture/living-room	10	741	581	950	107.7506844526	0	5.4436581383	164.0678160724	30862.6
/furniture/bedroom	10	631	389	827	157.5221889132	0	7.3691967576	206.1388863301	28644.4
TOTAL	30	682	389	950	146.3524057427	0	10.8577633008	369.0013685306	34800.6666666667

As you can see this is trick helps a lot but it doesn't compete with Magento's FPC or Varnish and it still needs lots of polishing. The good think is that is doesn't require holepunching or lots of Magento knowledge. Now a few pointers... this is not the full implementation, as this code doesn't work with load balancers and there is not a way to clear it here. But implementing both is trivial.

PHP Offloading... Nginx helps so much!

noreply@blogger.com (Anonymous) — Thu, 26 Jul 2012 22:06:00 +0000

So let's continue our attempt to improve Magento's performance and this time around let's focus on the web server itself.
There are a bunch of web servers out there but the most known are, a least to me: Apache, Nginx and Lighttpd
My favorite, and of course I am biased here, is Nginx and it approach to serving content. As you may all know Nginx is an event-based web server and the main advantage of this asynchronous approach is scalability.
Because at the end of the day what we are going to have is a complete static page (after it has been cached once) being served from the web server.

Apache is out of the question because nginx is faster at serving static files and consumes much less memory for concurrent requests. As Nginx is event-based it doesn't need to spawn new processes or threads for each request, so its memory usage is very low and that is the key.

We could use lighttpd (it's even more compatible with apache) but the cpu and memory fingerprint is bigger than nginx and we are all about performance aren't we?

So here are my recommended configurations for both Nginx and fast-cgi, of course suggestions are welcome:

server {
    listen 80 default;
    listen 443 default ssl;
    ssl_certificate /etc/pki/tls/certs/2014-www.example.com.pem;
    ssl_certificate_key /etc/pki/tls/certs/2014-www.example.com.pem;
    server_name www.example.com example.com; ## Domain is here twice so server_name_in_redirect will favour the www
    root /var/www/vhosts/example.com;
 
    location / {
        index index.html index.php;
        try_files $uri $uri/ @handler; 
        expires 30d; ## Assume all files are cachable
    }
 
    ## These locations would be hidden by .htaccess normally
    location /app/                { deny all; }
    location /includes/           { deny all; }
    location /lib/                { deny all; }
    location /media/downloadable/ { deny all; }
    location /pkginfo/            { deny all; }
    location /report/config.xml   { deny all; }
    location /var/                { deny all; }
 
#    location /var/export/ { 
#        auth_basic           "Restricted"; ## Message shown in login window
#        auth_basic_user_file htpasswd; ## See /etc/nginx/htpassword
#        autoindex            on;
#    }
 
    location  /. { ## Disable .htaccess and other hidden files
        return 404;
    }
 
    location @handler { ## Magento uses a common front handler
        rewrite / /index.php;
    }
 
    location ~ .php/ { ## Forward paths like /js/index.php/x.js to relevant handler
        rewrite ^(.*.php)/ $1 last;
    }
 
    location ~ .php$ { ## Execute PHP scripts
        if (!-e $request_filename) { rewrite / /index.php last; } ## Catch 404s that try_files miss
 
        expires        off; ## Do not cache dynamic content
 fastcgi_read_timeout 120;
 fastcgi_buffer_size 128k;
 fastcgi_buffers 4 256k;
 fastcgi_busy_buffers_size 256k;
        fastcgi_pass   unix:/var/run/php-fpm/php-fpm.sock;
        fastcgi_param  HTTPS $fastcgi_https;
        fastcgi_param  PHP_VALUE "memory_limit = 341M";
 fastcgi_param  PHP_VALUE "max_execution_time = 18000";
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
        fastcgi_param  MAGE_RUN_CODE default; ## Store code is defined in administration > Configuration > Manage Stores
        fastcgi_param  MAGE_RUN_TYPE store;
        #fastcgi_param  MAGE_IS_DEVELOPER_MODE true; //enable developer mode?
        include        fastcgi_params; ## See /etc/nginx/fastcgi_params
    }
}

; Start a new pool named 'www'.
[www]

listen = /var/run/php-fpm/php-fpm.sock
user = apache
group = apache
pm = dynamic
pm.max_children = 50
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 50
pm.max_requests = 500

pm.status_path = /phpfpm-status
request_terminate_timeout = 5m
request_slowlog_timeout = 2m
slowlog = /var/log/php-fpm/www-slow.log

php_admin_value[error_log] = /var/log/php-fpm/www-error.log
php_admin_flag[log_errors] = on

Contents of /etc/nginx/fastcgi_params


fastcgi_param  QUERY_STRING       $query_string;
fastcgi_param  REQUEST_METHOD     $request_method;
fastcgi_param  CONTENT_TYPE       $content_type;
fastcgi_param  CONTENT_LENGTH     $content_length;

fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
fastcgi_param  REQUEST_URI        $request_uri;
fastcgi_param  DOCUMENT_URI       $document_uri;
fastcgi_param  DOCUMENT_ROOT      $document_root;
fastcgi_param  SERVER_PROTOCOL    $server_protocol;
fastcgi_param  HTTPS              $https if_not_empty;

fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;

fastcgi_param  REMOTE_ADDR        $remote_addr;
fastcgi_param  REMOTE_PORT        $remote_port;
fastcgi_param  SERVER_ADDR        $server_addr;
fastcgi_param  SERVER_PORT        $server_port;
fastcgi_param  SERVER_NAME        $server_name;

# PHP only, required if PHP was built with --enable-force-cgi-redirect
fastcgi_param  REDIRECT_STATUS    200;

PHP offloading for better performance

noreply@blogger.com (Anonymous) — Wed, 25 Jul 2012 02:19:00 +0000

So you recently launched your PHP application (read Magento) just to realize that it is really slow even though you have state of the art hardware, software and it is configured to kick ass. You decided to install varnish to help full page caching and your CDN, which it only has gzipped merged and minified assets, to serve content faster now your site seems to be flying but New Relic still reveals so many bottlenecks specially for the fillers you needed to keep some dynamic content rocking.

You decide to run promotion that would bring 20K plus customers and decided to add more servers, virtual machines and the cloud can be so awesome. Soon to realize that your site is now really slow specially when checking out and some time later your site is DOWN!!! WHY!?!!!

Because your reverse proxy is not serving all of the customers let alone is not serving the whole content from cache as it has to fetch some from the server, more on this later. So believe it or not, your server is still getting hammered and might be even worse than before.

Reverse proxies are really awesome but if applied to a inefficient application, it will come back to hunt you and be your worst nightmare.

Caching is supposed to be good but nowadays it has become the best way to hide all of the snafus and foobars in the application.

So what is the problem? For once almost everyone does development one way through the programming language they know and rarely, if any, venture or dare to think of a possible world outside of that realm. Aside from the copy paste dogma of course.
One problem is that most people tend to try to keep the dynamic content in the page through a hybrid combination of fresh content (retrieved from the server) and cached content. Most caching systems have developed sophisticated algorithm to make it work which adds a new layer of complexity.

Let's go back to your PHP application, one of the areas that usually needs to be hole-punched is the header because it contains the cart widget, the account (log-in or out) and wishlist links. But do we really need to hole-punched it? Nop!

PHP offloading is a very useful technique to improve performance, it refers to efficently distribute the workload among all the servers or other programming languages in the sytem. Also known as Operation Driven Development (ODD) where the type of operation dictates in which server or programming language the task it will take place.

So let's take the cart widget and see how we could do it without having to even think about hole-punching:

Every time an operation that involves the cart object takes place, this one gets saved. This means when you remove, add or update an item in your shopping cart you get this event checkout_cart_save_after so why don't we create a cookie that contains off all the items information?

Snippets below, I assume you do Magento development:

In your module etc/config.xml add:

 
        
            
                
                    
                        phpoffloading/observer
                        cartUpdate
                    
                
            
        
        
            
                
                    phpoffloading.xml

Replace phpoffloading with the name of your model class.
Now in your observer add:

    /**
     * Process all the cart / quote updates to ensure we update the cookie correctly
     * @param type $observer 
     */
    public function cartUpdate($observer) {
        // using singleton instead of the event to use the same function for different events
        $cart = Mage::getSingleton('checkout/cart');
        $quote = $cart->getQuote();

        /**
         * @var $totals needed for the subtotal 
         */
        $totals = $quote->getTotals();
        /**
         * @var $checkouthelper used for translation and price formatting 
         */
        $checkoutHelper = Mage::helper('checkout');
        $items = (int) $cart->getSummaryQty();
        if ($items > 0) {
            $cartInfo = array(
                'subtotal' => $checkoutHelper->formatPrice($totals['subtotal']->getValue()),
                'items' => $items,
                'itemsmsg' => $items == 1 ? "item" : "items",
                'viewcart' => $checkoutHelper->__('View Cart')
            );
            $cartInfo = json_encode($cartInfo);
             $this->setCookie('cart_cookie', $cartInfo);
        } else {
            $this->deleteCartCookie();
        }
        return $this;
    }

In your template:








            


You have no items in your cart.

You can have this javascript either on the same template or add it as an external file:

Every time a page gets loaded now instead of going to the server the content will remain in the browser and you don't need to request content from the server again. This is a very small change how much improvement do I get? Say you go from 750ms to 2ms in the server response time what would you say? This is because varnish 1-) doesn't need to understand what ESI is and 2-) all of the content is served from cache. This doesn't apply solely to varnish but to Magento's own full page caching solution.

This approach works fine for: wishlist, logged in/out but it doesn't work quite well with recently viewed items. For that we'll do a follow up article.

FireFox vs Virtualized environments

noreply@blogger.com (Anonymous) — Mon, 23 Jul 2012 17:55:00 +0000

Firefox rocks no doubt about it... but using more memory than virtualbox just to run magento is not cool at all!

This is why I really dislike using FireFox and love using SwiftFox. How come Firefox is taking 2GB of ram, that is as much as my virtualized Ubuntu Desktop and more than my virtualized ubuntu server. In other words with that much memory I could be running a few more computers.

Swiftfox is a highly optimized firefox which aims to be lightweight while still being robust, powerful and most importantly, compatible with all of the FifeFox addons.

Restore Ubuntu to the default settings without re-installing it

noreply@blogger.com (Anonymous) — Thu, 19 Jul 2012 05:55:00 +0000

As a web developer (read app freak) I am always installing new IDEs, media players and libraries to try out. This madness cycle can definitely take a toll on your computer and slow it down to the point of no return.

Usually that is when you'd decide to format the computer and start anew. So when my wireless network was experiencing some hiccups and my computer will crash every 2 seconds for no reason or worse when it started using all of the swap memory I knew it was time to wipe it clean. Then I look at my home folder and I have over 300GB of data and I am kind of lazy. That is when I think to myself, no problem the home folder is in its own partition, right? Nope. It is not *BUMMER*

I don't want to re-install and reconfigure all of the programs I use but don't want to restart my computer every hour or so - that is when I remembered that some bash commands could save my day.

Currently I am using ubuntu 12.04 and I want to restore it to be the same it was when I first installed it, sure no problem let's start by getting two simple files from the ISO file (you still have the iso file right?):

casper/filesystem.manifest
casper/filesystem.manifest-remove

or just casper/filesystem.manifest-desktop for *buntu 11.10 or earlier
Now let's combine these 2 files:

comm -3 <(cat filesystem.manifest | awk '{print $1}' | sort) <(cat filesystem.manifest-remove | sort) > default.txt

If you have Ubuntu 11.10 or earlier then to get the default.txt file do:

cat filesystem.manifest-desktop | awk '{print $1}' | sort > default.txt

default.txt contains a sorted list of all the programs our default installation should include. Now let's get the software currently installed in our computer:

dpkg --get-selections | awk '{print $1}' | sort > currently_installed.txt

If we diff the default software vs what is currently installed, we can get what was added:

 diff -u default.txt currently_installed.txt | grep "^+[^+]" | cut -c 2- > additions.txt

or what was removed:

diff -u default.txt currently_installed.txt | grep "^-[^-]" | cut -c 2- > removed.txt

Now if we want to remove all of the additions, including the configuration files:

sudo apt-get purge $(cat additions.txt)

To add back the files we removed:

sudo apt-get install $(cat removed.txt)

Remove the configuration files:

rm -rf .gnome .gnome2 .gconf .gconfd .metacity

That should bring your current ubuntu based to the default state (I believe it should work with every Linux Distro, but the judge is still out there).

This is a bit radical though, before removing any files make sure that you only remove anything that you don't want to keep. And just in case, make a backup of your files just in case.

Varnish implementation hints and good to know

noreply@blogger.com (Anonymous) — Tue, 17 Jul 2012 04:30:00 +0000

Sharing knowledge is always a good idea. Lately I have been doing lots of integration with Varnish and Magento with different clients and it will be interesting to find out if you have any ideas or suggestions (read improvements) to make it better and ultimately help the community even more:

Setting up varnish in switch mode:

In my home directory I always have a bash aliases (vim ~/.bash_aliases) file where I add the following iptable rules:

alias ipforward='sysctl net.ipv4.ip_forward=1'
alias varnishon='iptables -t nat  -A PREROUTING -i eth0 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8888'
alias varnishmasq='iptables -t nat -A POSTROUTING -j MASQUERADE'
alias varnishoff='iptables -t nat -D PREROUTING -i eth0 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8888'
alias varnishstatus='iptables -L -t nat | grep -q 8888; if [ "test$?" = "test0" ]; then echo "Varnish On"; else echo "Varnish Off"; fi'

This allows me to switch varnish on and off without changing the existing architecture. The best thing about this is that if something were to go wrong I can immediately turn varnish off and test the web server alone. Also internal routes won't go through varnish which let's me test side by side the pages and speeds.

Setting up a varnish on flag:

When Implementing ESI, always set a variable to tell you whether varnish is on or not. For instance on my varnish configuration file (/etc/varnish/default.vcl), at the very end I have:

   
/**
   * Set a flag for web store to check if varnish is on or not
   * Very important when doing switch on / off installations with ESI  
  **/

 set req.http.X-Varnish-On = 1;

    if (isset($_SERVER['HTTP_X_VARNISH_ON'])): 
           
    else: 
           getNormalContent() ?>
   endif;

I have it a the very because, all of the whitelist and passes have already been checked – so I am assure that the only pages left are “varnished” pages.

Remove all the cookies

The only cookie value Magento requires is the frontend cookie:

//check that the cookie is still set
if (req.http.Cookie){

 set req.http.X-Cookie = ";" +req.http.Cookie;
 set req.http.X-Cookie = regsuball(req.http.X-Cookie, "; +", ";");
 set req.http.X-Cookie = regsuball(req.http.X-Cookie, ";(frontend)=", "; \1=");
 set req.http.X-Cookie = regsuball(req.http.X-Cookie, ";[^ ][^;]*", "");
 set req.http.X-Cookie = regsuball(req.http.X-Cookie, "^[; ]+|[; ]+$", "");
}

// Remove cookies
unset req.http.Cookie;

Then assuming you only have the x-cookie set for the frontend, in your php before the headers are sent:


$_frontend = isset ($_SERVER['HTTP_X_COOKIE']) ? $_SERVER['HTTP_X_COOKIE'] : null;
if ($_frontend){
    list ($name,$value) = explode("=",$_frontend);
    $_COOKIE[$name] = $value; //setCookie didn't work for me here
}

Clear Cms / Product / Category pages

Update the content frequently? When you save a cms page, product or category page simply hear the corresponding event and “purge” or “refresh” that URL. The important thing to remember is that if you can: 1-) use cURL in BASH over PHP's cURL and 2-) when clearing the product URLs you also need to clean the categories as well. In the terminal you could do:

curl -X PURGE URL_TO_CLEAN

Clear / Refresh all pages

A favorite of mine – when finding all of the URLs in the system, use Magento's sitemap resource collections:

    //get all the url rewrites urls and cms pages (including the home page)
    public function getAllUrls() {

        $collections = array('sitemap/catalog_category', 'sitemap/catalog_product', 'sitemap/cms_page');

        $baseUrl = Mage::app()->getStore()->getBaseUrl();
        $storeId = Mage::app()->getStore()->getId();
        $urls = array();

        foreach ($collections as $collection) {
            $collection = Mage::getResourceModel($collection)->getCollection($storeId);
            $key = md5(serialize($collection));
            if ($data = $this->getDataFromCache($key)) {
                $urls = array_merge($urls, $data);
            } else {
                foreach ($collection as $url) {
                    $urls[] = htmlspecialchars($baseUrl . $url->getUrl());
                }
                $this->setKey($key)
                        ->setData($urls)
                        ->saveDataInCache();
            }
            unset($collection);
        }

        $urls = array_merge($urls, $this->getSearchUrls());
        //account for the homepage 
        $urls[] = $baseUrl;

        return $urls;
    }
     public function getSearchUrls() {
        $urls = array();
        $collection = Mage::getResourceModel('catalogsearch/query_collection')
                ->setPopularQueryFilter(Mage::app()->getStore()->getId())
                ->setOrder('popularity', 'DESC');

        $catalogSearchHelper = Mage::helper('catalogsearch');
        foreach ($collection as $search) {
            $urls[] = $catalogSearchHelper->getResultUrl($search->getData('name'));
        }

        return $urls;
    }

Please note that I use some extra functions here: setKey, setData, saveDataInCache & getDataFromCache. Ultimately is all about performance and I rely on Magento's cache to save the collection of urls. That will get you all of Magento URLs that your system might be caching. Now you could iterate that array and either refresh or purge varnish's cache

Roma....

noreply@blogger.com (Anonymous) — Sun, 12 Feb 2012 06:07:00 +0000

Roma tomato loves the water

Trip to IOP

noreply@blogger.com (Anonymous) — Tue, 24 Jan 2012 05:35:00 +0000

I hate Jellyfish

Magento 301 Redirects

noreply@blogger.com (Anonymous) — Fri, 11 Feb 2011 14:10:00 +0000

Did you ever want to create a redirect in a Magento Store without having to use htacess?
Using Observers and the powerful Request and Response objects from Zend Framework you can. In this example I will show how you can do it.
First you need to create the module, more on module creations here: http://goo.gl/vvT2j.

Create a directory under app/code/community and name it FooBar, then create another directory under FooBar and name it Redirect. Create two directories inside FooBar: etc and Model.
Inside etc create a config.xml file and paste this into it:
Note: For some reason blogger keeps putting all tags in lowercase, foobar_redirect should be FooBar_Redirect.


    
        
            0.1
        
    
            
        
            
                FooBar_Redirect_Model
            
        
         
            
                
                    
                        singleton
                        fooredirect/observer
                        redirect

Now in the Model directory create the Observer.php file and paste this into it:

class FooBar_Redirect_Model_Observer
{
 /**
  * @var array
  */ 
    protected $blacklist = array('checkout');
 /**
  * Takes a request and redirects it a new url based on some params
  */ 
    public function redirect()
    {
        try
        {
            //Let's get the Request object
            $request = Mage::app()->getRequest();
            //We only want to redirect in the frontend and exclude the checkout 
            $storeId = Mage::app()->getStore()->getId();
            if(!in_array($request->getModuleName(), $this->blacklist) &&  $storeId != 0)
            {
                $response = Mage::app()->getResponse();

                $uri = $request->getServer("REQUEST_URI"); //requested url
                $temp = strtolower($uri); //lowercase version

                if(strcmp($uri, $temp) != 0) //if the url are different then redirect
                {
                     /**
                      * Dont need to do this but let's get fancy here 
                      */ 
                    $parsedUrl = parse_url(Mage::getBaseUrl());
                    $scheme = $parsedUrl['scheme'];
                    $host = $parsedUrl['host'];
                    $path = $parsedUrl['path'];
                     //build the url
                    $url = $scheme . '://' . $host . $temp;
                    /**
                     * return the response, note the sendHeaders() method
                     * If you don't add it Magento [Zend_Framework] won't redirect you
                     */ 
                    return $response->setRedirect($url, 301)->sendHeaders();
                }
            }
        }
        catch(Exception $e)
        {
            Mage::logException($e);
        }
    }
}

Let's enable this module, create an xml file under app/etc/modules and name it FooBar_Redirect.xml and add this to it:


    
        
            true
            community

Now just clear Magento's cache and simply visit your site. If you type http://mysite_url.com/UpperUrl you should be redirected to http://mysite_url.com/upperurl.
Now imagine what you can do with this when you have to create a 301 redirect for an entire site, isn't this easier than creating an htacces rule for each url?

My first mobile blog post

noreply@blogger.com (Anonymous) — Thu, 10 Feb 2011 14:59:00 +0000

This Blogger app is sweet. I cannot believe that they took so long to build it.