Insight

"Using Lock tables" error while taking Mysqldump

2011-12-28T10:46:00.000-08:00

This is the comman error which we get while taking the mysqldump of large databases.

File './databasename/tablename.MYD' not found (Errcode: 24) when using LOCK TABLES

If you will print the system error code 24 by using perror, it will return "OS error code 24: Too many open files"
This error can be resolve in several ways -
1- set open-files-limit=10000(or any higher number)
or define open-files-limit and open_files_limit in configuration files.

Note- if you are using MySQL version < 5.1.52. only use open-files-limit don't use open_files_limit. In these version open_files_limit can't set limit higher than 65535. But for newer version these issues are fixed.

2- By using --single-transaction with mysqldump i.e - mysqldump --single-transactions -uroot -p database>database.sql
This option sends a START TRANSACTION SQL statement to the server before dumping data. It is useful only with transactional tables such as InnoDB and BDB, because then it dumps the consistent state of the database at the time when BEGIN was issued without blocking any applications.
When using this option, you should keep in mind that only InnoDB tables are dumped in a consistent state. For example, any MyISAM or MEMORY tables dumped while using this option may still change state.(src -http://dev.mysql.com/doc/refman/5.0/en/mysqldump.html)

3- while taking the mysql dump, use –-lock-tables=false option. But remember it is not recomended for innoDB engines

Every dumped database, lock all tables to be dumped before dumping them. These tables locked to allow concurrent inserts in the case of MyISAM tables with READ LOCAL . In case of Innodb (transactional) tables, --single-transaction better than --lock-tables because it does not need to lock the tables.

Since this option locks tables for each database separately, it is not guaranteed that the tables in the dump file are logically consistent between databases. Tables in different databases may be dumped in completely different states

Setting up mod_wsgi for python in the web

2011-06-30T23:53:00.000-07:00

Mostly web servers cannot execute Python code directly. They need some kind of bridge between the program and server. The purpose of these bridges is to define how programs interact with the server.

Python Installation -
Python Installation from Source -
1- Download python source from
http://www.python.org/getit/
2- tar xfz Python-3.1.tgz
3-cd Python-3.1
4-make
5-make install

Python Installation on Windows -
1- Download python installer from
http://www.python.org/getit
2- Double click the installer
3- Follow the steps of installer
4- Once you have done, just check Start>>Programs>>Python 3.1>>IDLE (Python GUI). You will see the Python IDE

Now Download mod_wsgi from -
http://code.google.com/p/modwsgi/downloads/list
Just check your Python and Apache version and download according to that.
From Source code follow this step -
Unpack the mod_wsgi and run the usual:
1 - untar it
2 ./configure
3 make
4 sudo make install
If you have downloaded binary Apache Module for Win32, rename to - mod_wsgi.
5 Copy it in apache modules directory. In my case it was -
C:\xampp\apache\modules
6 Add the following line to your httpd.conf ( C:\xampp\apache\conf\httpd.conf)
LoadModule wsgi_module modules/mod_wsgi.so
7 Now restart Apache.

Write a WSGI Python code my.py and put into you htdocs/myapp-
def application(env, start_response):
start_response("200 OK", [])
output = "Hello World! Request: %s"
output %= env['PATH_INFO']
return [output]

Now add following lines to httpd.conf.
WSGIScriptAlias /myapp "C:/xampp/htdocs/myapp/"

Order deny,allow
Allow from all

Modify mime.types by adding 'py' -

text/html html htm py

Now restart the apache and try this in your browser -http://localhost/myapp/my.py

Cross Domain Ajax Request - Proxy, JSON style, FlXHR, XDR

2010-05-24T03:19:00.001-07:00

Same origin policy is the security measures implemented by browser to to prevent a resource loaded from one site manipulating or communicating with another site. Due to same origin policy Cross domain ajax XHR(XMLHttpRequest) is not possible.

But now there are many ways to implement this. Some of them are -
1. Proxy
2. JSON with padding (JSONP)
3. Flash based - FlXHR
4. XDR XDomainRequest in Internet Explorer 8 (I have not tried this)

Proxy - This is the most comman approach. In this apporoach we send ajax request to our own domain and then send a backend request from our servert to another server, Now once we get a response we return it to the browser

Suppose abc.com application wants data from xyz.com in ajax look. App will send the request to abc.com server now abc.com server will send a backend request using any method to xyz.com and after getting the response it will return to the browser.
There is one more proxy solution - using apache's mod_rewrite or mod_proxy we can request from our server to another server.

JSON - This is also one way to do cross-site scripting by On-Demand Javascript. This is very simple method in which we have to insert a new script elements into our application DOM, with dynamically created src attributes. This look like an XMLHttpRequest().

But there is a limitation in this - other party's service has to return a valid JavaScript. It should supports some kind of JSONP reply structure, Means The reponse back will load a JSON object as a parameter of the callback function that we specified in our request. For example - Yahoo has implemented this feature in their web services API's.

http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Madonna&results=2&output=json

Above url will return a valid JSON object, but if I append a callback param, then it will load this object as a parameter of my callback function. See the output of this -

http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Madonna&results=2&output=json&callback=getMyData

Now, If we create an script element with this url as a source it will call the callback function with parameter of JSON output, now we have to only define the callback function and do the rest according to your need.
// Callback function
function getMyData(jsonData) {
alert('Titile = ' + jsonData.ResultSet.Result[0].Title + ' Summary = ' + jsonData.ResultSet.Result[0].Summary);
head.removeChild(script);
}

// Web service call
var req = 'http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Madonna&results=2&output=json&callback=getMyData';
// Create a new request object
var head= document.getElementsByTagName('head')[0];
var script= document.createElement('script');
script.type= 'text/javascript';
script.src= req;
head.appendChild(script);

Flash based -flXHR - flXHR is a client-based cross-browser, XHR-compatible tool for cross-domain Ajax. It utilizes an invisible flXHR.swf instance that acts as sort of a client-side proxy for requests, combined with a Javascript object/module wrapper that exposes an identical interface to the native XMLHttpRequest (XHR) browser object. I have just tried this and it is working perfectly fine. You have to do following thisngs only -
1- Download flXHR from http://flxhr.flensed.com/download.php
2- If you are using any library(jquery, prototype, dojo etc ) then it is good, there supports are also available.
In my case i tested it simply-
3- Extarct code, copy depoly folder on your app directory
4- On your page include flXHR.js
5 - Include this script code on your page

var ajobj = new flensed.flXHR();
ajobj.onreadystatechange = ajCallback;
ajobj.onerror = ajError;
ajobj.loadPolicyURL = "http://staging.itimes.com/crossdomain.xml";

function ajCallback(loadObj) {
if (loadObj.readyState == 4) {
alert(loadObj.responseXML);
alert(loadObj.responseText);
}
}
ajobj.open("POST","http://staging.itimes.com/flXHR-tests/xyz.php");
ajobj.send();

function ajError(errObj) {
alert("Error"+errObj);
}

6- Thing has to remember is on thirdparty.com root directory, there should be a crossdomain.xml which defines the acess somthing like -

cross-domain-policy
allow-access-from domain="mysite.com"
allow-http-request-headers-from domain="mysite.com" headers="*"
cross-domain-policy

Above are xml tag. Instead of mysite.com you can use * which will give access to all domain.

XDR - XDomainRequest is Cross-domain Request in Internet Explorer 8, It is one of the way to make anonymous requests to third-party sites that support XDR and opt in to making their data available across domains. It has two components: a client side that makes a request for data to a URL across domains, and a server side that responds with the Access-Control-Allow-Origin header of either * or the exact URL of the requesting page.IE8 to request data from the domain’s server by sending an Origin header with the serialized value of the origin of the requestor and response will return if server responds with Access-Control-Allow-Origin. You can read more about XDR - http://msdn.microsoft.com/en-us/library/dd573303%28VS.85%29.aspx

Practically I have not tried this but i'll try it soon.

Alternate options to acheive Multiple Inheritance property to some extent in php

2010-04-28T23:03:00.000-07:00

As we all know that php does not support multiple inheritance, but we can acheive it through some extent by writing our code. suppose we have a absract class Ingredents -

abstract class Ingredents {

private $DishId;
abstract public function requirement();

public function __construct($DishId) {
$this->DishId = $DishId;
}
public function getDishId() {
return $this->DishId;
}
}

Now If we want to make milk cake we can extend it simply -

class Milkcake extends Ingredents {

private $req = array();

public function __construct($req){
$this->req = $req;
}
public function requirement() {
return $this->req;
}
}

And if we want to make kesar rice then also we can get it simply

class KesarRice extends Ingredents {

private $req = array();

public function __construct($req) {
$this->req = $req;
}
public function requirement() {
return $this->req;
}
}

But now if we want to make kheer then ? ok we are going to use property of Milkcake and kesar rice both.

class Kheer {

private $req = array();
private $dishes = array();

public function requirement() {
foreach ($this->dishes as $dishes) {
$this->req[] = $dishes->requirement();
}
}
public function setDishes($dishes) {
$this->dishes = $dishes;
}
public function getRequirement() {
return $this->req ;
}
}

Now Execute the code -

$kesar_rice = new KesarRice(array('Rice','kesar'));
$kesar_rice_requirement = $kesar_rice->requirement();
print_r($kesar_rice_requirement);

$milk_cake = new Milkcake(array('Milk','sugar'));
$milk_cake_requirement = $milk_cake->requirement();
print_r($milk_cake_requirement);

$kheer = new Kheer();
$kheer->setDishes(array(
'KesarRice' => new KesarRice(array('Rice','kesar')),
'Milkcake' => new Milkcake(array('Milk','sugar'))
));
$kheer->requirement();
$kheer_requirement = $kheer->getRequirement();
print_r($kheer_requirement);

skipfish - web application security scanner from Google.

2010-04-20T04:16:00.001-07:00

Few days earlier I tried, skipfish - web application security scanner from Google.
It uses dictionary- based and and recursive crawl approach. The way it displays the result is very impressing, It has rich javascript based user interface. It divides the results in several parts like - High risk, Medium risk, Low risk, Warnings, Notes, Unique children nodes.

Skipfish is purely written in C and easily achieves 2000 requests per second. Skipfish is realy very easy to install
Installation -
1. Download skipfish from - http://skipfish.googlecode.com/files/skipfish-1.32b.tgz
2. Untar it.
3. Now we need libidn, I installed it with add/remove software option on fedora. You can download it from libidn.
4. cd skipfish
5. make
(it does not standalone installation (make install))
6.Copy dictionary file. According to your use you can copy any dictionary file.
cp dictionaries/default.wl skipfish.wl

Now you can test your web application using skipfish.

./skipfish -o siteresult http://example

Skipfish provide various option like -
-A user:pass (for simple HTTP credentials)
-C cookiename=cookieval (for cokkie credentials)
-X /logout.php (matching url will not be feched)
-I (only crawl URLs matching a substring)
-S (ignore links on pages)
-D (allow to specify additional domains )
-B (prevents crawl to third-party domain)
and so on continue... for more details you can visit - http://code.google.com/p/skipfish/wiki/SkipfishDoc

skipfish is very useful, its high risks includes -

Server-side SQL injection (including blind vectors, numerical parameters).
Explicit SQL-like syntax in GET or POST parameters.
Server-side shell command injection (including blind vectors).
Server-side XML / XPath injection (including blind vectors).
Format string vulnerabilities.
Integer overflow vulnerabilities.
Locations accepting HTTP PUT.

Before using skipfish you make sure that you run this test on staging instance because skipfish will crawl to every link that appears on the site. It can perform any operation of insert, delete, and modify so you should not run on a production site. When I used skipfish it gives very valuable information. Detecting flaws early is surely something worth doing.
skipfish is good quality - it updates it dictionary on run time. Before using skipfish please read skipfish/dictionaries/README-FIRST. So go ahead and use it but not on production server.

APC - Alternative PHP Cache

2010-03-23T02:22:00.000-07:00

I am working on a heavy traffic social networking site (build in php), we are already using Memcache and some other file based caching techniques. Our team decided to use any opcode cahing which can cache compiled code, we decided to use APC.

APC (Alternative PHP Cache)

We all know that php is an interpreted Language, when we request for any page, following process happens -
1. Load the script.
2. Parse the script.
3. Compile into opcode.
4. Execute the script.
When we use any opcode caching(here is APC), it cache the compile code in memory. So when this page will again accessed it reads the compile code from memory and execute it. This process saves the step #1,#2,#3. So it saves time and computer resources both.

Installing APC - APC is free open source package by PECL,. It is actively maintained and most probably will be include in php6. Here are the easy steps to install it -
1. Download the package from http://pecl.php.net/package/APC
2. Untar it
$ tar -xzvf APC-3.1.3p1.tgz
3. Go in APC folder
$ APC-3.1.3p1
4. $ phpize
5. $ ./configure --enable-apc --enable-apc-mmap --with-apxs --with-php-config=/usr/local/bin/php-config (check your php config path it may be /usr/bin/php-config also)
6.$ make
7.$ make test
8.$ make install
9 Insert following entry in your php.ini
extension=apc.so
6. Move apc.so in your php modules folder
$ mv /usr/local/lib/php/extensions/no-debug-non-zts-20060613/apc.so /usr/lib/php/modules/
7. Now restart your apache.

Once the APC has installed you can configure it. APC has vast range of run time configuration option( lke setting segments, segments size, file ttl, user ttl etc). You can find full list here - http://in3.php.net/manual/en/apc.configuration.php
With the use of APC_UPLOAD_PROGRESS you can create progress meter also.

After configuration move apc.php in your root directory. Now run apc.php, it will show you all the valuable information, Even it shows the memory usage and fragmentation graph, all the files and user variable cached etc.

I have not use eAccelerator but i read at several places that in performance eAccelerator is little better than APC, but eAccelerator is not stable. eAccelerator produce regular fatal errors, needing to restart Apache. Please read these links -
http://drupal.org/node/323736
http://www.simplemachines.org/community/index.php?topic=364083.0

Since APC is stable, actively maintained, undergoing with quick development and most probably will include in php 6 so, i recommend APC. But definately i'll test eAccelerator and Xcache also to compare the result with APC.