Ravi Raj's Blog

mysql prepared statements caching

noreply@blogger.com (Anonymous) — Mon, 13 Oct 2014 10:04:00 -0700

PreparedStatement is cached inside the J2EE server connection pool manager. The J2EE server keeps a list of prepared statements for each database connection in the pool. When an application calls prepareStatement on a connection, the application server checks if that statement was previously prepared. If it was, the PreparedStatement object will be in the cache and this will be returned to the application. If not, the call is passed to the jdbc driver and the query/preparedstatement object is added in that connections cache.

We need a cache per connection because that's the way jdbc drivers work. Any preparedstatements returned are specific to that connection.

If we want to take advantage of this cache,We need to use parameterized queries so that they will match ones already prepared in the cache. Most application servers will allow you to tune the size of this prepared statement cache.

PHP/MYSQL

So there are good reasons to use prepared statements:

Save on query parsing
Save on data conversion and copying
Avoid SQL Injection
Save memory on handling blobs

There are also drawback of using prepared statements:

Query cache does not work
Extra server round trip required if statement used only once
Not all statements can be prepared. So you can’t use prepared API exclusively you’ll need to fall back to normal API for some statements
Newer and sometimes buggy code. I had a lot of problems with PHP prepared statements. It is getting better but still it is less mature than standard API
You can’t use placeholders in place of all identifiers. For example you can’t use them for table name. In certain version it even does not work for LIMIT boundaries
Inconvenient list handling. Unlike in for example PEAR emulated prepard statements there is no nice way to pass list of values to IN
Harder tracing. Logs were now fixed to include full statement text not only “Execute” but in SHOW INNODB STATUS you would still see statements without actual values – quite inconvenient for analyses.

For a prepared statement executed via the binary protocol, comparison with statements in the query cache is based on the text of the statement after expansion of ? parameter markers. The statement is compared only with other cached statements that were executed via the binary protocol. That is, for query cache purposes, statements issued via the binary protocol are distinct from statements issued via the text protocol.

We can't re-use a mysql statement prepared during a previous request in php.

If PHP application uses connection pooling to the database, and the database caches prepared statements, then yes, the caching will persist between pages. If the prepared statement caching is done by the client library, then this is more nebulous.

Reusing the same variable name in PHP won't invalidate the MySQL prepare "cache".

$stmt = $dbh->prepare("SELECT column_A FROM Table1 WHERE id=?");
$stmt->bindValue(1, $id, PDO::PARAM_INT);
$stmt->execute();

$stmt = $dbh->prepare("UPDATE Table2 SET column_B=? WHERE column_A=?");
$stmt->bindValue(1, $name);
$stmt->bindValue(2, $column_A);
$stmt->execute();

MYSQL persistent connection

Mysql server offer statement caching.

mysql> PREPARE stmt_name FROM "SELECT name FROM Country WHERE code = ?";
Query OK, 0 rows affected (0.09 sec)
Statement prepared

mysql> SET @test_parm = "FIN";
Query OK, 0 rows affected (0.00 sec)

mysql> EXECUTE stmt_name USING @test_parm;
+---------+
| name |
+---------+
| Finland |
+---------+
1 row in set (0.03 sec)

mysql> DEALLOCATE PREPARE stmt_name;

Query OK, 0 rows affected (0.00 sec)

When a request is served php "cleans" the instance and frees resources and other variables. This is done in several steps. Since apache keeps the process alive after a request not all steps are executed and not all memory is freed. There is e.g. EG(persistent_list) which is used by mysql_pconnect(), pg_pconnect(), ... This list isn't emptied between requests as long as the process keeps alive (could be, depending on the actual implementation, but that would defy the purpose of EG(persistent_list)). If you use persistent connections your script might get a "re-used" connection established during a previous request.
To (re-)use a prepared statement directly you need the identifier for that statement (and that connection). When using (php-)postgresql this is simply a (connection-wise) unique string you pass to pg_execute(), so your script has no problem to gain access to the statement previously prepared by another instance (using the same connection).

Using mysqli or PDO-mysql you need a resource/object as statement identifier. That's kind of a problem since neither the mysqli nor the pdo extension seem to offer a way of storing the resource in EG(persist_list) between requests and you can't recreate it either, so it's seems impossible to re-use a mysql prepared statement directly.

But as we know we should not use persistent connection , here are some good points.

When you lock a table, normally it is unlocked when the connection closes, but since persistent connections do not close, any tables you accidentally leave locked will remain locked, and the only way to unlock them is to wait for the connection to timeout or kill the process.
Normally temporary tables are dropped when the connection closes, but since persistent connections do not close, temporary tables aren't so temporary. If you do not explicitly drop temporary tables when you are done, that table will already exist for a new client reusing the same connection. The same problem occurs with setting session variables.
If PHP and MySQL are on the same server or local network, the connection time may be negligible, in which case there is no advantage to persistent connections.
Apache does not work well with persistent connections. When it receives a request from a new client, instead of using one of the available children which already has a persistent connection open, it tends to spawn a new child, which must then open a new database connection. This causes excess processes which are just sleeping, wasting resources, and causing errors when you reach your maximum connections, plus it defeats any benefit of persistent connections.

Also Try to use it smartly. For example.

$ids = array(12,34,56);

$sql = “SELECT * FROM address WHERE address_id = :address_id”;

$databaseHandle = CustomDB::getDBH();

$rows = array();

$preparedStatementHandle = $databaseHandle->prepare($sql);

foreach ($ids as $id) {

$preparedStatementHandle->execute(array(‘address_id’ => $id));

$result = $preparedStatementHandle->fetchAll();

$rows = array_merge($rows, $result);

}

It doesn't make sense when we can get gain using SELECT id FROM address WHERE address_id IN (12,34,56);

Prepared Statements are useful when they are related to session-long caching alongside a database wrapper that’s just good enough to programmatically generate most typical lookups.

Unfortunately, “good enough” caching looks complicated:

Example Queries:

SELECT * FROM address WHERE address_id = 12;

SELECT * FROM address WHERE address_id = 34;

SELECT * FROM address WHERE address_id = 56;

Example Code, assuming PDO & PHP, to produce those queries using Prepared Statements with caching:

$ids = array(12,34,56);

$sql = “SELECT * FROM address WHERE address_id = :address_id”;

$databaseHandle = CustomDB::getDBH();

Cache::setStrategy(Cache_LRU::getLabel());

$rows = array();

if (!Zend_Registry::isRegistered(self::PREP_STMT_CACHE_KEY)) {

Zend_Registry::set(self::PREP_STMT_CACHE_KEY, Cache::getInstance());

}

$preparedStatementHandle = Zend_Registry::get(self::PREP_STMT_CACHE_KEY)->get($sql);

if (!$preparedStatementHandle) {

$preparedStatementHandle = $databaseHandle->prepare($sql);

// Use the sql itself as the index/hash

Zend_Registry::get(self::PREP_STMT_CACHE_KEY)->set($sql, $preparedStatementHandle);

}

foreach ($ids as $id) {

$preparedStatementHandle->execute(array(‘address_id’ => $id));

$result = $preparedStatementHandle->fetchAll();

$rows = array_merge($rows, $result);

}

Prepared Statements fail

Prepared Statements only exist for the current session, so holding onto a handle after a session closes will lead to failures
Each Prepared Statement consumes a handle from the instance’s Prepared Statement pool, which is the “max_prepared_stmt_count”
Out Of Memory on the client side

Case 1:
A connection closes while a cache of Prepared Statement Handles exists

Solution:
Update your PDO wrapping class to have a __destruct method defined to clear the relevant cache of Prepared Statement Handles before calling the parent’s destruct method.

Case 2:
The max_prepared_stmt_count value is reached on a database

Solution:
Immediately drop all local caches of Prepared Statements and try again. If there is still an issue, activate PDO’s ATTR_EMULATE_PREPARES flag to silently convert calls of ->prepare and ->exec into standard SQL Statements.

Case 3:
Out Of Memory (OOM) on the client-side.

Solution:
Reduce the Prepared Statement Handle Cache size. The cache does not have to be large if it is well managed. Even my company’s complicated webapp’s web requests do not fill a 200-statement-long FIFO cache.

Remaining Points:

Monitor (ie: Nagios) Prepared_stmt_count vs max_prepared_stmt_count
Monitor (ie: StatsD) the Prepared Statement Handle Cache hit, miss, and purge rate.
An LRU’s extra minimal overhead is only worthwhile over a simple FIFO if your Prepared Statement Handle Cache is too small for all the queries that should be cached.
Note: Your cache should be small due to the unbounded and invisible memory consumption of Prepared Statement Handles on the database server
A best-case Prepared Statement instantiation against localhost with a simple select costs me, on average, about 300 microseconds. Pulling a handle from a cache is about 6 microseconds.
Coworkers have shown me that long lists of Named Parameters (ie: “:id1, :id2, :id3, [...]“) get more expensive with quantity whereas long lists of Ordered Parameters (ie: “?,?,?,[...]“) remain cheap even in large number. Numerically quantifying this slowdown will be a future post.
Ordered Parameters’ values are not decoded in SHOW PROCESSLIST. Named Parameters’ values are displayed, however, which makes them, to me, far preferable.

smooth mobile scrolling

noreply@blogger.com (Anonymous) — Wed, 8 Oct 2014 04:43:00 -0700

All GUIs generally work the same way. There is a main thread with a loop that processes messages from a queue. Messages can range from "move view to this location" or "user has performed a touch at location". The whole point is that it is a queue so every message generally gets processed one at a time and in a first come first serve order.

For the majority of UI toolkits, including those found on iOS and Android, accessing and modifying objects must be done in the main thread. Despite sometimes being called the UI thread, it is usually also the main thread and often is responsible for not just painting, changing colors, moving objects but also for loading files, decoding images, handling network responses etc.

In Android, if you want to animate an object and make it move an object from location1 to location2, the animation API figures out the intermediate locations (twinning) and then queues onto the main thread the appropriate move operations at the appropriate times using a timer. This works fine except that the main thread is usually used for many other things -- painting, opening files, responding to user inputs etc. A queued timer can often be delayed. Well written programs will always try to do as many operations as possible in background (non main) threads however you can't always avoid using the main thread. Operations that require you to operate on a UI object always have to be done on the main thread. Also, many APIs will funnel operations back to the main thread as a form of thread-safety. It is usually almost impossible to keep all operations on the main thread down to 1/60th of a second in order to allow animations to be processed smoothly. Even if Google could manage to get their code to do just that, it doesn't mean third party Application writers will be able to.

In iOS operations on UI objects also must be done on the main thread with the exception of animation operations done via CoreAnimation. CoreAnimation runs on a background thread and is able to directly manipulate, move, recolor and reshape UI objects on a background (CoreAnimation) thread. Compositing, rendering is also performed in this thread. It does this through a combination of hardware and software, providing very smooth and fast animations. From the main thread you can basically issue a call to CallAnimation and tell it to move object1 from location1 to location2. This animation will continue to run even if the main thread is blocked performing another operation. This is why animations will almost never stutter on iOS.The main thread manages application data and UI application state (UI application state includes things such as the strings to be displayed in a ListView etc) but issues physical UI state change requests to a separate and dedicated high priority CoreAnimation thread (physical states include things such as color, position and shape). All physical state changes can be animated and CoreAnimation will also perform the twinning for you (like the Android animation APIs). Non animated physical state changes will be issued directly by CoreAnimation and the main thread (not the CoreAnimation thread) will block until those are performed. Animated physical state changes that are issued by the main thread will be performed asynchronously by the CoreAnimation thread. Because physical UI state and only physical UI state is managed by the CoreAnimation thread, the main thread can be blocked or busy but the CoreAnimation thread will still continue to not only accurately render the last known state of the UI (as issued by the main thread) but also continue to render any pending or incomplete animated UI physical state changes as requested by the main thread.

In Windows Vista, Microsoft introduced desktop composition whereby the OS maintained a separate pixel buffer for every window. This meant that even if an application hung, the last state of the window (how it looked) is still rendered rather than just being drawn as white (the OS partially managed the state of the pixels in the window). CoreAnimation goes beyond this and offloads much of the UI work traditionally managed by the main thread including managing the state of not just the pixels (like Vista) but of higher level concepts such as widgets, widget locations, widget colors etc.

At Android animation model, It's the way many toolkits work including Flash which was definitely very animation heavy. I would say the iOS model makes the overall user experience nicer and offloads one more worry for the developer back to the operating system. I'm sure Google will continue to recognize the importance of animation on touch screen devices and continue to accelerate (or re architecture) Android in coming releases.

A 5 year old 1st generation iPhone will perform smoother and more reliable animations than the latest quad core Samsung Android phone. It's a software design problem and not something you can throw more cores at (not least of which because the main thread will only ever run on one core!). Don't believe people when they excuse stutter and lag as "oh just the Android Java garbage collector". Modern compacting, generational garbage collectors generally aren't the cause of the kind of stutter you see on Android.

Let’s not forget that Android does true multitasking and background processes execution, unlike iOS, which adds to the overhead. Also adding to the overhead: Native iOS apps are binaries pre-compiled for their own hardware while Android uses the Dalvik virtual machine with just-in-time compilation to run Dalvik dex-code (Dalvik Executable), which is usually translated from Java bytecode.

Kinetic scrolling

Kinetic scrolling is the combination of regular, drag-finger-on-screen scrolling with an additional movement after the finger is lifted off the screen.Based on how fast the finger was dragged on the screen, the duration, speed and deceleration of the additional movement can vary.

kinetic scrolling can be viewed as the sum of two features, it can be implemented in two steps.

The first step is click & drag scrolling. It can be achieved by installing an event filter and intercepting mouse press, move and release events. When a press event is received the scrolling starts, when a move event is received the list is scrolled, and finally when a release event is received the scrolling stops. To avoid accidental clicks, all the events are blocked inside the filter function.

For step two, the scroller continues to scroll the list automatically after the user has lifted their finger off the screen, gradually slows down and then stops. To display a pleasing effect, the scroller must decide how fast to scroll, how far to scroll and how fast to slow down.

A good starting point is "how fast to scroll". In physics velocity represents the direction in which and magnitude by which an object changes its position. Speed is another word for magnitude in this context. The "how fast to scroll" question can be answered by recording the cursor’s drag velocity on the screen. A simple but imprecise way to do this is to poll the cursor position at specific time intervals; the difference in positions represents the speed (measured in pixels / timer interval) and the mathematical sign of the difference represents the direction. This algorithm will give a good enough idea on whether the cursors is moving fast or slow.

Next up is "how far to scroll".How far is actually connected to how fast to slow down because the list is scrolled with a certain velocity and then it decelerates until it stops. Since the velocity has previously been established, the only thing left is to calculate the deceleration based on friction. In physics, kinetic friction is the resistance encountered when one body is moved in contact with another. Of course, there can be no friction between pixels, but kinetic scrolling is a simulation and one can pretend that the list items are moving over the list container and that this movement generates friction. In reality friction is calculated based on the nature of the materials, mass, gravitational force and so on. In the simulation a numeric value is used to alter the speed of scrolling.

Having determined the deceleration,"how far" the list scrolls kinetically is simply a function of the time that it needs to reach a speed of zero.

Final Steps:

Need to measure the velocity of finger cursor.
Implement a simple particle physics loop. information about how to do that here
give your particle "bounds" using math derived from the width of your scrolling plane, and the width of your viewport
continuously Add the the difference between the mouse velocity and the particle velocity, to the particle's velocity, so the particle's velocity "matches" the mouse's velocity for as long as it's moving.
Stop doing step 4 as soon as the user lifts their finger. The physics loop takes care of inertia.
Add your personal flourishes such as "bumper" margins, and smooth scrolling "anchor" points

Basic scrolling functionality: This task consisted in handling drag events in the simplest way, which is by scrolling the contents according to the drag distance and direction. This was relatively straightforward to implement, the only tricky aspect was to know when to start a drag operation vs. when to pass down a tap event to a child widget inside the scrollable area.

Scroll inertia: This one was the most challenging. The idea here is that scrolling should continue for some time after the user lifts the finger, slowing down until it stops completely. For this I needed to have an idea of the scroll velocity. Unfortunately it is not accurate to compute the velocity from a single sample, so while the user is scrolling I record the last N motion events in a circular buffer, along with the time at which each event occurred. I found N=4 to work just fine on the iPhone and on the HP TouchPad. When the finger is lifted I can compute an approximate start velocity for the inertial scrolling from the recorded motion. I defined a negative acceleration coefficient and used standard motion formulas (see here) to let the scrolling die down nicely. If the scroll position reaches a border while still in motion I just reset the velocity to 0 to prevent it from going out of range (the abrupt stop is addressed next).

Flexible scrolling limits: instead of going into an abrupt stop when the scroll reaches the end I wanted the widget to scroll some, but offering resistance. For this I extended the allowed scroll range on both ends by an amount that I defined as a function of the widget dimensions. I've found that adding half the width or height on each end worked nicely. The trick to give the scrolling the feeling that it is offering some resistance was to adjust the displayed scroll positions when they are out of range. I used a scaling down plus a deceleration function for this (there are some good easing functions here).

Spring behavior: since now it is possible to scroll past the valid range, I needed a way to bring the scroller back to a valid position if the user left it out of range. This is achieved by adjusting the scroll offset when the scroller comes to a stop at an out of range position. The adjustment function that I've found to give a nice springy look was to divide the distance from the current position to the desired position by a constant and moving the offset by that amount. The bigger the constant the slower motion.

Scrollbars: the final touch was to add overlay scrollbars, which fade in when scrolling starts and fade out when it ends.

Quick list for android:

Reduce the number of conditions used in the getView of your adapter.
Check and reduce the number of garbage collection warnings that you get in the logs
If you're loading images while scrolling, get rid of them
Set scrollingCache and animateCache to false (more on this later)
Simplify the hierarchy of the list view row layout
Use the view holder pattern
Use dragging proper way (is the type of scrolling that occurs when a user drags her finger across the touch screen Simple dragging is often implemented by overriding onScroll() in GestureDetector.OnGestureListener.)
Use flinging proper way (is the type of scrolling that occurs when a user drags and lifts her finger quickly. After the user lifts her finger, you generally want to keep scrolling (moving the viewport), but decelerate until the viewport stops moving. Flinging can be implemented by overriding onFling() in GestureDetector.OnGestureListener, and by using a scroller object.)

In JS, Kinetic scrolling explained here https://github.com/ariya/kinetic & http://ariya.ofilabs.com/2013/11/javascript-kinetic-scrolling-part-2.html

Some Mobile SEO related useful links

noreply@blogger.com (Anonymous) — Mon, 23 Jun 2014 10:24:00 -0700

General guidelines

Google's developer guide to smartphone sites - this is the most important document and the best place to start:

https://developers.google.com/webmasters/smartphone-sites/

For sites with separate mobile and desktop URLs, this section is especially important:

https://developers.google.com/webmasters/smartphone-sites/details#separateurls

These are common mistakes Google has seen in mobile sites:

https://developers.google.com/webmasters/smartphone-sites/common-mistakes

Changing in rankings of smartphone search results:

http://googlewebmastercentral.blogspot.co.uk/2013/06/changes-in-rankings-of-smartphone_11.html

25 key principles of mobile site design:

http://www.google.com/think/multiscreen/whitepaper-sitedesign.html

Responsive sites

Where to start with responsive design:

http://googlewebmastercentral.blogspot.co.uk/2012/04/responsive-design-harnessing-power-of.html

Switching over from a separate mobile site to a responsive site (or vice-versa):

https://developers.google.com/webmasters/smartphone-sites/change-configuration

Crawling and indexing

Crawling mobile sites:

http://googlewebmastercentral.blogspot.co.uk/2014/01/a-new-googlebot-user-agent-for-crawling.html

Videos by Matt Cutts covering mobile issues

Is there a way to tell Google about a mobile version of a page?

https://www.youtube.com/watch?v=HVQHbfpqXHM

Is there an SEO disadvantage to using responsive design...?

https://www.youtube.com/watch?v=D03wRb4s7MU

Is page speed a more important factor for mobile sites?

https://www.youtube.com/watch?v=-I4rWnQxxkM

Does indexing a mobile website create a duplicate content issues? (Also covers cloaking)

https://www.youtube.com/watch?v=mY9h3G8Lv4k

Should we create a mobile version of our site? (old video from 2010)

https://www.youtube.com/watch?v=pJ_UP_1JERc

Should I use the Vary HTTP header...?

http://www.youtube.com/watch?v=va6qtaiZRHg

Analyzing and improving mobile sites

Google's Maile Ohye has made some great videos about analyzing and improving mobile sites using data in Webmaster Tools and Google Analytics. They're summarized and linked from this blog post:

Checklist and videos for mobile website improvement

http://googlewebmastercentral.blogspot.co.uk/2013/12/checklist-and-videos-for-mobile-website.html

Analyzing search queries for mobile sites:

http://googlewebmastercentral.blogspot.co.uk/2014/01/improved-search-queries-stats.html

Speed and user experience

Making smartphone sites load fast
http://googlewebmastercentral.blogspot.co.uk/2013/08/making-smartphone-sites-load-fast.html

That post introduces PageSpeed Insights, which will give you a report on the speed and user experience of your mobile and desktop sites:

https://developers.google.com/speed/pagespeed/insights/

Quick fixes in mobile website performance (a video by Maile)
https://www.youtube.com/watch?v=gy_m44X3I84

Avoiding faulty redirects in sites with separate desktop and mobile URLs:

http://googlewebmastercentral.blogspot.co.uk/2014/06/faulty-redirects.html

Tablets

Giving Tablet Users the Full-Sized Web - Pierre Far explains why tablets are often best served with desktop sites:
http://googlewebmastercentral.blogspot.co.uk/2012/11/giving-tablet-users-full-sized-web.html

android views, widgets, view groups, layouts, containers

noreply@blogger.com (Anonymous) — Wed, 7 May 2014 18:29:00 -0700

The user interface is implemented as collection of view objects (e.g., a date widget, or editable text) -- a view is a class and a widget (e.g., a button) which is drawn on some part of the screen and is responsible for event handling such as when the user interacts with the UI (e.e., clicks on a button, enters the date). User's can construct sophisticated UIs by bundling views together using layouts (or ViewGroups) which can be considered as invisible containers. These containers can hold child containers. Each container defines its views (or other ViewGroups) and their layout properties.

awk and log parsing

noreply@blogger.com (Anonymous) — Sun, 20 Apr 2014 09:53:00 -0700

Find the number of total unique visitors:

cat access.log | awk '{print $1}' | sort | uniq -c | wc -l

2. Find the number of unique visitors today:

cat access.log | grep `date '+%e/%b/%G'` | awk '{print $1}' | sort | uniq -c | wc -l

3. Find the number of unique visitors this month:

cat access.log | grep `date '+%b/%G'` | awk '{print $1}' | sort | uniq -c | wc -l

4. Find the number of unique visitors on arbitrary date – for example March 22nd of 2007:

cat access.log | grep 22/Mar/2007 | awk '{print $1}' | sort | uniq -c | wc -l

5. (based on #3) Find the number of unique visitors for the month of March:

cat access.log | grep Mar/2007 | awk '{print $1}' | sort | uniq -c | wc -l

6. Show the sorted statistics of “number of visits/requests” “visitor’s IP address”:

cat access.log | awk '{print "requests from " $1}' | sort | uniq -c | sort

7. Similarly by adding “grep date”, as in above tips, the same statistics will be produces for “that” date:

cat access.log | grep 26/Mar/2007 | awk '{print "requests from " $1}' | sort | uniq -c | sort

Most Common 404s (Page Not Found)
cut -d'"' -f2,3 /var/log/apache/access.log | awk '$4=404{print $4" "$2}' | sort | uniq -c | sort -rg

2 - Count requests by HTTP code

cut -d'"' -f3 /var/log/apache/access.log | cut -d' ' -f2 | sort | uniq -c | sort -rg

3 - Largest Images
cut -d'"' -f2,3 /var/log/apache/access.log | grep -E '\.jpg|\.png|\.gif' | awk '{print $5" "$2}' | sort | uniq | sort -rg

4 - Filter Your IPs Requests
tail -f /var/log/apache/access.log | grep

5 - Top Referring URLS
cut -d'"' -f4 /var/log/apache/access.log | grep -v '^-#39; | grep -v '^http://www.yoursite.com' | sort | uniq -c | sort -rg

6 - Watch Crawlers Live
For this we need an extra file which we'll call bots.txt. Here's the contents:

Bot
Crawl
ai_archiver
libwww-perl
spider
Mediapartners-Google
slurp
wget
httrack

This just helps is to filter out common user agents used by crawlers.
Here's the command:
tail -f /var/log/apache/access.log | grep -f bots.txt

7 - Top Crawlers
This command will show you all the spiders that crawled your site with a count of the number of requests.
cut -d'"' -f6 /var/log/apache/access.log | grep -f bots.txt | sort | uniq -c | sort -rg

How To Get A Top Ten
You can easily turn the commands above that aggregate (the ones using uniq) into a top ten by adding this to the end:
| head

That is pipe the output to the head command.
Simple as that.

Zipped Log Files
If you want to run the above commands on a logrotated file, you can adjust easily by starting with a zcat on the file then piping to the first command (the one with the filename).

So this:
cut -d'"' -f3 /var/log/apache/access.log | cut -d' ' -f2 | sort | uniq -c | sort -rg
Would become this:
zcat /var/log/apache/access.log.1.gz | cut -d'"' -f3 | cut -d' ' -f2 | sort | uniq -c | sort -rg

confusion related to HMVC design in codeigniter

noreply@blogger.com (Anonymous) — Sun, 20 Apr 2014 09:44:00 -0700

There are two main different features that HMVC adds to CodeIgniter which often confuses people:

Modular MVC
Hierarchal MVC

Modular MVC is the feature that most people want to use and is essentially just a way to have a cleaner folder structure.

HMVC is the practise of calling controllers from other controllers without the need for a new HTTP request. This is very rarely useful in my opinion, other than for things like calling a custom 404 page or "widgets".

MMVC adds barely anything to performance, calling a controller via HMVC is obviously almost twice as slow.

Either way neither will be noticeable. If your site is starting to crawl under high traffic then this is one of the last things you'll need to worry about.

nice website based on HTML5

noreply@blogger.com (Anonymous) — Sun, 20 Apr 2014 09:42:00 -0700

https://chains.cc/
http://hypem.com/popular/
http://pinterest.com/
http://www.endomondo.com/
http://habitforge.com/
http://visual.ly/
http://www.evernote.com/
http://www.thesixtyone.com/
https://www.crunch.co.uk/
https://andbang.com/
http://kippt.com/
http://101in365.com/
http://mugtug.com/
http://www.tweetaboogle.com/
http://www.zocial.tv/
http://www.dearmap.com/

Technology related Blogs You must read

noreply@blogger.com (Anonymous) — Sun, 20 Apr 2014 09:40:00 -0700

Here is the list. :-) Few links could be dead so please report me about broken links so that i can update.

meminfo Explained

noreply@blogger.com (Anonymous) — Sun, 20 Apr 2014 09:32:00 -0700

"Free," "buffer," "swap," "dirty." What does it all mean? If you said, "something to do with the Summer of '68", you may need a primer on 'meminfo'.

The entries in the /proc/meminfo can help explain what's going on with your memory usage, if you know how to read it.

Example of "cat /proc/meminfo":

root: total:     used:     free:    shared: buffers: cached:
Mem:   1055760384 1041887232 13873152 0 100417536  711233536
Swap:  1077501952   8540160  1068961792     
MemTotal:  1031016 kB 
MemFree:  13548 kB
MemShared:  0 kB
Buffers:  98064 kB
Cached:   692320 kB
SwapCached:  2244 kB
Active:   563112 kB
Inact_dirty:  309584 kB
Inact_clean:  79508 kB
Inact_target:  190440 kB
HighTotal:  130992 kB
HighFree:  1876 kB
LowTotal:  900024 kB
LowFree:  11672 kB
SwapTotal:  1052248 kB
SwapFree:  1043908 kB
Committed_AS:  332340 kB
The information comes in the form of both high-level and low-level statistics. At the top you see a quick summary of the most common values people would like to look at. Below you find the individual values we will discuss. First we will discuss the high-level statistics.

High-Level Statistics

MemTotal: Total usable ram (i.e. physical ram minus a few reserved bits and the kernel binary code)
MemFree: Is sum of LowFree+HighFree (overall stat)
MemShared: 0; is here for compat reasons but always zero.
Buffers: Memory in buffer cache. mostly useless as metric nowadays
Cached: Memory in the pagecache (diskcache) minus SwapCache
SwapCache: Memory that once was swapped out, is swapped back in but still also is in the swapfile (if memory is needed it doesn't need to be swapped out AGAIN because it is already in the swapfile. This saves I/O)

Detailed Level Statistics

VM Statistics

VM splits the cache pages into "active" and "inactive" memory. The idea is that if you need memory and some cache needs to be sacrificed for that, you take it from inactive since that's expected to be not used. The vm checks what is used on a regular basis and moves stuff around.

When you use memory, the CPU sets a bit in the pagetable and the VM checks that bit occasionally, and based on that, it can move pages back to active. And within active there's an order of "longest ago not used" (roughly, it's a little more complex in reality). The longest-ago used ones can get moved to inactive. Inactive is split into two in the above kernel (2.4.18-24.8.0). Some have it three.

Active: Memory that has been used more recently and usually not reclaimed unless absolutely necessary.
Inact_dirty: Dirty means "might need writing to disk or swap." Takes more work to free. Examples might be files that have not been written to yet. They aren't written to memory too soon in order to keep the I/O down. For instance, if you're writing logs, it might be better to wait until you have a complete log ready before sending it to disk.
Inact_clean: Assumed to be easily freeable. The kernel will try to keep some clean stuff around always to have a bit of breathing room.
Inact_target: Just a goal metric the kernel uses for making sure there are enough inactive pages around. When exceeded, the kernel will not do work to move pages from active to inactive. A page can also get inactive in a few other ways, e.g. if you do a long sequential I/O, the kernel assumes you're not going to use that memory and makes it inactive preventively. So you can get more inactive pages than the target because the kernel marks some cache as "more likely to be never used" and lets it cheat in the "last used" order.

Memory Statistics

HighTotal: is the total amount of memory in the high region. Highmem is all memory above (approx) 860MB of physical RAM. Kernel uses indirect tricks to access the high memory region. Data cache can go in this memory region.
LowTotal: The total amount of non-highmem memory.
LowFree: The amount of free memory of the low memory region. This is the memory the kernel can address directly. All kernel datastructures need to go into low memory.
SwapTotal: Total amount of physical swap memory.
SwapFree: Total amount of swap memory free.
Committed_AS: An estimate of how much RAM you would need to make a 99.99% guarantee that there never is OOM (out of memory) for this workload. Normally the kernel will overcommit memory. That means, say you do a 1GB malloc, nothing happens, really. Only when you start USING that malloc memory you will get real memory on demand, and just as much as you use. So you sort of take a mortgage and hope the bank doesn't go bust. Other cases might include when you mmap a file that's shared only when you write to it and you get a private copy of that data. While it normally is shared between processes. The Committed_AS is a guesstimate of how much RAM/swap you would need worst-case.

Android Memory Management

noreply@blogger.com (Anonymous) — Sun, 20 Apr 2014 09:28:00 -0700

Android OS, as of V2.2 (and 2.3) have two general types of memory: internal storage (sometimes known as application stroage), and SD card (which may be flash memory that works like SD card, but not physical SD card, such as in the Nexus S).

All apps (in the form of APKs) are loaded into "app storage" part of the "ROM" (actually flash RAM). Part of the ROM is the boot ROM which loads the system. The other half of the ROM is "app storage". For example, in Motorola Droid, 256MB is RAM, and 512MB is ROM. Out of 512MB ROM, 256MB is Android System itself (actually a bit less), and the rest is "app storage", to max of 256MB.

With Android 2.2 and "Move2SD", a portion of the APK can be moved onto the "SD card", but main portion must remain in [s]internal[/s] app storage. The size of the main portion that stays would depend on the app. Some apps cannot be moved or will not function if moved. "Protected" apps cannot be moved. Apps that primarily consist of a service and a widget may not work if moved. add Services or widgets needed for startup should not be moved.

For example, If you have a 256MB system (shows as 262MB due to 1024 vs. 1000 KB size difference) and have 130MB of apps and data/cache loaded, then that leaves about 130MB for the system to actually RUN programs. That sounds like a lot, but in reality that is not enough, since the system itself takes 50-80MB, and services will take up another 30-50MB, leaving almost nothing.

256MB RAM phone such as my Moto Droid, AutoKiller shows...

acore : 4.55MB (system)
dialer: 8.95MB (system)
system: 20.38MB (system)
autokiller:5.68MB
messaging: 3.41MB (system)
Swiftkey: 6.59MB
JuiceDefender 4.14MB
Calendar Storage 4.1 MB (system)
acore: 7.7MB (different pid) (system)
smart taskbar 3.81 MB
seePU 3.44MB
Screebl 4.38MB
SetCPU: 3.83MB
ATK Froyo 3.01MB
gapps: 7.79MB (system)
and 2 more at 4.66MB and 3.56MB

That adds up to... 99.88, or 100 MB.

But that is supposed to leave 156MB, right? Wrong. The system itself takes about 100 MB by itself, in addition to loaded programs, according to this thread about T-Mobile G1 (which has 192 MB of RAM, and has about 96000 KB after booting)

MemTotal: 231740 KB, or 226MB
MemFree: 3376
Buffers 272
Cache: 34960
SwapCache: 0

So the system (before OS kernel) uses about 30MB leaving about 226 MB
Cache itself used another 35 MB. , leaving about 189 MB
Minus 100 MB of auto-loaded apps, and you get... 89 MB.

If you run any programs that need more than that, programs and services will be killed to make room.

Native vs. Dalvik

There are two types of Android programs... "Native" programs, and VM programs.

Native programs are written for the specific CPU in the machine. While this gives better performance, this is much harder to achieve, so most people write program for the VM, or "Virtual Machine".

A "virtual machine" is basically a CPU emulator. You feed it a program, and it will run this program, as if it's a real CPU. The good thing about using a VM is it doesn't matter what the actual physical CPU the device uses. You write the program once, and never have to worry about converting it to other CPUs.

Android's VM is called Dalvik, and it is similar to Java's virtual machine. (In fact, Sun/Oracle sued Google for violating Java copyrights on JVM)

Different pieces of a single app

Most apps have either just an activity, or activity along with a service.

"Activity" is basically the user interface that takes your inputs and displays something back. Foreground app would be an activity.

"Service" is a background program that updates something. Common services includes input, widget updates, mail notification, and so on. Other services include Bluetooth, network updates, and so on.

(Actually there are two more types: broadcast receiver, and content provider, but those are not that pertinent to our discussion)

An app can use a widget, and the widget can use a lot of memory, usually several MB at once. You can see the different services and how much memory they are taking under Settings / Applications / Manage Services

How Services Use Memory

As explained above, Android OS have to run programs from within the limited space available, which, on older phones, isn't much. [s]From within that much memory, it needs system work space to load all the services (you probably have a dozen loaded, taking up at least 30 MB) System itself uses about 60-70 MB (acore, phone, gapps, messaging, etc.) That's 100 MB used. That doesn't leave much memory for anything else, if you have 100 MB of apps loaded. (256-100-100=56)[/s] 100MB for system itself, about 100MB used for apps and services, and you got almost nothing left.

If you look at the services screen, at the bottom, there's a bar: red, yellow, and green. There is a number in the red section and some in green. Your services adds up to the number in the green section. The yellow portion is some memory that can be freed. The red stuff are system stuff and can't be moved.

What Happens When System Runs Out of Memory

When the system needs to load programs, but don't see enough available, it will start killing programs and services (to the system, they are all considered "process") from memory based on the following priority:
Empty App: the app is in standby, not being used, but is still in memory. These can be killed without any effect.
Content Provider: process that provides content to the foreground, such as "contacts content provider", "calendar content provider", and so on. Various "storage" are also content providers. Those can be restarted when needed.

Hidden Application: apps not visible, but still running in the background. These are not exactly running, so killing them should have no serious consequences.

Secondary server: services that stay in background and apps such as Launcher (or other home replacements). Most services go here, like music player, clock updater, background sync, and so on, that's not built into the OS. If these are killed there may be some problems, such as the playback is interrupted, background sync stops, widget no longer updates, and so on.

Visible app: the app is running and visible, but due to multi-tasking or such is not currently "on top". Any program with a display in the notification area is considered "visible". Android OS will not kill these programs unless absolutely necessary, but it can happen.

Foreground app: you see this app on screen, currently running, but also includes the system itself and "phone". These are never killed. In any case, system and phone have much higher priority than any app to make sure those are never killed.

Each category above has a certain number associated with it, sometimes known as a "minfree" value (in either "pages" or megabytes, depending on the app). When Android OS free memory drops below the minfree value for that category, apps in that category are killed. The killing starts Empty App group as that has the highest number. if that's not enough, it then starts killing apps in the Content Provider Group, and it keeps going until it has finally freed up enough memory to load the app and all related processes (such as services).

NOTE: Having a constant "notification" in the notification area makes the program "visible app" instead of "hidden app", thus making it less likely to be killed by the system to make room for other apps.

A lot of problems with Android device occur when the system tries to make room by killing "secondary server" processes that are needed. Playback of audio (music or podcast) stopped, download stopped, location services stopped... etc. This especially happens on phones with little RAM. First Android phone, T-Mobile G1 / HTC Magic, has 192MB of RAM. Moto Droid have 256MB of RAM. Second generation of Android phones, like HTC Wildfire, got 384MB of RAM. Recent phones, like Droid X, Galaxy S, and so on got 512MB.

NOTE: Some apps, like web browser, can exit but still save the URL you were browsing. So when the process reloads, it is almost as if it was never unloaded. Unfortunately not all apps can do that.

So what is the solution?

There are two approaches to the problem: make more memory available, or pre-empt the auto-kill by killing apps yourself.

Making More Memory Available

There are four ways to make more memory available short of exchanging the phone for a more powerful one.

1) Free up more app storage / internal storage

[s]Either uninstall the apps altogether, or move2sd as much as possible. Keep in mind move2SD may not work for all apps, and amount that can be freed varies greatly. Uninstall an app is best, as it both frees up the space itself takes, and if it loads a service, that service is loaded either, saving even more space. [/s]

While it's true that the app that wasn't run won't take up any space, every widget is served by a service, and a small app can load a HUGE service by calling existing libraries and declare a large buffer for downloads. And just because you don't actually use the app doesn't mean the system will not load it. The only way to make sure the app will NOT be loaded is to uninstall it (or if you have Titanium Backup premium, you can "freeze" the app)

2) VMHeap

VMHeap adjusts the the amount of memory that can be dedicated to the Dalvik Virtual Machine (VM). In general this should not be touched, and does not really make more memory available. It is available only for experimentation purposes.

This usually is NOT tweakable without mod ROM such as Cyanogen Mod. And benefits are unproven so far. Don't change anything yet.

3) CompCache

CompCache, or "compressed cache", is handled by the Linux kernel. It takes a portion of your memory, and use it as a cache space, but compressed. By using on-the-fly compression it is able to make your memory appear to be a bit larger than it actually is. However, the result is slower performance.

memory usage of my app in Android

noreply@blogger.com (Anonymous) — Sat, 7 Dec 2013 01:55:00 -0800

Memory usage on modern operating systems like Linux is an extremely complicated and difficult to understand area. Android apps have more memory available to them than ever before, but are you sure you're using it wisely? This talk will cover the memory management and explore tools and techniques for profiling the memory usage of Android apps.

first question is in my mind is how to know application heap size available to an Android app. ofcourse it may vary on the basis of phone device.

How much heap can my app use before a hard error is triggered? And
How much heap should my app use, given the constraints of the Android OS version and hardware of the user's device?

For item 1 above: maxMemory()

which can be invoked (e.g., in your main activity's onCreate() method) as follows:

Runtime rt = Runtime.getRuntime();
long maxMemory = rt.maxMemory();
Log.v("onCreate", "maxMemory:" + Long.toString(maxMemory));

This method tells you how many total bytes of heap your app is allowed to use.

For item 2 above: getMemoryClass()

which can be invoked as follows:

ActivityManager am = (ActivityManager) getSystemService(ACTIVITY_SERVICE);
int memoryClass = am.getMemoryClass();
Log.v("onCreate", "memoryClass:" + Integer.toString(memoryClass));

This method tells you approximately how many megabytes of heap your app should use if it wants to be properly respectful of the limits of the present device, and of the rights of other apps to run without being repeatedly forced into the onStop() / onResume() cycle as they are rudely flushed out of memory while your elephantine app takes a bath in the Android jacuzzi.

At runtime, the heap grows dynamically in size as the Dalvik VM requests system memory from the operating system. The Dalvik VM typically starts by allocating a relatively small heap. Then after each GC run it checks to see how much free heap memory there is. If the ratio of free heap to total heap is too small, the Dalvik VM will then add more memory to the heap (up to the maximum configured heap size).

Memory Anatomy

PSS : Proportional Set Size

Amount of memory shared with other processes, account in a way that the amount is divided evenly between the processes that share it. This is memory that would not be released if the process was terminated, but is indicative of the amount that this process is “contribution” to overall memory load.

USS : Unique Set Size

USS is the set of pages that are unique to a process. This is the amount of memory that would be freed if the application gets terminated.

Heap

Runtime memory available for allocation (Used by applications, services, daemons)

Dalvik Heap

The dalvik heap is preloaded with classes and data by zygote.

Commands for run time memory usage

To spit out a bunch of information about the memory use of each JAVA process

$adb shell dumpsys meminfo

To see memory for particular process: ( e.g. System)

$adb shell dumpsys meminfo system

Summary of the overall memory

$adb shell cat /proc/meminfo

Run Time Memory: Gallery Use Case

Rum time memory usage will depends on kind of use case running on android system. Following are the context where – run time memory will be allocated by android system.

All java process will run with the instance of DVM. DVM will again have its own run time heap requirement based on java application code complexity.
Binder IPC will allocate run time memory for marshalling objects
Service or daemon will allocated run time memory for internal usage
Following example will show run time heap change with respect to gallery application use case.

Size : Total size of particular heap
Allocated : Portion of heap is allocated to process
Native Usage : Usage by application or service code
Dalvik Usage : Usage by Dalvik virtual machine (libdvm)

Android's inbult API ActivityManager.getProcessMemoryInfo return few more interesting numbers.

The Pss number is a metric the kernel computes that takes into account memory sharing -- basically each page of RAM in a process is scaled by a ratio of the number of other processes also using that page. This way you can (in theory) add up the pss across all processes to see the total RAM they are using, and compare pss between processes to get a rough idea of their relative weight.

The other interesting metric here is PrivateDirty, which is basically the amount of RAM inside the process that can not be paged to disk (it is not backed by the same data on disk), and is not shared with any other processes. Another way to look at this is the RAM that will become available to the system when that process goes away (and probably quickly subsumed into caches and other uses of it).

Android added this crucial topic in detail. http://developer.android.com/training/articles/memory.html

Here is nice youtube link http://www.youtube.com/watch?v=_CruQY55HOk

Five years at infoedge india ltd.

noreply@blogger.com (Anonymous) — Fri, 6 Dec 2013 23:38:00 -0800

I would like to take this opportunity to talk about a great milestone here at InfoEdge. I complete 5 years of my journey at this company.A journey filled with stories of struggles,success,fights,fun & joy. This journey is made even more memorable,just because of the great co-operation that each one of you always gave. I take pride in THANKING each and every one of you here for the support.I have innumerous moments that make experiences with many ups and downs.But with each experience,whether good or bad,I strongly feel connected.Because the learning cycle never stops.

I have learned from everything,from my mistakes to my appreciation.Of course there were differences,fights & issues but thats all healthy in the quest for success.I intend to continue to support new upcoming leadership talent as we begin our new quest for a great new tomorrow.

Obviously,I intend to carry new burdens and responsibilities too.Just to learn more & get each moment with new excitement in this journey.A JOURNEY THAT WILL NEVER END.

Once again thanks alot!!!

R.I.P Codeigniter

noreply@blogger.com (Anonymous) — Wed, 6 Nov 2013 23:30:00 -0800

See my tweet https://twitter.com/raviraj4u/status/398343198151430144 It's really sad that
EllisLab stop support for codeigniter. CI is very fantastic framework and need to take advantages of PHP's upcoming releases which are more mature and stable. Well i have stopped working on CI since last an year ( currently i am working on Android native app development) but i loved it very much and proudly says that i was involved with CI at core level. Here are few links which shows my contribution to codeigniter.

http://ravirajsblog.blogspot.in/search?q=codeigniter

http://ellislab.com/forums/search_results/569e629d27bb6a8e70353d392cd0d1f8/

http://ellislab.com/forums/member/76560/

Even some developer made plugins using my CI code. http://getsparks.org/packages/login-cookie/versions/HEAD/show

I wish that it should go with safe hand. Bye bye Codeigniter !!

Google Developers Blog: Google I/O 2013: For the developers

noreply@blogger.com (Anonymous) — Sat, 18 May 2013 06:38:00 -0700

Google Developers Blog: Google I/O 2013: For the developers: By Scott Knaster, Google Developers Blog Editor “Google I/O is an annual developer conference featuring highly technical, in-depth sessi...

Hybrid app development PhoneGap

noreply@blogger.com (Anonymous) — Tue, 9 Apr 2013 19:27:00 -0700

Creating mobile apps is a trend nowadays. we've chosen PhoneGAP for creating mobile apps to look at more closely. So, let’s start. but big question is should we use native app or hybrid app or HTML5 based webapp.

PhoneGap application is a “native-wrapped” web application. Let’s explore how the web application is “wrapped”.

Many native mobile development SDKs provide a web browser widget (a “web view”) as a part of their UI framework (iOS and Android, for example). In purely native applications, web view controls are used to display HTML content either from a remote server, or local HTML packaged along with the native application in some way. The native “wrapper” application generated by PhoneGap loads the end developer’s HTML pages into one of these web view controls, and displays the resulting HTML as the UI when the application is launched.

If JavaScript files are included in a page loaded by a web view, this code is evaluated on the page as normal. However, the native application which creates the web view is able to (in different ways, depending on the platform) asynchronously communicate with JavaScript code running inside of the web view. This technology is usually referred to as “the bridge” in the context of PhoneGap architecture – the “bridge” means something slightly different in Titanium, as we will see later.

PhoneGap takes advantage of this to create a JavaScript API inside a web view which is able to send messages to and receive messages from native code in the wrapper application asynchronously. The way the bridge layer is implemented is different per platform, but on iOS, when you call for a list of contacts, your native method invocation goes into a queue of requests to be sent over the bridge. PhoneGap will then create an iframe which loads a URI scheme (“gap://”) that the native app is configured to handle, at which point all the queued commands will be executed. Communication back into the web view is done by evaluating a string of JavaScript in the context of the web view from native code.
There is much more to PhoneGap than that, but the messaging from web view to native code via the bridge implementation is the key piece of technology which allows local web applications to call native code.There’s quite a bit happening behind the scenes in a Titanium application. But basically, at runtime, your application consists of three major components – your JavaScript source code (inlined into a Java or Objective-C file and compiled as an encoded string), the platform-specific implementation of the Titanium API in the native programming language, and a JavaScript interpreter that will be used to evaluate your code at runtime (V8 (default) or Rhino for Android, or JavaScriptCore for iOS). Except in the browser, of course, where the built-in JavaScript engine will be used.

So major differences are:

PhoneGap:

JavaScript API that provides access to Native Functions
Supports HTML5/CSS3
Supports Web Standards & Re-use Across Enterprise Apps
Supports DOM based JavaScript Libraries/Frameworks
Supports the most platforms

Appcelerator Titanium:

JavaScript API that provides access to Native Functions
Compiles to Native Code
Could provide better performance.

Issues:

Of course, both Titanium and Phonegap fall into the category of “hybrid” but the key difference is implementation: while Phonegap application runs inside browser, Titanium’s app runs inside javascript interpreter. Runtime performances are SLOWER than native code because it's using a javascript engine as a bridge. Especially with a big TableView, it's much more slower, and the feeling is just not the same.

I have build two diff applications based on phonegap for testing purpose and i found that both are very slow as compare with browser rendering. When i cross checked same application in my phone's browsers, it looks much responsive than in app. Might be current browsers are using hardware acceleration by default. I have used mobile Jquery & phone gap APIs for demo.

I know that jquery mobile can be a little slow by nature, imagine my surprise and disappointment when my compiled phonegap app UI is magnitudes slower running on my phone than running in the phone's native browser. How can the compiled app be so noticeably less responsive than the same app running in the phone's native browser? The responsiveness is completely unacceptable.

http://apachecordova.blogspot.in/2012/11/who-is-murdering-phonegap-its-jquery.html

Even phonegap provide better solution to use JS Framework.

http://cordova.codeplex.com/

Back to blog : understanding ppi,dpi,mdpi, hdpi, xhdpi

noreply@blogger.com (Anonymous) — Sat, 23 Mar 2013 10:56:00 -0700

Sorry guys !!! was little bit busy with Quora :-) https://www.quora.com/Ravi-Raj-7

Anyway A man who is losing his house in the morning and reach in evening in house you can not say him the loser :D
Actually in last one -two months i am working on mobile site and apps and found very interesting things. I will share my experience here in my coming posts. So enjoy reading and be sure to leave comments here.

In Feb 1993, Marc Andreessen had requested for new HTML tag, the proposal was proposal for image basically about html tag. We have seen drastically improvements in web & HTML since last few years. People ( http://movethewebforward.org/ , http://www.sultansofspeed.com/ , http://web-performance.meetup.com/all/ ) are contributing to make web faster. Accenture like companies started campaign with slogan "High Performance. Delivered".

According to the HTTP Archive http://httparchive.org/interesting.php,the average web page is 1292 KB, with 801 KB of that page weight — more than 60% — being taken up by images. In shiksha its around 50 -60%. we have seen many algorithms for image optimization http://www.netmagazine.com/features/best-image-compression-tools

Then i found WebP :-) really i appreciate Google Engg. who are trying to make our life easier ...

So here are my findings.

Lossy and lossless compression
Transparency (alpha channel)
Great compression for photos
Animation support
Metadata
Color profiles

Gmail, Drive, Picasa, Instant Previews, Play Magazines, Image Search, YouTube, ...) with WebP support. Most recently, Chrome Web Store switched to WebP, saw ~30% byte reduction on average, and is now saving several terabytes of bandwidth per day!

Back to basic:
A digital photograph is made up of millions of tiny dots called pixels. For example, one camera might produce photos that are 2272 pixels wide and 1704 pixels tall (2272 x 1704). Another camera will produce an image that is 4492 x 3328. You can find out the number of megapixels by multiplying the horizontal and vertical pixels. In the first example, the camera captures about 3.9 megapixels (2272 x 1704 = 3,871,488). In the second example, the camera captures about 15 megapixels (4492 x 3328 = 14,949,376).

Computer monitor generally displays images at 72 pixels per inch. This means that our 3.8 megapixel image is going to measure about 32 inches by 24 inches when viewed on a monitor. We can determine the display size of the image by dividing the horizontal and vertical pixels by 72.

In this case, 2272 / 72 = 31.6 and 1704 / 72 = 23.7.
Use the 72ppi standard when you want to post an image to the Internet.

But printer needs to use more pixels per inch to produce a high-quality image than our monitor does. If print a photo at 100ppi, it is not going to look like a professional print. You will be able to see grain and fuzziness, the actual pixels that make up the digital photograph. If we print at 250ppi, it increase the pixels per inch, so it reduce the size of the printed photo. Let's say you print 3.9 megapixel photo at 100ppi. This isn't high-quality, but you can print a photo that measures 22.7 by 17 inches (2272 / 100 = 22.7 and 1704 / 100 = 17).

Now you print the same photo at 250ppi. You get a great looking photo, but the print size is 9.1 by 6.8 inches (2272 / 250 = 9.1 and 1704 / 250 = 6.8).

Hope you enjoyed little bit maths that we did above but hold on ... Now screens are changed ..

iOS devices measure density in PPI (pixels per inch) and Android in DPI (dots per inch). The more pixels or dots you fit in one square inch on a screen, the higher the density and resolution of it.
The original iPhones and iPads had a screen density that was classified as non-retina. The current generation of iOS devices sport higher density displays referred to as retina. Android devices have evolved from low density, ldpi, all the way to extra high density, xhdpi.

There are five widely used densities across iOS and Android devices, which fall into four progressively larger groups:

non-retina (iOS) and mdpi (Android)
hdpi (Android)
retina (iOS)
xhdpi (Android)

Scaling the UI elements of your design and understanding how an asset at one density would scale to another can be confusing at times.

According to the official Android “Supporting Multiple Screens” documentation:

xhdpi = 320dpi
hdpi = 240dpi
mdpi = 160dpi
ldpi = 120dpi
Retina iPhones = 326dpi (roughly equivalent to xhdpi)
Retina iPads = 264dpi (roughly equivalent to hdpi)

See more details here. http://developer.android.com/guide/practices/screens_support.html

So we have seen here some basic ABC about images pixels, ppi and dpi. So image can be render with diff resolutions in diff screen density.

Will back with more details about responsive image rendering with all possible options. Till then enjoy reading and happy learning !!!

is HTML5 slow ?

noreply@blogger.com (Anonymous) — Fri, 22 Feb 2013 20:30:00 -0800

http://www.quora.com/HTML5/Is-HTML5-really-slower-on-mobile-devices/answer/Ravi-Raj-7/quote/309549

Asynchronous JS loading without blocking onload

noreply@blogger.com (Anonymous) — Wed, 7 Nov 2012 19:01:00 -0800

Asynchronous JS is best way to load js file in your HTML page but it still blocks window.onload event (except in IE before version 10).

checkout here how onload blocked even we use asyn js loading techniques

http://www.stevesouders.com/blog/2012/01/13/javascript-performance/

http://calendar.perfplanet.com/2010/the-truth-about-non-blocking-javascript/

So here is another solution for same :-)

create an iframe without setting src to a new URL. This fires onload of the iframe immediately and the whole thing is completely out of the way

style the iframe to make it invisible

get the last script tag so far, which is the snippet itself. This is in order to glue the iframe to the snippet that includes it.

insert the iframe into the page

get a handle to the document object of the iframe

write some HTML into that iframe document

this HTML includes the desired script

Code:

(function(url){
var iframe = document.createElement('iframe');
(iframe.frameElement || iframe).style.cssText = "width: 0; height: 0; border: 0";
var where = document.getElementsByTagName('script');
where = where[where.length - 1];
where.parentNode.insertBefore(iframe, where);
var doc = iframe.contentWindow.document;
doc.open().write('<body onload="'+
'var js = document.createElement(\'script\');'+
'js.src = \''+ url +'\';'+
'document.body.appendChild(js);">');
doc.close();
})('http://www.jspatterns.com/files/meebo/asyncjs1.php');

Issues:

1. Avoid SSL warnings: iframe.src defaults to “about:blank” in IE6, which it then treats as insecure content on HTTPS pages. We found that initializing iframe.src to “javascript:false”.

2. Avoid crossdomain exceptions: anonymous iframe access will throw exceptions if the host page changed the document.domain value in IE. The original Meebo code falls back to a “javascript:” URL when this happens.

3. The script (asyncjs1.php) runs is in an iframe, so all document and window references point to the iframe, not the host page.There's an easy solution for that without changing the whole script. Just wrap it in an immediate function and pass the document object the script expects:

(function(document){
document.getElementById('r')... // all fine
})(parent.document);

4. The script works fine in Opera, but blocks onload. Opera is weird here. Even regular async scripts block DOMContentLoaded which is a shame.

seems below code solves our problem .. try it and let me know the results ...

https://github.com/pablomoretti/jcors-loader

PHP's register_shutdown_function

noreply@blogger.com (Anonymous) — Tue, 6 Nov 2012 18:13:00 -0800

PHP has approx. 5800 functions defined in global space .. Uff .. that is one cause that people does not consider it as thoughtful language but these APIs work as strong enough tool to accomplish various task and it provide tremendous functionality to end user.

Today we will discuss one magical function named "register_shutdown_function".

function allows you to execute a block of code whenever your script ends for any reason.
Whether your page exit()s or die()s or just finishes, a developer has a hook to run whatever code he/she deems necessary. And not just one function either… you can use this call to register as many shutdown functions as you want, and they will get executed in the order that they get applied. But of course, you must be careful: PHP will happily give you more rope than you will ever need to hang yourself. A lot of people may consider the use of this function to be magic, and you’ll want to be very clear that what you’re doing is documented.

Use of this function is very straight-forward.

I just tested with Apache, PHP being used as Apache module. I created an endless loop like this:

class X
{
function __destruct()
{
$fp = fopen("/var/www/dtor.txt", "w+");
fputs($fp, "Destroyed\n");
fclose($fp);
}
};

$obj = new X();
while (true) {
// do nothing
}

Here's what I found out:-

pressing STOP button in Firefox does not stop this script
If I shut down Apache, destructor does not get called
It stops when it reaches PHP max_execution_time and destuctor does not get called

However, doing this:

function shutdown_func() {
$fp = fopen("/var/www/htdocs/dtor.txt", "w+");
fputs($fp, "Destroyed2\n");
fclose($fp);
}

register_shutdown_function("shutdown_func");

while (true) {
// do nothing
}

shutdown_func gets called. So this means that class destuctor is not that good as shutdown functions. :-)

Enjoy !! :-)

Things web developers must know

noreply@blogger.com (Anonymous) — Tue, 30 Oct 2012 19:36:00 -0700

The idea here is that most of us should already know most of what is on this list. But there just might be one or two items you haven't really looked into before, don't fully understand, or maybe never even heard of.

Interface and User Experience

Be aware that browsers implement standards inconsistently and make sure your site works reasonably well across all major browsers. At a minimum test against a recent Gecko engine (Firefox), a WebKit engine (Safari, Chrome, and some mobile browsers), your supported IE browsers (take advantage of the Application Compatibility VPC Images), and Opera. Also consider how browsers render your site in different operating systems.
Consider how people might use the site other than from the major browsers: cell phones, screen readers and search engines, for example. — Some accessibility info: WAI and Section508, Mobile development: MobiForge.
Staging: How to deploy updates without affecting your users. Ed Lucas's answer has some comments on this.
Don't display unfriendly errors directly to the user.
Don't put users' email addresses in plain text as they will get spammed to death.
Add the attribute rel="nofollow" to user-generated links to avoid spam.
Build well-considered limits into your site - This also belongs under Security.
Learn how to do progressive enhancement.
Redirect after a POST if that POST was successful, to prevent a refresh from submitting again.
Don't forget to take accessibility into account. It's always a good idea and in certain circumstances it's a legal requirement. WAI-ARIA and WCAG 2 are good resources in this area.

Security

It's a lot to digest but the OWASP development guide covers Web Site security from top to bottom.
Know about SQL injection and how to prevent it.
Never trust user input nor anything else that comes in the request (which includes cookies and hidden form field values!).
Hash passwords using salt to prevent rainbow attacks. Use a slow hashing algorithm, such as bcrypt (time tested) or scrypt (even stronger, but newer) (1, 2), for storing passwords. (How To Safely Store A Password)
Don't try to come up with your own fancy authentication system. It's such an easy thing to get wrong in subtle and untestable ways and you wouldn't even know it until after you're hacked.
Know the rules for processing credit cards. (See this question as well)
Use SSL/HTTPS for login and any pages where sensitive data is entered (like credit card info).
How to resist session hijacking.
Avoid cross site scripting (XSS).
Avoid cross site request forgeries (XSRF).
Keep your system(s) up to date with the latest patches.
Make sure your database connection information is secured.
Keep yourself informed about the latest attack techniques and vulnerabilities affecting your platform.
Read The Google Browser Security Handbook.
Read The Web Application Hacker's Handbook.

Performance

Implement caching if necessary, understand and use HTTP caching properly as well as HTML5 Manifest.
Optimize images - don't use a 20 KB image for a repeating background.
Learn how to gzip/deflate content (deflate is better).
Combine/concatenate multiple stylesheets or multiple script files to reduce number of browser connections and improve gzip ability to compress duplications between files.
Take a look at the Yahoo Exceptional Performance site, lots of great guidelines including improving front-end performance and their YSlow tool. Google page speed is another tool for performance profiling. Both require Firebug to be installed.
Use CSS Image Sprites for small related images like toolbars (see the "minimize HTTP requests" point)
Busy web sites should consider splitting components across domains. Specifically...
Static content (i.e. images, CSS, JavaScript, and generally content that doesn't need access to cookies) should go in a separate domain that does not use cookies, because all cookies for a domain and its subdomains are sent with every request to the domain and its subdomains. One good option here is to use a Content Delivery Network (CDN).
Minimize the total number of HTTP requests required for a browser to render the page.
Utilize Google Closure Compiler for JavaScript and other minification tools.
Make sure there’s a favicon.ico file in the root of the site, i.e. /favicon.ico. Browsers will automatically request it, even if the icon isn’t mentioned in the HTML at all. If you don’t have a/favicon.ico, this will result in a lot of 404s, draining your server’s bandwidth.

SEO (Search Engine Optimization)

Use "search engine friendly" URLs, i.e. use example.com/pages/45-article-title instead ofexample.com/index.php?page=45
When using # for dynamic content change the # to #! and then on the server$_REQUEST["_escaped_fragment_"] is what googlebot uses instead of #!. In other words,./#!page=1 becomes ./?_escaped_fragments_=page=1. Also, for users that may be using FF.b4 or Chromium, history.pushState({"foo":"bar"}, "About", "./?page=1"); Is a great command. So even though the address bar has changed the page does not reload. This allows you to use ? instead of #! to keep dynamic content and also tell the server when you email the link that we are after this page, and the AJAX does not need to make another extra request.
Don't use links that say "click here". You're wasting an SEO opportunity and it makes things harder for people with screen readers.
Have an XML sitemap, preferably in the default location /sitemap.xml.
Use when you have multiple URLs that point to the same content, this issue can also be addressed from Google Webmaster Tools.
Use Google Webmaster Tools and Bing Webmaster Tools.
Install Google Analytics right at the start (or an open source analysis tool like Piwik).
Know how robots.txt and search engine spiders work.
Redirect requests (using 301 Moved Permanently) asking for www.example.com to example.com(or the other way round) to prevent splitting the google ranking between both sites.
Know that there can be badly-behaved spiders out there.
If you have non-text content look into Google's sitemap extensions for video etc. There is some good information about this in Tim Farley's answer.

Technology

Understand HTTP and things like GET, POST, sessions, cookies, and what it means to be "stateless".
Write your XHTML/HTML and CSS according to the W3C specifications and make sure theyvalidate. The goal here is to avoid browser quirks modes and as a bonus make it much easier to work with non-standard browsers like screen readers and mobile devices.
Understand how JavaScript is processed in the browser.
Understand how JavaScript, style sheets, and other resources used by your page are loaded and consider their impact on perceived performance. It may be appropriate in some cases to move scripts to the bottom of your pages.
Understand how the JavaScript sandbox works, especially if you intend to use iframes.
Be aware that JavaScript can and will be disabled, and that AJAX is therefore an extension, not a baseline. Even if most normal users leave it on now, remember that NoScript is becoming more popular, mobile devices may not work as expected, and Google won't run most of your JavaScript when indexing the site.
Learn the difference between 301 and 302 redirects (this is also an SEO issue).
Learn as much as you possibly can about your deployment platform.
Consider using a Reset Style Sheet.
Consider JavaScript frameworks (such as jQuery, MooTools, Prototype, Dojo or YUI 3), which will hide a lot of the browser differences when using JavaScript for DOM manipulation.
Taking perceived performance and JS frameworks together, consider using a service such as theGoogle Libraries API to load frameworks so that a browser can use a copy of the framework it has already cached rather than downloading a duplicate copy from your site.
Don't reinvent the wheel. Before doing ANYTHING search for a component or example on how to do it. There is a 99% chance that someone has done it and released an OSS version of the code.

Bug fixing

Understand you'll spend 20% of your time coding and 80% of it maintaining, so code accordingly.
Set up a good error reporting solution.
Have a system for people to contact you with suggestions and criticisms.
Document how the application works for future support staff and people performing maintenance.
Make frequent backups! (And make sure those backups are functional) Ed Lucas's answer has some advice. Have a restore strategy, not just a backup strategy.
Use a version control system to store your files, such as Subversion, Mecurial or Git.
Don't forget to do your Acceptance Testing. Frameworks like Selenium can help.
Make sure you have sufficient logging in place using frameworks such as log4j, log4net or log4r. If something goes wrong on your live site, you'll need a way of finding out what.
When logging make sure you're capture both handled exceptions, and unhandled exceptions. Report/analyse the log output, as it'll show you where the key issues are in your site.

Lots of stuff omitted not necessarily because they're not useful answers, but because they're either too detailed, out of scope, or go a bit too far for someone looking to get an overview of the things they should know. If you're one of those people you can read the rest of the answers to get more detailed information about the things mentioned in this list. If I get the time I'll add links to the various answers that contain the things mentioned in this list if the answers go into detail about these things. Please feel free to edit this as well, I probably missed some stuff or made some mistakes.

original source: stackoverflow.com

What are some good resources to learn about mobile

noreply@blogger.com (Anonymous) — Tue, 9 Oct 2012 10:07:00 -0700

Here is my answer @ Quora ...

http://www.quora.com/Mobile-Applications/What-are-some-good-resources-to-learn-about-mobile/answer/Ravi-Raj-7

Checkout my profile @ Quora http://www.quora.com/Ravi-Raj-7

AJAY BHAGESWAR NARKAR SIR PHYSICS TEACHER

noreply@blogger.com (Anonymous) — Mon, 8 Oct 2012 12:09:00 -0700

दो लम्हे का जीवन है, एक क्षण उन्माद का, एक क्षण आह्लाद का, बस इतना ही जियूँगा

"Maut kitne rang badle, dhang badle,

Jab talak zinda kalam hai hum

Tumhe marne na denge"....

Today i heard that Ajay sir has no more ... Few people said that he was suffering

from cancer but other said that he under went an operation.

It's Very sad for such a young life with so much potential.

He leaves behind his wife and 2 daughters.

Its true that he was a bit arrogant and pompous at first glance but His

life story was full of persistent struggle and no doubt he was the most selfless

and bright teacher in Kanpur. He was more than a teacher of physics, in fact

he was a teacher of life. I really admire him as a person and love his soul in the true sense.

My deepest sympathy is there with his family.

I'm glad to see how much respect for him is there among

his students.

He is still alive in us showing the right path.

He always said that believe in God that will bring you all

the successes and strength in life.

In 1997-98, I went to get admission in his coaching

classes.

Those days, He took Physics classes at Mahesh

Chauhan's(IIT MATHS Teacher) house.

First time I saw him, classic old style spectacles and

woodland shoes,shirt and Lee jeans.

On that day he learnt so many things. I still

remember how he denied parents for admission

because boy's father was chewing PAAN.

He was music lover, specially Kishore Kumar,

Still remember his song "musafir ho yarro".

One day He sent his wife to teach us Physics

and another day his daughter wrote a typical physics

formula on board.

Classes's pin drop silence is still in my mind.

On that time he used to mark signature in IE Irodov

book's last page as a recieiving of 2nd instalment of

tuition fee.( Still i have that book and signature).

He was crazy personality and self made figure.

He had holds record of slapping up to 1200 student and teaching 20 hours in a day.

He used to present self as A GOD among students and give cusses to them like devil.

He is obsessed with self praising and mechanics. According to him, you can't play Holi

or go to movie because playing Holi or shopping or watching cinema for 2 hours will stop

you from getting into IIT.

I was slapped once by Sir. Still remember that day, whole class was punished for three hours,

after class we both discussed, He learnt me a new way to watch life.

His few famous words are:-

"Hum hum hain,baki pani kam hain..."
"Baangduon assignment pura hona chaiye , bhai logo kya bolte ho..."
"Ghansu PAAN wale ka BETA IIT me aa gaya ..."
"MOTI, Angothi ya mala se koi IIT me nahi ata ..."
"Maine teen saal se Diwali aur Dhasahara nahi manaya..."
"JO Log NAVRATRE KA 9 DIN KA FAST HAI VO KABHI IIT ME NAHI AYNGE..."

The Famous one is:

"Physics to ek bahaana hai tmhe insaan jo bnana hai".

He was real HERO.. Still when we think and trying to solve typical equations of life...

He helped us ... at least HE was Hero for students who learn Physics from him (1994 - OCT 2011) ....

Zindagi to bewafa hai ek din thukrayegi,

maut mehbooba hai

apne saath lekar jayegi.

mar ke jeene ke ada jo duniya ko dikhlayega

wo muqaddar ks sikandar ! janeman Kah layaga ...

http://www.youtube.com/watch?v=oNg9volIkYU

how to write good lines of code

noreply@blogger.com (Anonymous) — Tue, 25 Sep 2012 12:10:00 -0700

This is a daunting task indeed, and there's a lot of ground to cover. So I'm humbly suggesting this as somewhat comprehensive guide for your team, with pointers to appropriate tools and educational material.
Remember: These are guidelines, and that as such are meant to adopted, adapted, or dropped based on circumstances.
Beware: Dumping all this on a team at once would most likely fail. You should try to cherry-pick elements that would give you the best bang-for-sweat, and introduce them slowly, one at a time.
Note: not all of this applies directly to Visual Programming Systems like G2. For more specific details on how to deal with these, see the Addendum section at the end.

Executive Summary for the Impatient

Define a rigid project structure, with:
- project templates,
- coding conventions,
- familiar build systems,
- and sets of usage guidelines for your infrastructure and tools.
Install a good SCM and make sure they know how to use it.
Point them to good IDEs for their technology, and make sure they know how to use them.
Implement code quality checkers and automatic reporting in the build system.
Couple the build system to continuous integration and continuous inspection systems.
With the help of the above, identify code quality "hotspots" and refactor.

Now for the long version... Caution, brace yourselves!

Rigidity is (Often) Good

This is a controversial opinion, as rigidity is often seen as a force working against you. It's true for some phases of some projects. But once you see it as a structural support, a framework that takes away the guesswork, it greatly reduces the amount of wasted time and effort. Make it work for you, not against you.
Rigidity = Process / Procedure.
Software development needs good process and procedures for exactly the same reasons that chemical plants or factories have manuals, procedures, drills and emergency guidelines: preventing bad outcomes, increasing predictability, maximizing productivity...
Rigidity comes in moderation, though!!

Rigidity of the Project Structure

If each project comes with its own structure, you (and newcomers) are lost and need to pick up from scratch every time you open them. You don't want this in a professional software shop, and you don't want this in a lab either.

Rigidity of the Build Systems

If each project looks different, there's a good chance they also build differently. A build shouldn't require too much research or too much guesswork. You want to be able to do the canonical thing and not need to worry about specifics: configure; make install, ant, mvn install, etc...
Re-using the same build system and making it evolve over the time also ensures a consistent level of quality.
You do need a quick README to point the project's specifics, and gracefully guide the user/developer/researcher, if any.
This also greatly facilitates other parts of your build infrastructure, namely:

So keep your build (like your projects) up to date, but make it stricter over time, and more efficient at reporting violations and bad practices.
Do not reinvent the wheel, and reuse what you have already done.
Recommended Reading:

Continuous Integration: Improving Software Quality and Reducing Risk (Duval, Matyas, Glover, 2007)
Continuous Delivery: Release Sotware Releases through Build, Test and Deployment Automation (Humble, Farley, 2010)

Rigidity in the Choice of Programming Languages

You can't expect, especially in a research environment, to have all teams (and even less all developers) use the same language and technology stack. However, you can identify a set of "officially supported" tools, and encourage their use. The rest, without a good rationale, shouldn't be permitted (beyond prototyping).
Keep your tech stack simple, and the maintenance and breadth of required skills to a bare minimum: a strong core.

Rigidity of the Coding Conventions and Guidelines

Coding conventions and guidelines are what allow you to develop both an identity as a team, and a shared lingo. You don't want to err into terra incognita every time you open a source file.
Nonsensical rules that make life harder or forbid actions explicity to the extent that commits are refused based on single simple violations are a burden. However:

a well thought-out ground ruleset takes away a lot of the whining and thinking: nobody should break under no circumstances;
and a set of recommended rules provide additional guidance.

Personal Approach: I am aggressive when it comes to coding conventions, some even say nazi, because I do believe in having a lingua franca, a recognizable style for my team. When crap code gets checked-in, it stands out like a cold sore on the face of an Hollywood star: it triggers a review and an action automatically. In fact, I've sometimes gone as far as to advocate the use of pre-commit hooks to reject non-conformimg commits. As mentioned, it shouldn't be overly crazy and get in the way of productivity: it should drive it. Introduce these slowly, especially at the beginning. But it's way preferable over spending so much time fixing faulty code that you can't work on real issues.

Some languages even enforce this by design:

Java was meant to reduce the amount of dull crap you can write with it (though no doubt many manage to do it).
Python's block structure by indentation is another idea in this sense.
Go, with its gofmt tool, which completely takes away any debate and effort (and ego!!) inherent to style: run gofmt before you commit.

Make sure that code rot cannot slip through. Code conventions, continuous integration and continuous inspection, pair programming and code reviews are your arsenal against this demon.
Plus, as you'll see below, code is documentation, and that's another area where conventions encourage readability and clarity.

Rigidity of the Documentation

Documentation goes hand in hand with code. Code itself is documentation. But there must be clear-cut instructions on how to build, use, and maintain things.
Using a single point of control for documentation (like a WikiWiki or DMS) is a good thing. Create spaces for projects, spaces for more random banter and experimentation. Have all spaces reuse common rules and conventions. Try to make it part of the team spirit.
Most of the advice applying to code and tooling also applies to documentation.

Rigidity in Code Comments

Code comments, as mentioned above, are also documentation. Developers like to express their feelings about their code (mostly pride and frustration, if you ask me). So it's not unusual for them to express these in no uncertain terms in comments (or even code), when a more formal piece of text could have conveyed the same meaning with less expletives or drama. It's OK to let a few slip through for fun and historical reasons: it's also part of developing a team culture. But it's very important that everybody knows what is acceptable and what isn't, and that comment noise is just that: noise.

Rigidity in Commit Logs

Commit logs are not an annoying and useless "step" of your SCM's lifecycle: you DON'T skip it to get home on time or get on with the next task, or to catch up with the buddies who left for lunch. They matter, and, like (most) good wine, the more time passes the more valuable they become. So DO them right. I'm flabbergasted when I see co-workers writing one-liners for giant commits, or for non-obvious hacks.
Commits are done for a reason, and that reason ISN'T always clearly expressed by your code and the one line of commit log you entered. There's more to it than that.
Each line of code has a story, and a history. The diffs can tell its history, but you have to write its story.

Why did I update this line? -> Because the interface changed.
Why did the interface changed? -> Because the library L1 defining it was updated.
Why was the library updated? -> Because library L2, that we need for feature F, depended on libary L1.
And what's feature X? -> See task 3456 in issue tracker.

It's not my SCM choice, and may not be the best one for your lab either; but Git gets this right, and tries to force you to write good logs more than most other SCMs systems, by using short logs and long logs. Link the task ID (yes, you need one) and a leave a generic summary for the shortlog, and expand in the long log: write the changeset's story.
It is a log: It's here to keep track and record updates.

Rule of Thumb: If you were searching for something about this change later, is your log likely to answer your question?

Projects, Documentation and Code Are ALIVE

Keep them in sync, otherwise they do not form that symbiotic entity anymore. It works wonders when you have:

clear commits logs in your SCM, w/ links to task IDs in your issue tracker,
where this tracker's tickets themselves link to the changesets in your SCM (and possibly to the builds in your CI system),
and a documentation system that links to all of these.

Code and documentation need to be cohesive.

Rigidity in Testing

Rules of Thumb:

Any new code shall come with (at least) unit tests.

Any refactored legacy code shall come with unit tests.

Of course, these need:

to actually test something valuable (or they are a waste of time and energy),
to be well written and commented (just like any other code you check in).

They are documentation as well, and they help to outline the contract of your code. Especially if you use TDD. Even if you don't, you need them for your peace of mind. They are your safety net when you incorporate new code (maintenance or feature) and your watchtower to guard against code rot and environmental failures.
Of course, you should go further and have integration tests, and regression tests for each reproducible bug you fix.

Rigidity in the Use of the Tools

It's OK for the occasional developer/scientist to want to try some new static checker on the source, generate a graph or model using another, or implement a new module using a DSL. But it's best if there's a canonical set of tools that all team members are expected to know and use.
Beyond that, let members use what they want, as long as they are ALL:

productive,
NOT regularly requiring assistance
NOT regularly adjusting to your general infrastructure,
NOT disrupting your infrastructure (by modifying common areas like code, build system, documentation...),
NOT affecting others' work,
ABLE to timely perform any task requested.

If that's not the case, then enforce that they fallback to defaults.

Rigidity vs Versatility, Adapatability, Prototyping and Emergencies

Flexibility can be good. Letting someone occasionally use a hack, a quick-n-dirty approach, or a favorite pet tool to get the job done is fine. NEVER let it become a habit, and NEVER let this code become the actual codebase to support.

Team Spirit Matters

Develop a Sense of Pride in Your Codebase

Develop a sense of Pride in Code
- Use wallboards
  - leader board for a continuous integration game
  - wallboards for issue management and defect counting
- Use an issue tracker / bug tracker

Avoid Blame Games

DO use Continuous Integration / Continuous Inspection games: it fosters good-mannered and productive competition.
DO keep track defects: it's just good house-keeping.
DO identifying root causes: it's just future-proofing processes.
BUT DO NOT assign blame: it's counter productive.

It's About the Code, Not About the Developers

Make developers conscious of the quality of their code, BUT make them see the code as a detached entity and not an extension of themselves, which cannot be criticized.
It's a paradox: you need to encourage ego-less programming for a healty workplace but to rely on ego for motivational purposes.

From Scientist to Programmer

People who do not value and take pride in code do not produce good code. For this property to emerge, they need to discover how valuable and fun it can be. Sheer professionalism and desire to do good is not enough: it needs passion. So you need to turn your scientists into programmers (in the large sense).
Someone argued in comments that after 10 to 20 years on a project and its code, anyone would feel attachment. Maybe I'm wrong but I assume they're proud of the code's outcomes and of the work and its legacy, not of the code itself or of the act of writing it.
From experience, most researchers regard coding as a necessity, or at best as a fun distraction. They just want it to work. The ones who are already pretty versed in it and who have an interest in programming are a lot easier to persuade of adopting best practices and switching technologies. You need to get them halfway there.

Code Maintenance is Part of Research Work

Nobody reads crappy research papers. That's why they are peer-reviewed, proof-read, refined, rewritten, and approved time and time again until deemed ready for publication. The same applies to a thesis and a codebase!
Make it clear that constant refactoring and refreshing of a codebase prevents code rot and reduces technical debt, and facilitates future re-use and adaptation of the work for other projects.

Why All This??!

Why do we bother with all of the above? For code quality. Or is it quality code...?
These guidelines aim at driving your team towards this goal. Some aspects do it by simply showing them the way and letting them do it (which is much better) and others take them by the hand (but that's how you educate people and develop habits).
How do you know when the goal is within reach?

Quality is Measurable

Not always quantitatively, but it is measurable. As mentioned, you need to develop a sense of pride in your team, and showing progress and good results is key. Measure code quality regularly and show progress between intervals, and how it matters. Do retrospectives to reflect on what has been done, and how it made things better or worse.
There are great tools for continuous inspection. Sonar being a popular one in the Java world, but it can adapt to any technologies; and there are many others. Keep your code under the microscope and look for these pesky annoying bugs and microbes.

But What if My Code is Already Crap?

All of the above is fun and cute like a trip to Never Land, but it's not that easy to do when you already have (a pile of steamy and smelly) crap code, and a team reluctant to change.
Here's the secret: you need to start somewhere.

Personal anecdote: In a project, we worked with a codebase weighing originally 650,000+ Java LOC, 200,000+ lines of JSPs, 40,000+ JavaScript LOC, and 400+ MBs of binary dependencies.
After about 18 months, it's 500,000 Java LOC (MOSTLY CLEAN), 150,000 lines of JSPs, and 38,000 JavaScript LOC, with dependencies down to barely 100MBs (and these are not in our SCM anymore!).
How did we do it? We just did all of the above. Or tried hard.
It's a team effort, but we slowly inject in our process regulations and tools to monitor the heart-rate of our product, while hastily slashing away the "fat": crap code, useless dependencies... We didn't stop all development to do this: we have occasional periods of relative peace and quiet where we are free to go crazy on the codebase and tear it apart, but most of the time we do it all by defaulting to a "review and refactor" mode every chance we get: during builds, during lunch, during bug fixing sprints, during Friday afternoons...
There were some big "works"... Switching our build system from a giant Ant build of 8500+ XML LOC to a multi-module Maven build was one of them. We then had:

clear-cut modules (or at least it was already a lot better, and we still have big plans for the future),

automatic dependency management (for easy maintenance and updates, and to remove useless deps),

faster, easier and reproduceable builds,

daily reports on quality.

Another was the injection of "utility tool-belts", even though we were trying to reduce dependencies: Google Guava and Apache Commons slim down your code and and reduce surface for bugs in your code a lot.
We also persuaded our IT department that maybe using our new tools (JIRA, Fisheye, Crucible, Confluence, Jenkins) was better than the ones in place. We still needed to deal with some we despised (QC, Sharepoint and SupportWorks...), but it was an overall improved experience, with some more room left.
And every day, there's now a trickle of between one to dozens of commits that deal only with fixing and refactoring things. We occasionally break stuff (you need unit tests, and you better write them before you refactor stuff away), but overall the benefit for our morale AND for the product has been enormous. We get there one fraction of a code quality percentage at a time. And it's fun to see it increase!!!

Note: Again, rigidity needs to be shaken to make room for new and better things. In my anecdote, our IT department is partly right in trying to impose some things on us, and wrong for others. Or maybe they used to be right. Things change. Prove that they are better ways to boost your productivity. Trial-runs and prototypes are here for this.

The Super-Secret Incremental Spaghetti Code Refactoring Cycle for Awesome Quality

       +-----------------+      +-----------------+
       |  A N A L Y Z E  +----->| I D E N T I F Y |
       +-----------------+      +---------+-------+
                ^                           |
                |                           v
       +--------+--------+      +-----------------+
       |    C L E A N    +<----- f="f" font="font" i="i" x="x">

Once you have some quality tools at your toolbelt:

Analyze your code with code quality checkers.
Linters, static analyzers, or what have you.
Identify your critical hotspots AND low hanging fruits.
Violations have severity levels, and large classes with a large number of high-severity ones are a big red flag: as such, they appear as "hot spots" on radiator/heatmap types of views.
Fix the hotspots first.
It maximizes your impact in a short timeframe as they have the highest business value. Ideally, critical violations should dealt with as soon as they appear, as they are potential security vulnerabilities or crash causes, and present a high risk of inducing a liability (and in your case, bad performance for the lab).
Clean the low level violations with automated codebase sweeps.
It improves the signal-to-noise ratio so you are be able to see significant violations on your radar as they appear. There's often a large army of minor violations at first if they were never taken care of and your codebase was left loose in the wild. They do not present a real "risk", but they impair the code's readability and maintainability. Fix them either as you meet them while working on a task, or by large cleaning quests with automated code sweeps if possible. Do be careful with large auto-sweeps if you don't have a good test suite and integration system. Make sure to agree with co-workers the right time to run them to minimze the annoyance.
Repeat until you are satisfied.
Which, ideally, you should never be, if this is still an active product: it will keep evolving.

Quick Tips for Good House-Keeping

When in hotfix-mode, based on a customer support request:
- It's usually a best practice to NOT go around fixing other issues, as you might introduce new ones unwillingly.
- Go at it SEAL-style: get in, kill the bug, get out, and ship your patch. It's a surgical and tactical strike.
But for all other cases, if you open a file, make it your duty to:
- definitely: review it (take notes, file issue reports),
- maybe: clean it (style cleanups and minor violations),
- ideally: refactor it (reorganize large sections and their neigbors).

Just don't get sidetracked into spending a week from file to file and ending up with a massive changeset of thousands of fixes spanning multiple features and modules - it makes future tracking difficult. One issue in code = one ticket in your tracker. Sometimes, a changeset can impact multiple tickets; but if it happens too often, then you're probably doing something wrong.

Addendum: Managing Visual Programming Environments

The Walled Gardens of Bespoke Programming Systems

Multiple programming systems, like the OP's G2, are different beasts...

No Source "Code"
Often they do not give you access to a textual representation of your source "code": it might be stored in a proprietary binary format, or maybe it does store things in text format but hides them away from you. Bespoke graphical programming systems are actually not uncommon in research labs, as they simplify the automation of repetitive data processing workflows.
No Tooling
Aside from their own, that is. You are often constrained by their programming environment, their own debugger, their own interpreter, their own documentation tools and formats. They are walled gardens, except if they eventually capture the interest of someone motivated enough to reverse engineer their formats and builds external tools - if the license permits it.
Lack of Documentation
Quite often, these are niche programming systems, which are used in fairly closed environments. People who use them frequently sign NDAs and never speak about what they do. Programming communities for them are rare. So resources are scarce. You're stuck with your official reference, and that's it.

The ironic (and often frustrating) bit is that all the things these systems do could obviously be achieved by using mainstream and general purpose programming languages, and quite probably more efficiently. But it requires a deeper knowledge of programming, whereas you can't expect your biologist, chemist or physicist (to name a few) to know enough about programming, and even less to have the time (and desire) to implement (and maintain) complex systems, that may or may not be long-lived. For the same reason we use DSLs, we have these bespoke programming systems.

Personal Anecdote 2: Actually, I worked on one of these myself. I didn't do the link with the OP's request, but my the project was a set of inter-connected large pieces of data-processing and data-storage software (primarily for bio-informatics research, healthcare and cosmetics, but also for business intelligence, or any domain implying the tracking of large volumes of research data of any kind and the preparation of data-processing workflows and ETLs). One of these applications was, quite simply, a visual IDE that used the usual bells and whistles: drag and drop interfaces, versioned project workspaces (using text and XML files for metadata storage), lots of pluggable drivers to heterogeneous datasources, and a visual canvas to design pipelines to process data from N datasources and in the end generate M transformed outputs, and possible shiny visualizations and complex (and interactive) online reports. Your typical bespoke visual programming system, suffering from a bit of NIH syndrome under the pretense of designing a system adapted to the users' needs.
And, as you would expect, it's a nice system, quite flexible for its needs though sometimes a bit over-the-top so that you wonder "why not use command-line tools instead?", and unfortunately always leading in medium-sized teams working on large projects to a lot of different people using it with different "best" practices.

Great, We're Doomed! - What Do We Do About It?

Well, in the end, all of the above still holds. If you cannot extract most of the programming from this system to use more mainstream tools and languages, you "just" need to adapt it to the constraints of your system.

About Versioning and Storage

In the end, you can almost always version things, even with the most constrained and walled environment. Most often than not, these systems still come with their own versioning (which is unfortunately often rather basic, and just offers to revert to previous versions without much visibility, just keeping previous snapshots). It's not exactly using differential changesets like your SCM of choice might, and it's probably not suited for multiple users submitting changes simultaneously.
But still, if they do provide such a functionality, maybe your solution is to follow our beloved industry-standard guidelines above, and to transpose them to this programming system!!
If the storage system is a database, it probably exposes export functionalities, or can be backed-up at the file-system level. If it's using a custom binary format, maybe you can simply try to version it with a VCS that has good support for binary data. You won't have fine-grained control, but at least you'll have your back sort of covered against catastrophes and have a certain degree of disaster recovery compliance.

About Testing

Implement your tests within the platform itself, and use external tools and background jobs to set up regular backups. Quite probably, you fire up these tests the same that you would fire up the programs developed with this programming system.
Sure, it's a hack job and definitely not up to the standard of what is common for "normal" programming, but the idea is to adapt to the system while trying to maintain a semblance of professional software development process.

The Road is Long and Steep...

As always with niche environments and bespoke programming systems, and as we exposed above, you deal with strange formats, only a limited (or totally inexistant) set of possibly clunky tools, and a void in place of a community.
The Recommendation: Try to implement the above guidelines outside of your bespoke programming system, as much as possible. This ensures that you can rely on "common" tools, which have proper support and community drive.
The Workaround: When this is not an option, try to retrofit this global framework into your "box". The idea is to overlay this blueprint of industry standard best practices on top of your programming system, and make the best of it. The advice still applies: define structure and best practices, encourage conformance.
Unfortunately, this implies that you may need to dive in and do a tremendous amount of leg-work. So...
Famous Last Words, and Humble Requests:

Document everything you do.
Share your experience.
Open Source any tool your write.

By doing all of this, you will:

not only increase your chances of getting support from people in similar situations,
but also provide help to other people, and foster discussion around your technology stack.

Who knows, you could be at the very beginning of a new vibrant community of Obscure Language X. If there are none, start one!

Ask questions on Stack Overflow,
Maybe even write a proposal for a new StackExchange Site in the Area 51.

Maybe it's beautiful inside, but nobody has a clue so far, so help take down this ugly wall and let others have a peek!

original source : http://programmers.stackexchange.com/questions/155488/ive-inherited-200k-lines-of-spaghetti-code-what-now

codeigniter 2.x cheatsheet

noreply@blogger.com (Anonymous) — Tue, 18 Sep 2012 01:47:00 -0700

Few commands that can save your time

noreply@blogger.com (Anonymous) — Sat, 15 Sep 2012 06:59:00 -0700

Here i am listing few commands that i am using. I am sure it will be helpful.

1. Create UML from PHP code. ( annoying and little bit hacky but works well :P)

sudo phpuml ~/shiksha/trunk/LDB/controller/LDB_Server.php -f htmlnew -o ~/Desktop/ -x 1 -n ldb -m *.php6 -i .svn
sudo phpuml ~/shiksha/trunk/CodeIgniter/ -o ~/Desktop/ -x 1 -n Foo -m *.php6 -i .svn

2. VIM coomands

zz save and exit
? search backword
p command for paste
ma mark then can use "d'a" to delte entire
y for copy [can do same y'a or yy ]
:split to open other file
CTRL ww to jump file
> move to right
ZZ save and exit
ZQ exit without save
:tabs - View a list of tabs that are open with the file names. Use the command ':tabs' and Vim will display a list of all the files in the tabs. The current window is shown by a ">" and a "+" is shown for any modifiable buffers.
:tabc - Close the current tab.
:tabnew - Open a new file for editing in a separate tab.
:tab split - Open the file in the current buffer in a new tab page.
:tabn - Switching to the next tab page.
:tabp - Switch to the previous tab page.
:tabr[ewind] - Go to the first tab page. You get the same effect when you use the :tabf[irst] command.
:tabo - Close all other tab pages.

3. PHP Codebase Reports

phploc --count-tests /usr/local/src/ezcomponents/trunk/Workflow
phpmd /path/to/source text codesize,unusedcode,naming
sudo php_beautifier -l "Pear()" svrSearch.php svrSearch_n.php
php_beautifier -l "Pear(add_header=apache) ArrayNested() ListClassFunction()" index.php index_new.php

4. SYSTEM INFO

netstat -antp | awk '{print $6}' | sort | uniq -c | sort -rn
netstat -antp | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | > sort -rn
ps -e -o vsz | awk '{size += $1}END{print(size)}'
iostat -x -d
vmstat 5
top -bn1
ps -auxf | sort -nr -k 4 | head -10
ps -auxf | sort -nr -k 3 | head -10
kill $(ps aux | grep 'MailQueueRecordsMass' | awk '{print $2}')
echo stats | nc 127.0.0.1 11211
watch "echo stats | nc 127.0.0.1 11211"
watch 'php -r '"'"'$m=new Memcache;$m->connect("127.0.0.1", 11211);print_r($m->getstats());'"'"
netstat -antp
top -bn1
free -m
ps -e -o vsz | awk '{size += $1}END{print(size)}'
mpstat -P ALL 5
find /home/tmp/ -name '*.html' -print0 | xargs -0 rm -f
free -mto
top

While the output of the top command displayed, press F, which will display the following message and show all fields available for sorting, press n (which is for sorting the processes by Memory) and press enter. This will display the processes in the top output sorted by memory usage.
Current Sort Field: K for window 1:Def
Select sort field via field letter, type any other key to return

ps aux | more
ps axuf
ps U $USER

function psgrep ()
{
ps aux | grep "$1" | grep -v 'grep'
}
function psterm ()
{
[ ${#} -eq 0 ] && echo "usage: $FUNCNAME STRING" && return 0
local pid
pid=$(ps ax | grep "$1" | grep -v grep | awk '{ print $1 }')
echo -e "terminating '$1' / process(es):\n$pid"
kill -SIGTERM $pid
}

find ./ -type f -name \*.php -exec php -l {} \;
# how many mysql connections
netstat -n -a |grep 3306
ps ax | grep sshd
netstat -ln | grep 22
grep -ril john /root
find /etc -name "*mail*" [name start mail]
find / -type f -size +100M [ size more than 100 MB]
find . -mtime +60 [ more than 60 days]
find . –mtime -2 [ last 2 days]
cat file.txt > /dev/null [only error]
cat invalid-file-name.txt 2> /dev/null [only output no error]
find / -name *.jpg -type f -print | xargs tar -cvzf images.tar.gz
cat url-list.txt | xargs wget –c
/sbin/ifconfig -a
df -h
df -Tha
du -sh ~
find /var/log -type f -size +100000k -exec ls -lh {} \; | awk '{ print $9 ": " $5 }'

4. SVN REPORT

java -jar statsvn.jar searchAgents/logfile.log searchAgents/ -output-dir svnreport/

5. SQL to get database table size

SELECT TABLE_SCHEMA AS 'Database', TABLE_NAME AS 'Table', CONCAT(ROUND(((DATA_LENGTH + INDEX_LENGTH - DATA_FREE) / 1024 / 1024),2)," MB") AS Size FROM INFORMATION_SCHEMA.TABLES where TABLE_SCHEMA like '%mailer%';

6. SVN

svn delete $( svn status | sed -e '/^!/!d' -e 's/^!//' )
svn add $( svn status | sed -e '/^?/!d' -e 's/^?//' )
svn resolve $( svn st | grep "^C" | awk '{print $2}' )
svn add * --force find ./ -type f -name \*.php -exec php -l {} \;
find ./ -type f -name \*.php -exec php -l {} \; | grep "Errors parsing "; //-not -regex '.*/.svn/*.*'
find . -name .svn -print0 | xargs -0 rm -rf find -type d -name '.svn' -exec rm -rfv {} \;
svn log | sed -n '/raviraj/,/-----$/ p' svn log -v --use-merge-history URL/branches/Branch01