The tunnels are very important to making things easy while improving security. It’s not easy to understand at first, but basically it means you can now connect to ANY IP address:port as if you were on that very machine connecting to localhost, like if you pinged yourself!. The result is any traffic you want is now encrypted, and you can set up your servers to only accept connections from localhost, which could save you tons of memory, bandwidth, and security attack vectors to think about. So I configure everything to use these tunnels as proxies, like Mozilla Thunderbird and Chrome, Firefox, Pidgeon, all portable versions and running from my encrypted usb drive. This means you can walk into my house with that usb key, plug into any computer here, and surf the web/check your emails all across SSH… I know for a fact I wouldn’t be able to snoop that traffic! There is a lot of exciting things going on around here, new servers and all.. Its going to take a couple more posts for me to finish this up, enjoy the article and comment.

Buy a couple USB Mini Drives

The first thing to do, is purchase a USB thumb drive.. My favorite store, TigerDirect.com, has over 104 tiny usb drives for under $24.. I’ve used them since the late 90’s.

I bought some 4GB PNY’s the size of a fingernail at a gas station and they are amazing, way faster than say a dvd drive. Just try to do some research of the differences between the 16GB vs the $4 1GB drives.. You want speed because the whole drive will be encrypted. If you can afford the super excellent and crazy fast ones, hey send me one! Buying cheap means you can buy 3 or 4 so you can always have backups. This device will make you Internationally mobile, untethered from a box, maybe getting some work done at a cafe in Florenze, or at a beach hotel in Miami. Keep dreaming, but that is more possible with a better organized system.

Backup the USB Drive

You only need to know 1 way that works, there are several. The way I do backups is to copy the entire disk image of the usb, that way I can always access it in case of usb key failure, which does happen. Free software like CloneZilla live CD with its crazy cluster computing power, or Self Image, which is free for both linux and Windows. And you could never go wrong with GHOST, one of the first to make mega bucks in the market.. it’s some seriously impressive software but not open-source. Even easier for some is to just set a cron job for dd to pipe the entire drive image to a remote computer using netcat, or sshfs, or curlftpfs, or just simple ssh like below. Once setup (without stupid, bulky, dangerous software), the files on your encrypted usb don’t change often, otherwise I would want to sync a backup to happen automatically every X number of logins or days (test logfile time in bash_profile?)..

SSH Back-ups To Remote Server

Files and data on your drives slow it down tremendously, meaning a web server storing backups locally is slower than one storing them externally.

Notice how much safer this command is by optimizing both the CPU and DISK I/O.. Though it’s much smarter to create a new separate ssh user, one with no shell and a passwordless safer key-based encryption. Then in your /etc/security/limits.conf file or your initscript you can cause that user to have nice -19 and ionice -c2 -n7 priority set all the time automatically, since sshd, compression, and disk writing are this accounts only job. turboslow is an alias defined in a ssh_config file so you don’t have to type the host, port, and settings each time.

#
# much better ways to do this on google!!!!!!!
#
ionice -c2 -n7 nice -n 19 dd if=/dev/sdb2 bs=1k conv=sync,noerror | gzip -c | ssh turboslow "dd of=sdb2.gz bs=1k"

Note that you may decide it would be better to configure the ssh connection to a less CPU intensive algorithm, perhaps even protocol 1 and DES. That’s perfectly alright, but the tradeoff is that the encryption can be broken much quicker, and so you would have to implement a cron job to create new keys on both ends of the tunnel every few hours.. It’s really not a big deal to setup, kind of sweet way to use key-based encryption. Also, important files ( those containing passwords, any database ) are encrypted before transport using private GPG keys, which don’t need to be changed. The other thing to think about too is only letting your main PC send/write on the backup host, so the backup host is only authorized to rx and can never login back to yours.

Hey! the Internet is a dangerous place you better believe it! And it’s only going to get more interesting with cloud computing’s breakthrough’s… More people who know they’re way around… I can always use an extra server, I’d love to expand my network another node without having to pay for it (free cloud computing?), so make sure your servers are locked up strenuously. Not super perfect, just a little unique or creative in your defense to avoid any coming super-worm’s that may be employing vast arsenals of the deadliest attack-engines like metasploit.. Scarry rumors.

Compression Speeds: PBZip2, Rzip, Lzop, Gzip

Probably the fastest is to use rsync over ssh, which is what I’m doing, since the algorithms used by rsync are much faster and safer. Rsync also lets you specify a compression program, so depending on your machine you will want pbzip2 (for multi processors) or rzip which are the 2 fastest I know of, though I have had some reliability issues with rzip for gigabyte transfers. Pbzip2 is amazing, blew me away the first time being 8x faster (8 CPUs) then anything. You can get it and compile a static binary for your thumb drive if want at Parallel BZIP2 (PBZIP2). Heavy code, re: this note by Jeff Gilchrist

NOTE: If you are looking for a parallel BZIP2 that works on cluster machines, you should check out MPIBZIP2 which was designed for a distributed-memory message-passing architecture.

tar cpf "$G" --use-compress-prog=pbzip2 ./

Benchmarking for Performance

Finally a couple tips, you should get an idea what the device can do, format it a few times for linux and test it on windows, and vice versa.. Some drives are too small or too old and can only support fat32 filesystems on winblows, you DO NOT want fat32 because this drive is going to be 100% encrypted and then 100% transparently decrypted as you use it,

# note this is 512MB
dd if=/dev/sda1 of=/dev/null bs=512 count=1000000
512000000 bytes (512 MB) copied, 5.16588 s, 99.1 MB/s

Part II: Encrypted AutoRunning USB Key with TrueCrypt

Now this section anyone can do, it’s so easy on Windows. What I’m going to show you how to do is get setup the right way super-fast. There are many ways to use TrueCrypt, it’s one of the nicest built software programs’s I’ve ever used… Sadly, it is not licensed open-source, and that is often a deal-breaker for security-conscious folks or anti-pirate anarchists. From the very helpful TrueCrypt web site:

• Creates a virtual encrypted disk within a file and mounts it as a real disk.
• Encrypts an entire partition or storage device such as USB flash drive or hard drive.
• Encrypts a partition or drive where Windows is installed (pre-boot authentication).
• Encryption is automatic, real-time (on-the-fly) and transparent.
• Parallelization and pipelining allow data to be read and written as fast as if the drive was not encrypted.
• Provides plausible deniability, in case an adversary forces you to reveal the password: Hidden volume (steganography) and hidden operating system.
• Encryption algorithms: AES-256, Serpent, and Twofish. Mode of operation: XTS.

Further Reading

• Network File Copy using SSH
• Check out the trunk version of PuTTY:~ svn co svn://svn.tartarus.org/sgt/putty

The real fun doesn’t start till all the automation starts, automating all of that from a couple batch files I wrote, one click setup. Kind of like building your own knoppix for when you have to use Windows. To begin this tutorial, setup a truecrypt traveller setup on your usb and also install the portaputty package onto the usb. You do this by creating a 3GB or whatever file on the usb and then mounting that file like you would mount an iso file. I will show the Windows Batch file I use and the tricks with Windows Volume names and how to consistently make it all work. Then we will setup MyEnTunnel with a customized batch file that forces all puttys to use portaputty (sweet hack stolen from sysinternals pagedefrag tool).Stay Tuned!

PortaPutty Auto-Reconnecting SSH Tunnels on an Encrypted TrueCrypt Portable USB Key w GPG originally appeared on .

Nice example of chaotic beats, but they make melody somehow, it’s rather an addictive beat.

What random chaos have I spent the last 6 months on? Experiments with DNS, Fail-over technology for MySQL/DNS/WWW/MAIL servers for free or for very cheap, chrooting servers and of course testing new security idea…. But the exciting stuff for me personally is some very cool ways to make sure you are never unable to remotely access your server in case of catastrophic failure.. That’s what it’s all about for me you know, making things work so well that they run themselves and free up time for life! I’m around a 1/4 of a century old.. I’m done playing around, it’s time to make some cash before they put me in an old-folks home.. My time is short, my cause is just.. We will prevail! lol.. I’m 25 and I still got jokes!

For 2010 I have a few choice articles and ideas I’m debating about publishing.. I don’t want to give the wrong kind of people power, I want to give the power to the people — but at a certain point it takes a serious investment to actually empower yourself, whereas a company could afford to actually bring it to reality.. So who does the knowledge benefit then? No, the only way is to get serious myself and beat the profiteers to the punch so to offer a service that would be dirt-cheap if it weren’t in the hands of gluttons.

I’ve been pursuing research into several new areas of web technology (no not a smartphone app!) over the past year trying to increase my business profits and quailty of service, and I am becoming more aware every day that I can do it better and cheaper! Any industry always has those at the top who enjoy knowing things that keep them on top, and try to prevent others from doing what they did.. It’s not like the information is so wizard-like that it’s just not possible for average folks to comprehend.. It’s just only discussed behind closed doors, masked in schools by tenure and a defined curriculum. But since this is the Wild Wild Web the barriers can always be circumvented. Stick around, I am already looking for partners (both investors and linux/code kungfu masters).

Enjoy the jam session ;)

grilled cheese jam session now online originally appeared on .

Top 3 WordPress Speed Plugins

1. DB Cache Reloaded
2. WP Super Cache
3. AskApache Crazy Cache

WP Caching Overview

Each request to your blog has to fire up the php interpreter and query the mysql database to create the output that you see in your browser. Using a plugin like WP Super Cache simply saves the output of that request as a static HTML file and serves that to a request instead of using php and mysql every time. DB Cache Reloaded takes this a step further by optimizing the mysql queries. Finally, AskApache Crazy Cache is used to keep a cache fully primed and ready.

Why Caching?

If you have a private server, or you want to keep your MEMORY, BANDWIDTH, and CPU usage down for your server, these plugins will be dramatic. If you have a site that is updated maybe once a month and gets a very small amount of traffic, then the AskApache Crazy Cache would be redundant. That plugin is geared for the heaviest traffic sites.

Request and Response

Most HTTP communication is initiated by a user agent (browser) and consists of a request to a resource on some origin server. In the simplest case, this may be accomplished via a single connection (v) between the user agent (UA) and the origin server (O).

             request chain ------------------------>
          UA -------------------v------------------- O
             <----------------------- response chain

If a browser requests a WP-driven page, the server generates the response (the outputted html) by loading a php interpreter or module to read the WP php files and load all the settings. WP stores the settings in a mysql database and has to query the mysql database for all the data (like the content of your post that becomes html). Finally php sends the output through the server back to the browser. This is the norm for most PHP applications. Every time an interpreter is loaded additional CPU and Memory are used. And perhaps even more troublesome for shared-hosting using a virtual or network filesystem, each load causes many hard-disk accesses, additional processes, etc.

Cache PHP

By saving the php-generated output of a page to a static html file, your server can entirely skip loading a php interpreter or other process. Originally servers were needed and created to essentially open a static file on disk and send that file back to the requesting user. So servers are specialized for this as it’s their core function.

WP-Super cache does this for you by saving the output into a static html file, and by instructing the server to skip loading php. It basically only uses php for creating the cached version, in the same way that you can save this webpage as an html file, WP-Super cache saves all the WP-blog permalinks to static files.

DB-Cache Reloaded does something entirely different, it saves the mysql queries that are made to the WP-database, as well as the mysql results to static files, and then through php serves those cached-files instead of re-querying the mysql database. Most mysql databases are stored on separate servers, and although many are on the same local network there is a limit to how many queries, and how many connections can take place. But mysql is maybe the fastest thing I’ve seen, so your bottlenecks almost never happen there (if configured correctly).

So DB-Cache Reloaded basically makes WP-Super Cache work alot faster when generating the cache files, and DB-Cache Reloaded helps in a number of areas un-related to WP-Super Cache, like in the admin panel. And DB-Cache without WP-Super-Cache is a joke because it still uses the application-level and php for everything. Gotta use both (or just WPSC).

AskApache Crazy Cache is a plugin I wrote to do one thing very well, it runs at intervals via the WP-cron and forces WP-Super-Cache to create a static cache file for all the posts, pages, etc. on your site. Without this WPSC likes to do dumb things like try to manage it’s own cache with stale files and expired files, which equals a lot more php interpreters getting loaded instead of cached static files. For sites with more than a visit/page/10minutes this plugin keeps a full primed cache built by WPSC.

Compression and WP-Super Cache

Enabling compression in WP-Super Cache is almost always a great idea. I’ve never had a problem other than some php compat issues with not-updated php installations. This option basically lets WP-Super Cache compress the generated output of a page and save that to the static file. Normally Apache, Lighttpd, Nginx, etc. open the static file and compress it before it is sent to the browser, then the browser automatically decompresses it to view. This happens so fast because it is run by the server.

This lets WPSC instruct your server to send the compressed version to all browsers that accept compression, and send the uncompressed static file to any other browsers. So this is helpful because it eliminates your server having to do any transparent compressing, it can instead just focus on what it does best, serving static files.

PHP is an application so it requires memory, hard-drive access, and CPU time. Check out the protocol hierarchy:

       +------+ +-----+ +-----+       +-----+
       |Telnet| | FTP | |Voice|  ...  |     |  Application Level
       +------+ +-----+ +-----+       +-----+
             |   |         |             |
            +-----+     +-----+       +-----+
            | TCP |     | RTP |  ...  |     |  Host Level
            +-----+     +-----+       +-----+
               |           |             |
            +-------------------------------+
            |    Internet Protocol & ICMP   |  Gateway Level
            +-------------------------------+
                           |
              +---------------------------+
              |   Local Network Protocol  |    Network Level
              +---------------------------+

DB Cache Reloaded

DB Cache Reloaded Plugin Page – News

This plugin caches every database query with given lifetime. It is much faster than other html caching plugins and uses less disk space for caching.

I think you’ve heard of WP-Cache or WP Super Cache, they are both top plugins for WordPress, which make your site faster and responsive. Forget about them – with DB Cache Reloaded your site will work much faster and will use less disk space for cached files. Your visitors will always get actual information in sidebars and server CPU loads will be as low as possible.

Why is DB Cache Reloaded better than WP Super Cache?

This plugin is based on a fundamentally different principle of caching queries to database instead of full pages, which optimises WordPress from the very beginning and uses less disk space for cache files because it saves only useful information. It saves information separately and also caches hidden requests to database.

WP Super Cache

WP Super Cache Plugin Page – News

This plugin generates static html files from your dynamic WordPress blog. After a html file is generated your webserver will serve that file instead of processing the comparatively heavier and more expensive WordPress PHP scripts.

The static html files will be served to the vast majority of your users, but because a user’s details are displayed in the comment form after they leave a comment those requests are handled by PHP. Static files are served to:

· Users who are not logged in.
· Users who have not left a comment on your blog.
· Or users who have not viewed a password protected post.

99% of your visitors will be served static html files. Those users who don’t see the static files will still benefit because they will see regular WP-Cache cached files and your server won’t be as busy as before. This plugin will help your server cope with a front page appearance on digg.com or other social networking site.

Why is WP-Super-Cache better than WP-Cache?

This plugin is based on the excellent WP-Cache plugin and therefore brings all the benefits of that plugin to WordPress. On top of that it creates copies of every page that is accessed on a blog in a form that is quickly served by the web server. It’s almost as quick as if the you had saved a page in your browser and uploaded it to replace your homepage.

Will the Super Cache compression slow down my server?

No, it will do the opposite in fact. Super Cache files are compressed and stored that way so the heavy compression is done only once. These files are generally much smaller and are sent to a visitor’s browser much more quickly than uncompressed html. As a result, your server spends less time talking over the network which saves CPU time and bandwidth, and can also serve the next request much more quickly.

AskApache Crazy Cache

AskApache Crazy Cache Plugin Page – News

This sweet little plugin does one thing very well. It caches all the posts on your entire blog at the same time, if you are using WP-Cache, WP-Super-Cache, or Hyper-Cache.

                              +---------+ ---------\      active OPEN
                              |  CLOSED |            \    -----------
                              +---------+<---------\   \   create TCB
                                |     ^              \   \  snd SYN
                   passive OPEN |     |   CLOSE        \   \
                   ------------ |     | ----------       \   \
                    create TCB  |     | delete TCB         \   \
                                V     |                      \   \
                              +---------+            CLOSE    |    \
                              |  LISTEN |          ---------- |     |
                              +---------+          delete TCB |     |
                   rcv SYN      |     |     SEND              |     |
                  -----------   |     |    -------            |     V
 +---------+      snd SYN,ACK  /       \   snd SYN          +---------+
 |         |<-----------------           ------------------>|         |
 |   SYN   |                    rcv SYN                     |   SYN   |
 |   RCVD  |<-----------------------------------------------|   SENT  |
 |         |                    snd ACK                     |         |
 |         |------------------           -------------------|         |
 +---------+   rcv ACK of SYN  \       /  rcv SYN,ACK       +---------+
   |           --------------   |     |   -----------
   |                  x         |     |     snd ACK
   |                            V     V
   |  CLOSE                   +---------+
   | -------                  |  ESTAB  |
   | snd FIN                  +---------+
   |                   CLOSE    |     |    rcv FIN
   V                  -------   |     |    -------
 +---------+          snd FIN  /       \   snd ACK          +---------+
 |  FIN    |<-----------------           ------------------>|  CLOSE  |
 | WAIT-1  |------------------                              |   WAIT  |
 +---------+          rcv FIN  \                            +---------+
   | rcv ACK of FIN   -------   |                            CLOSE  |
   | --------------   snd ACK   |                           ------- |
   V        x                   V                           snd FIN V
 +---------+                  +---------+                   +---------+
 |FINWAIT-2|                  | CLOSING |                   | LAST-ACK|
 +---------+                  +---------+                   +---------+
   |                rcv ACK of FIN |                 rcv ACK of FIN |
   |  rcv FIN       -------------- |    Timeout=2MSL -------------- |
   |  -------              x       V    ------------        x       V
    \ snd ACK                 +---------+delete TCB         +---------+
     ------------------------>|TIME WAIT|------------------>| CLOSED  |
                              +---------+                   +---------+
 
                      TCP Connection State Diagram

Vetted – Top 3 WordPress Speed Plugins originally appeared on .

Want to get started? Type the following in your shell to download the script and run it. You should go over the code first and then you can just save it as your .bash_profile – whenever you want to get updated to the latest version just type aaup, a cool update function that auto dos2unix’s and runs via an exec command.

curl -O http://static.askapache.com/askapache-bash-profile.txt && source askapache-bash-profile.txt
Or if you trust me and my hosting provider (and you aren't on production)
curl -o ~/.bash_profile http://static.askapache.com/askapache-bash-profile.txt && exec bash -l

For those of you power users and server admins that use Bash, ksh, csh, vanilla sh, etc.., or if you are just passionate about shell scripting, because it allows you to get advanced tasks done fast and efficiently, not to mention automated automatically. I give you my .bash_profile file. You should edit it to fit your needs, (especially the exported vars like PATH, LDFLAGS, if you don’t understand something just comment it out) but it’s pretty universal because I work on alot of other people’s servers not to mention many different distros and platforms, and when I get hired to do some server work through a shell, I bring this script along for the ride.

askapache-bash-profile.txt

You can download the latest version: http://static.askapache.com/askapache-bash-profile.txt

The functions and variables below are the way bash sees them, using declare -f, and alias, to make it easier for you to read and understand them. The actual file at http://static.askapache.com/askapache-bash-profile.txt will always be the most updated version, as I use it personally. And it has the whole file the way I wrote it, meaning many extra notes and much simpler to follow. Enjoy!

Gist of the Script..

I also rely on this heavily from within shell scripts I write to access all the functions and stuff in this .bash_profile, and to do that I just do like:

#!/bin/bash
 
source ~/.bash_profile &>/dev/nulll
 
pm "PM is a function to output nice messages with color"
yn "Are you enjoying the shell" && pm "Thats great!" || pm "Perhaps you're better suited for DOS"
yn "Show Calendar" && aa_calendar
yn "Show Fortune" && aa_fortune

alias chmod='command chmod -c'
alias cpr='command cp -rpv'
alias df='command df -kTh'
alias df1='command df -iTa'
alias diff='diff -up'
alias dsiz='du -sk * | sort -n --'
alias du='command du -kh'
alias du1='echo *|tr " " "\n" |xargs -iFF command du -hs FF|sort'
alias env='command env | sort'
alias h='history'
alias inice='ionice -c3 -n7 nice'
alias j='jobs -l'
alias la='command ls -Al --color=auto'
alias lc='command ls -lAcr --color=auto'
alias less='vless'
alias lessc='ccze -A |`type -P less` -R'
alias lk='command ls -lASr --color=auto'
alias llh='ll -h'
alias lll='stat -c %a\ %N\ %G\ %U ${PWD}/*|sort'
alias lr='command ls -lAR --color=auto'
alias lt='command ls -lAtr --color=auto'
alias lu='command ls -lAur --color=auto'
alias lx='command ls -lAXB --color=auto'
alias mann='command man -H'
alias n='/usr/bin/nano3'
alias p='command ps -HAcl -F S -A f|uniq -w3'
alias path='echo -e ${PATH//:/\\n}'
alias php='php -d report_memleaks=1 -d report_zend_debug=1 -d log_errors=0 -d ignore_repeated_errors=0 -d ignore_repeated_source=0 -d error_reporting=30719 -d display_startup_errors=1 -d display_errors=1'
alias pp='command ps -HAcl -F S -A f'
alias ps1='command ps -lFA'
alias ps2='command ps -H'
alias resetw='echo $'\''\33[H\33[2J'\'''
alias subash='sudo sh -c '\''export HOME=/root; cd /root; exec bash -l'\'''
alias top='top -c'
alias tree='command tree -Csuflapi'
alias updatedb='( ( updatedb 2>/dev/null ) & )'
alias vim='command vim --noplugin'
alias who='command who -ar -pld'
alias wtf='watch -n 1 w -hs'

aa_aliases ()
{
    local F=`for i in $(alias|awk -F= '{print $1}');do echo $i;done|sed -e :a -e 's/^.\{1,15\}$/& /;ta'|tr ' ' '\032'`;
    i=1;
    for f in ${F};
    do
        echo -en "$f";
        (( $i % 4 ==0 )) && echo -en "\n";
        i=$(( $i + 1));
    done | tr '\032' " "
}
aa_beep ()
{
    local i;
    for i in `seq 1 ${1:-5}`;
    do
        echo -en "\a" && sleep 1;
    done
}
aa_calendar ()
{
    [[ -d /usr/share/calendar/ ]] && echo -en "\n${CC[4]}" && ( sed = $(echo /usr/share/calendar/calendar*) | sed -n "/$(date +%m\\/%d\\\|%b\*\ %d)/p" ) && echo -en "${R}"
}
aa_current_users ()
{
    echo -n "`command who -q |sed '/^# users=\([0-9]*\).*/!d; s//\1/;q'`"
}
aa_find_exec ()
{
    find -L ${1:-.} -type f -perm +0100 2>$N6 | xargs file | sed -e 's%^\([^:]*\):\([^,]*\).*$%\1:\2%g'
}
aa_fortune ()
{
    local FORTUNE=`command type -P fortune`;
    [[ -x $FORTUNE ]] && echo -en "\n${CC[5]}" && $FORTUNE -s && echo -en "${R}";
    [[ -x $FORTUNE ]] && echo -en "\n${CC[6]}" && $FORTUNE -s && echo -en "${R}";
    [[ -x $FORTUNE ]] && echo -en "\n${CC[7]}" && $FORTUNE -s && echo -en "${R}";
    [[ -x $FORTUNE ]] && echo -en "\n${CC[8]}" && $FORTUNE -s && echo -e "${R}"
}
aa_free_mem ()
{
    echo -n "`[[ -r /proc/meminfo ]] && sed '/^MemFree: */!d; s///;q' /proc/meminfo`"
}
aa_functions ()
{
    local F=`for i in $(declare -F | sed -e 's/declare -f //g'); do echo $i; done| sed -e :a -e 's/^.\{1,25\}$/& /;ta'|tr ' ' '\032'`;
    i=1;
    for f in ${F};
    do
        echo -en "$f";
        (( $i % 4 ==0 )) && echo -en "\n";
        i=$(( $i + 1));
    done | tr '\032' " "
}
aa_getsrc ()
{
    [[ "$#" -lt "1" ]] && echo "Usage: $FUNCNAME <url>" 1>&2 && return 2;
    [[ -z "$CPPFLAGS" ]] && pm "Setting CPPFLAGS" && export CPPFLAGS="-I/usr/local/include/libxml2 -I/opt/libiconv/include";
    [[ -z "$LD_FLAGS" ]] && pm "Setting LD_FLAGS" && export LD_FLAGS="-L/usr/local/lib -L/usr/local/lib/python2.4/site-packages -L/usr/local/lib -lxml2 -lz -L/opt/libiconv/lib -liconv -lm";
    function aa_err ()
    {
        echo '';
        local s;
        echo -en "\n [ ${1:-Press any key to continue} ]\n";
        read -n 1 s
    };
    function aa_install ()
    {
        ./configure --help;
        ./configure --prefix=/usr/local "$*" && make -j3 && sudo make install || aa_err;
        echo -en "\a"
    };
    function aa_extract ()
    {
        case $1 in
            *.tar.bz2)
                tar xjf $1
            ;;
            *.tar.gz)
                tar xzf $1
            ;;
            *.tar)
                tar xf $1
            ;;
            *.tbz2)
                tar xjf $1
            ;;
            *.tgz)
                tar xzf $1
            ;;
            *.bz2)
                bunzip2 $1
            ;;
            *.rar)
                unrar x $1
            ;;
            *.gz)
                gunzip $1
            ;;
            *.zip)
                unzip $1
            ;;
            *.Z)
                uncompress $1
            ;;
            *.7z)
                7z x $1
            ;;
            *)
                aa_err
            ;;
        esac
    };
    local OD=`pwd -L`;
    local FN=$(echo $1 | sed -e 's/^.*\/\([^\/]\{3,\}\)$/\1/g');
    local SN=$(echo $FN |  sed -e 's/\(.\{1\}\)\([^-.]\{1,\}\).*/\1\2/g');
    echo -e "OD: $OD \n1: $1 \nFN: $FN \nSN: $SN" && sleep 1;
    cd /opt/dist;
    ( curl -# -L -f -b c.txt --cookie-jar c.txt -m 400 -O -H 'Accept: */*' -s -S -A 'Mozilla/5.0 (Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/2009 Firefox/3.5.5' -e "$1" --url "$1" || v_err "CURL FAILED for $1" );
    [[ -f "$FN" ]] && aa_extract "$FN" || aa_err "EXTRACTING FAILED for $FN, go manual it";
    [[ -d "/opt/$SN" ]] && mv -v /opt/$SN /opt/.old-$SN;
    mv -v /opt/source/`echo $1|sed -e 's/.*\/\(.*\)\.\([tgz7bar2v]*\)\(\.[tgz7bar2v]*\)$/\1/g'` /opt/$SN || aa_err "Please mv $1 to /opt/$SN";
    cd /opt/$SN && command ls -lABls1c $AAC && aa_install;
    cd $OD
}

aa_hexdump ()
{
    local EX;
    [[ $# -eq 1 ]] && EX=exec;
    [[ -f "$1" ]] && EX='cat "$1"';
    case ${2:-1} in
        0)
            $EX hexdump -C
        ;;
        1)
            $EX hexdump -Cc
        ;;
        2)
            $EX hexdump -Cx
        ;;
        3)
            $EX hexdump -C -e '""  10/1 "'\''%_c'\''\t" "\n"' -e '"" 10/1 "0x%02x\t" "\n\n"'
        ;;
        4)
            $EX hexdump -e '""  10/1 "'\''%_c'\''\t" "\n"'
        ;;
        5)
            $EX hexdump -e '""  10/1 "'\''%_c'\''\t" "\n"' -e '"" 10/1 "0x%04x\t" "\n\n"'
        ;;
        6)
            $EX hexdump -e '90/1 "%_p" "\n"'
        ;;
        7)
            $EX hexdump -e '1/1 "%04_u" "%4_p" "   " 1/1 "%04x" "  " 1/1 "%04_c"  "\n"'
        ;;
        8)
            $EX hexdump -v -e '"x" 1/1 "%02X" " "'
        ;;
        9)
            $EX od -Ax -tx1 -v
        ;;
        10)
            $EX od -t o1z -w1 -Ao -v
        ;;
        11)
            $EX xxd -
        ;;
        12)
            $EX xxd -c1
        ;;
        13)
            $EX xxd -h
        ;;
        14)
            $EX
        ;;
    esac
}
aa_ips ()
{
    ahave ip && ip -o -f inet addr 2>&1 | sed -e 's/.*inet \([^/]*\).*/\1/g' | sort -u;
    ahave ss && ss -n 2>&1 | sed -e 's/[^:]*\ \([0-9]*\.[0-9]*\.[0-9]*\.[0-9]*\)\:.*/\1/' -e '/^\([^0-9]\|127\.\|10\.\|172\.\)/d' | sort -u;
    [[ -f "$HOME/.cpanel/datastore/_sbin_ifconfig_-a" ]] && sed -e '/inet/!d' -e 's/.*addr:\([0-9\.]*\).*/\1/g' "$HOME/.cpanel/datastore/_sbin_ifconfig_-a" | sort -u;
    sed -e 's/^\([0-9]*\.[0-9]*\.[0-9]*\.[0-9]*\).*/\1/' -e '/^\([^0-9]\|127\.\|10\.\|172\.\|$\)/d' /etc/hosts | sort -u
}
aa_isalpha ()
{
    [[ "$#" -lt "1" ]] && echo "Usage: $FUNCNAME str" 1>&2 && return 2;
    case $1 in
        *[!a-zA-Z]* | "")
            return -1
        ;;
        *)
            return 0
        ;;
    esac
}
aa_isdigit ()
{
    case $1 in
        *[!0-9]* | "")
            return -1
        ;;
        *)
            return 0
        ;;
    esac
}
aa_mkdir ()
{
    [[ "$#" -lt "1" ]] && echo "Usage: $FUNCNAME " 1>&2 && return 2;
    local e=0;
    for f in ${1+"$@"};
    do
        set fnord `echo ":$f"|sed -e 's/^:\//%/' -e 's/^://' -e 's/\// /g' -e 's/^%/\//'`;
        shift;
        p=;
        for d in ${1+"$@"};
        do
            p="$p$d";
            case "$p" in
                -*)
                    p=./$p
                ;;
                ?:)
                    p="$p/";
                    continue
                ;;
            esac;
            [[ ! -d "$p" ]] && mkdir -v "$p" || e=$?;
            p="$p/";
        done;
    done;
    return $e
}
aa_motd ()
{
    [[ -r /etc/motd && `cat /etc/motd |wc -l` -gt 7 ]] && ( echo -e "\n\n${CC[2]}`head -n 7 /etc/motd | tail -n 6`${R}\n" ) && return;
    ahave figlet && figlet $HOSTNAME
}
aa_num_procs ()
{
    echo -n `command ps aux|wc -l`
}
aa_prompt ()
{
    AAPS[0]='\n\e[1;30m[\e[0;37m${SHLVL}\e[1;30m:\e[0;37m\j:\!\e[1;30m][\e[1;34m\u\e[0;34m@\e[1;34m\h\e[1;30m:\e[1;37m${SSHTTY/\/dev\/}\e[1;30m]\e[0;37m[\e[0;37m\w\e[0;37m]\e[1;30m\n\[${R}\]\$ ';
    AAPS[1]='\n\e[1;30m[\e[0;37m${SHLVL}\e[1;30m:\e[0;37m\j:\!\e[1;30m][\e[0;32m\u\e[1;32m@\e[0;32m\h\e[1;30m:\e[1;37m${SSHTTY/\/dev\/}\e[1;30m]\e[0;37m[\e[0;37m\w\e[0;37m]\e[1;30m\n\[${R}\]\$ ';
    AAPS[2]='\n\e[1;30m[\e[0;37m${SHLVL}\e[1;30m:\e[0;37m\j:\!\e[1;30m][\e[0;35m\u\e[1;35m@\e[0;35m\h\e[1;30m:\e[1;37m${SSHTTY/\/dev\/}\e[1;30m]\e[0;37m[\e[0;37m\w\e[0;37m]\e[1;30m\n\[${R}\]\$ ';
    : ${PLVL=0};
    [[ "${#AAPS[@]}" -lt "$PLVL" || "${#AAPS[@]}" -eq "$PLVL" ]] && PLVL=0;
    export PS1=${AAPS[$PLVL]} && (( PLVL++ )) && export PLVL;
    echo "PLVL=$PLVL"
}
aa_random_under ()
{
    echo $(( $RANDOM % ${1:-$RANDOM} + 1 ))
}
aa_safe_aliases ()
{
    local MYMVR=$(command type -t mymvmymv);
    [[ "$MYMVR" == "alias" ]] && echo "Turning Safe Aliases Off" && for fa in "chmod" "mkdir" "rm" "cp" "mv" "mymv";
    do
        unalias $fa;
    done && return;
    echo "Turning Safe Aliases On";
    alias chmod='command chmod -c';
    alias mkdir='command mkdir -pv';
    alias rm='command rm -v';
    alias cp='command cp -v';
    alias mv='command mv -v';
    alias mymv='echo'
}

aa_savehist ()
{
    history ${HISTCMD:-5000} | sed -e 's/^[ 0-9]*\(.*\)/\1/g' | tee -a $HISTFILEMASTER_C >>$HISTFILEMASTER;
    cat $HISTFILE | tee -a $HISTFILEMASTER_C >>$HISTFILEMASTER;
    history -w;
    echo
}
aa_show_all_colors ()
{
    for c in `seq 0 255`;
    do
        t=5;
        [[ $c -lt 108 ]] && t=0;
        for i in `seq $t 5`;
        do
            echo -e "\e[0;48;$i;${c}m|| $i:$c `seq -s+0 $(($COLUMNS/2))|tr -d '[0-9]'`\e[0m";
        done;
    done;
    echo;
    echo Table for 16-color terminal escape sequences.;
    echo Replace ESC with \\033 in bash.;
    echo;
    echo "Background | Foreground colors";
    echo "---------------------------------------------------------------------";
    for ((bg=40; bg<=47; bg++))
    do
        for ((bold=0; bold<=1; bold++))
        do
            echo -en "\033[0m"" ESC[${bg}m   | ";
            for ((fg=30; fg<=37; fg++))
            do
                if [ $bold == "0" ]; then
                    echo -en "\033[${bg}m\033[${fg}m [${fg}m  ";
                else
                    echo -en "\033[${bg}m\033[1;${fg}m [1;${fg}m";
                fi;
            done;
            echo -e "\033[0m";
        done;
        echo "--------------------------------------------------------------------- ";
    done;
    echo;
    echo
}
aa_sleep ()
{
    echo -en "${CC[6]}${3:-.}";
    while `command ps -p $1 &>/dev/null`; do
        echo -n "${3:-.}";
        sleep ${3:-1};
    done;
    echo -e "$R$X" && sleep 1 && pd
}
aa_try_for_path ()
{
    local GP=$HOME/bin:$HOME/sbin;
    [[ -z "$PATH" || "$PATH" == "/bin:/usr/bin" ]] && export PATH="/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games";
    local P=$PATH:$HOME/libexec;
    [[ "$EUID" -eq 0 ]] && P="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11";
    P=${P}:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games;
    P=${P}:/bin:/etc:/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/bin/mh:/usr/libexec;
    P=${P}:/usr/X11R6/bin:/usr/libexec:/etc/X11:/etc/X11/xinit;
    [[ "$DREAMHOST" == "yes" ]] && ( P=${P}:/usr/local/dh/apache/template/bin:/usr/local/dh/apache2/template/bin;
    P=${P}:/usr/local/dh/apache2/template/build:/usr/local/dh/apache2/template/sbin;
    P=${P}:/usr/local/dh/bin:/usr/local/dh/java/bin:/usr/local/dh/java/jre/bin;
    P=${P}:/usr/lib/ruby/gems/1.8/bin:$HOME/.gem/ruby/1.8/bin );
    P=${P}:/usr/local/php5/bin;
    for p in ${P//:/ };
    do
        [[ -d "${p}" && -x "${p}" ]] && GP=${GP}:$p;
    done;
    export PATH=$( echo -en "${GP//:/\\n}" | sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; h; P' | tr "\n" : ).
}
aa_uniqhistory ()
{
    ( echo $HISTFILE;
    find $HISTFILEMASTER_DIR -type f 2>$N6 ) | xargs -iFFF cat FFF 2>$N6 | sed -e 's/^[ \t]*//;s/[ \t]*$//' -e '/[^ \t]\{1,\}/!d' | tr --squeeze ' ' | sort -u
}
aa_weather ()
{
    ahave lynx || return;
    local city weather=$HOME/.weather;
    echo -e "${CC[3]}";
    local res=`pkill -9 -f 15m &>$N6 || echo -n`;
    [[ -r $weather ]] && cat $weather && ( ( sleep 15m && command rm $HOME/.weather >&$N6 ) & );
    [[ ! -r $weather ]] && ( ( ( city=`lynx -dump http://api.hostip.info/get_html.php|sed -e '/^City/!d' -e 's/City: \([^,]*\), \(.*\)/\1+\2/g'` && lynx -dump "http://www.google.com/search?hl=en&q=weather+${city}" | sed -n '/Weather for/,/Search Res/p' | tr -d '\260' | sed -e '/iGoogle/d' -e 's/^[ \t]*//;s/[ \t]*$//' | sed '$d' | head -n 6 >$weather ) &  >&/dev/null ) );
    echo -e $R
}
aaup ()
{
    curl3 -0 -o ~/.bash_profile http://static.askapache.com/askapache-bash-profile.txt && dos2unix -dv ~/.bash_profile && ( rm -rf ~/{.bash_logout,.toprc,.lynxrc,.ncftp} || echo;
    echo ) && aa_savehist && exec bash -l
}
ahave ()
{
    unset -v ahave;
    command command type $1 >&$N6 && ahave="yes" || return 1
}
ahelp ()
{
    unset -v ahelp;
    [[ "$#" -gt "0" ]] && [[ "$1" == "-h" || "$1" == "--h" || "$1" == "--help" || "$1" == "-help" || "$1" == "-?" ]] && ahelp="yes"
}
arepeat ()
{
    [[ "$#" -lt "1" ]] && echo "Usage: $FUNCNAME " 1>&2 && return 2;
    local i max=$1;
    shift;
    for ((i=1; i <= max ; i++))
    do
        eval "$@";
    done
}
ascript_title ()
{
    ahave tput && tput cup 0 3;
    clear;
    lin 0;
    echo -e "| ${CC[${1:-2}]}                ___       __    ___                 __                  ${CC[0]} |";
    echo -e "| ${CC[${1:-2}]}               / _ | ___ / /__ / _ | ___  ___ _____/ /  ___             ${CC[0]} |";
    echo -e "| ${CC[${2:-9}]}              / __ |(_-</  '_// __ |/ _ \/ _ \`/ __/ _ \/ -_)            ${CC[0]} |";
    echo -e "| ${CC[${2:-9}]}             /_/ |_/___/_/\_\/_/ |_/ .__/\_,_/\__/_//_/\__/             ${CC[0]} |";
    echo -e "| ${CC[${2:-9}]}                                  /_/                                   ${CC[0]} |";
    lin 1;
    lin 2 "${USER} ";
    lin 2 "${1:-$AAPN} ";
    lin 2 "Version ${2:-$AAPV} - Built: ${3:-$AAPT} ";
    lin 3
}
asetup_colors ()
{
    create_colors;
    ahave dircolors && eval "`dircolors -b`";
    export GREP_COLOR='
cho $R;
    [[ -r $HOME/.dircolors ]] && eval "`dircolors $HOME/.dircolors`" || dircolors --print-database >$HOME/.dircolors;
    local L="no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:*.tar=01;31:*.tgz=01;31:*.svgz=01;31:*.arj=01;31:*.taz=01;31";
    L="${L}:*.lzh=01;31:*.lzma=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31";
    L="${L}:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.mng=01;35";
    L="${L}:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35";
    L="${L}:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.cpio=01;31";
    export LS_COLORS="${L}:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.htaccess=01;31:*.htpasswd=01;31:*.htpasswda1=01;31:*config.php=01;31:*wp-config.php=01;31:"
}

asetup_history ()
{
    export HISTFILE=${HISTFILE:-$HOME/.bash_history};
    export HISTCONTROL=${HISTCONTROL:-'ignoreboth'};
    export HISTSIZE=500;
    export HISTFILESIZE=500;
    export HISTIGNORE='clear:ls:ll:updatedb:top:h2:h1:h3:dir:cd ..:date:exit';
    unset HISTTIMEFORMAT;
    HISTFILEMASTER_DIR=$HOME/backups/.history;
    [[ ! -d "$HISTFILEMASTER_DIR/" ]] && mkdir -pv $HISTFILEMASTER_DIR;
    [[ ! -r ${HISTFILE} ]] && history -w $HISTFILE;
    HISTFILEMASTER=${HISTFILEMASTER_DIR}/combined.log;
    [[ ! -f $HISTFILEMASTER ]] && history -r && history -w $HISTFILEMASTER && history -c && echo "" >$HISTFILE;
    export HISTFILEMASTER;
    HISTFILEMASTER_C=$HISTFILEMASTER_DIR/combined-uniq.log;
    [[ ! -f "$HISTFILEMASTER_C" ]] && echo "" >$HISTFILEMASTER_C;
    export HISTFILEMASTER_C
}
askapache ()
{
    ascript_title;
    aa_motd;
    aa_weather;
    ahave who && pm "Users" && pm "Logged In" 3;
    command who -ar -pld && pm "Current Limits" 3;
    command ulimit -a;
    pm "Machine stats";
    ahave uptime && pm "uptime" 3 && command uptime;
    [[ -d /proc ]] && [[ -f /proc/meminfo ]] && cat /proc/meminfo;
    ahave who && pm "Users" 3 && command who;
    pm "Networking";
    ahave ip && pm "interfaces" 3 && ip -o addr | sed -e 's/ \{1,\}/\t/g';
    [[ -r /proc/net/sockstat ]] && pm "Sockets" 3 && head -n 2 /proc/net/sockstat;
    ahave ss && pm "Networking Stats" 3 && ss -s;
    ahave netstat && pm "Routing Information" 3 && netstat -r;
    pm "Disk";
    pm "Mounts" 3;
    command df -hai;
    ahave iostat && pm "I/O on Disks" && iostat -p ALL;
    pm "Processes";
    pm "process tree" 3;
    command ps -HAcl -F S -A f | uniq -w3;
    ahave procinfo && pm "procinfo" 3 && procinfo | head -n 13 | tail -n 11;
    pm "Functions";
    aa_functions;
    pm "Aliases";
    aa_aliases
}
clean_exit ()
{
    lin 0;
    lin 1;
    lin 2 "COMPLETED SUCCESSFULLY";
    lin 1;
    lin 3
}
cont ()
{
    local ans;
    echo -en "\n\n ${CC[15]}[ ${1:-Press any key to continue} ]$R\n\n";
    read -n 1 ans;
    aa_beep 1
}
create_colors ()
{
    for i in `seq 0 7`;
    do
        ii=$(($i+7));
        CC[$i]="\033[1;3${i}m";
        CC[$ii]="\033[0;3${i}m";
    done;
    CC[15]="\033[30;42m";
    export R=`tput sgr0`;
    export X='';
    export CC
}

curl1 ()
{
    command curl -A 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5' -e -L -H 'Accept: */*' "$@"
}
curl2 ()
{
    ( echo "\n\nurl_effective:\t\t%{url_effective}\ncontent_type:\t\t%{content_type}";
    echo "http_code:\t\t%{http_code}\t\tThe numerical code that was found in the last retrieved HTTPS page";
    echo "http_connect:\t\t%{http_connect}\t\tThe numerical code that was found in the last response from a proxy to a curl CONNECT request";
    echo "time_total:\t\t%{time_total}\t\tThe time will be displayed with millisecond resolution";
    echo "time_namelookup:\t%{time_namelookup}\t\tThe time, in seconds, it took from the start until the name resolving was completed";
    echo "time_connect:\t\t%{time_connect}\t\tThe time, in seconds, it took from the start until the connect to the remote host or proxy was completed";
    echo "time_pretransfer:\t%{time_pretransfer}\t\tThe time, in seconds, it took from the start until the file transfer is just about to begin. This includes all pre-transfer commands and negotiations to the particular protocols involved";
    echo "time_redirect:\t\t%{time_redirect}\t\tThe time, in seconds, it took for all redirection steps. time_redirect shows the complete execution time for multiple redirections";
    echo "time_starttransfer:\t%{time_starttransfer}\t\tThe time, in seconds, it took from the start until the first byte is just about to be transferred. This includes time_pretransfer and also the time the server needs to calculate the result";
    echo "size_download:\t\t%{size_download}\t\tThe total amount of bytes that were downloaded";
    echo "size_upload:\t\t%{size_upload}\t\tThe total amount of bytes that were uploaded";
    echo "size_header:\t\t%{size_header}\t\tThe total amount of bytes of the downloaded headers";
    echo "size_request:\t\t%{size_request}\t\tThe total amount of bytes that were sent in the HTTP request";
    echo "speed_download:\t\t%{speed_download}\tThe average download speed that curl measured for the complete download";
    echo "speed_upload:\t\t%{speed_upload}\t\tThe average upload speed that curl measured for the complete upload";
    echo "num_connects:\t\t%{num_connects}\t\tNumber of new connects made in the recent transfer" ) | curl "$@" -w '@-'
}
curl3 ()
{
    ahave curl && ( curl -A 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5' -e -L -b ~/.curl_cookie.$$ --cookie-jar ~/.curl_cookie.$$ -H 'Accept: */*' "$@" );
    ahave curl || ( command wget `sed -e 's/^-o /-O /g' -e 's/ -o / -O /g' -e 's/^--output /--output-file /g' -e 's/ --output / --output-file /g' <<< "$@"` )
}
cuttail ()
{
    [[ "$#" -lt "1" ]] && echo "Usage: $FUNCNAME " 1>&2 && return 2;
    sed -n -e :a -e "1,${2:-10}!{P;N;D;};N;ba" $1
}
diffdirs ()
{
    [[ "$#" -gt "0" && $1 == *-h* ]] && ahelp $1 && echo "Usage: $FUNCNAME" && return 2;
    [[ "$#" -lt "2" ]] && echo "Usage: $FUNCNAME dir1 dir2" 1>&2 && return 2;
    ahave colordiff && colordiff -urp -w $1 $2
}
dirty_exit ()
{
    echo "See ya..";
    sleep 1;
    exit
}
do_sleep ()
{
    local E D;
    echo -en "${CC[6]}${3:-.}";
    while `command ps -p $RJ &>$N6`; do
        sleep ${2:-3};
        echo -en "${3:-.}";
    done;
    echo -e "${CC[0]}" && sleep 1 && pd
}
dos2unixx ()
{
    [[ $# -eq 0 ]] && exec tr -d '\015\032' || [[ ! -f "$1" ]] && echo "Not found: $1" && return;
    for f in "$@";
    do
        [[ ! -f "$f" ]] && continue;
        tr -d '\015\032' <"$f" >"$f.t" && cmp "$f" "$f.t" >$N6 && rm -f "$f.t" || ( touch -r "$f" "$f.t" && mv "$f" "$f.b" && mv "$f.t" "$f" && rm -f "$f.b" ) >&$N6;
    done
}
dump_ps_environment ()
{
    command ps aux | command grep ${USER:0:3} | command awk '{print $2}' | xargs -t -ipid /bin/sh -c 'test -r /proc/pid/environ && cat /proc/pid/environ | tr "\000" " "'
}
ex ()
{
    [[ "$#" -gt "0" && $1 == *-h* ]] && ahelp $1 && echo "Usage: $FUNCNAME" && return 2;
    until [[ -z "$1" ]]; do
        if [[ -f "$1" ]]; then
            pm "Extracting $1 ...";
            case $1 in
                *.tar.bz2)
                    tar xjf $1
                ;;
                *.tar.gz)
                    tar xzf $1
                ;;
                *.tar)
                    tar xf $1
                ;;
                *.tbz2)
                    tar xjf $1
                ;;
                *.tgz)
                    tar xzf $1
                ;;
                *.bz2)
                    bunzip2 $1
                ;;
                *.rar)
                    unrar x $1
                ;;
                *.gz)
                    gunzip $1
                ;;
                *.zip)
                    unzip $1
                ;;
                *.Z)
                    uncompress $1
                ;;
                *.7z)
                    7z x $1
                ;;
                *)
                    pm "Don't know how to extract '$1'"
                ;;
            esac;
        else
            pm "'$1' is not a valid file";
        fi;
        shift;
    done
}
exitall ()
{
    aa_savehist;
    pkill -9 -t "${SSH_TTY/\/dev\/}"
}
figtest ()
{
    [[ "$#" -gt "0" && $1 == *-h* ]] && ahelp $1 && echo "Usage: $FUNCNAME word" && return 2;
    for a in /usr/share/figlet/*.flf;
    do
        r=`basename ${a%%.flf}`;
        echo -e "${r}";
        figlet -t -f "$r" "$1";
    done
}
find_symlinks ()
{
    local O=$IFS;
    IFS='';
    for file in "$( find ${1:-`pwd`} -type l )";
    do
        echo "$file";
    done | sort
}
get_crypt_user ()
{
    ahave gpg || return;
    local U=`gpg --list-keys|sed -e '/^uid /!d' -e 's/^uid[ ]*\(.*\)/\1/g'` || unknown;
    [[ $# -ne 0 ]] && U=${U//\<*};
    echo -n $U
}
get_latest_revision ()
{
    local URL=${1:-http://svn.automattic.com/wordpress/trunk};
    svn info $URL | grep ^Rev | sed -e 's/Revision: \([0-9]*\)/\1/g'
}
grepc ()
{
    [[ "$#" -gt "0" && $1 == *-h* ]] && ahelp $1 && echo "Usage: $FUNCNAME" && return 2;
    command grep --color=always "$@"
}
grepp ()
{
    [[ "$#" -gt "0" && $1 == *-h* ]] && ahelp $1 && echo "Usage: $FUNCNAME" && return 2;
    command grep -i --color=always "$@" -r .
}

h2 ()
{
    [[ "$#" -lt "1" ]] && echo "Usage: $FUNCNAME query" 1>&2 && return 2;
    command grep -h "$@" $HISTFILEMASTER_C
}
h2c ()
{
    [[ "$#" -lt "1" ]] && echo "Usage: $FUNCNAME query" 1>&2 && return 2;
    command grep --color=always -h "$@" $HISTFILEMASTER_C
}
h2i ()
{
    [[ "$#" -lt "1" ]] && echo "Usage: $FUNCNAME query" 1>&2 && return 2;
    command grep -h -i "$@" $HISTFILEMASTER_C
}
histfilemaster ()
{
    set -vx;
    history -a;
    [[ "`wc -l < $HISTFILE`" -gt "$HISTSIZE" ]] && ( cat $HISTFILE >>$HISTFILEMASTER;
    cat $HISTFILE >`date +$HISTFILEMASTER_DIR/%m-%d-%y.history`;
    cat $HISTFILE $HISTFILEMASTER_C | sort | uniq >$HISTFILEMASTER_DIR/hc.tmp;
    [[ "`wc -l < $HISTFILEMASTER_DIR/hc.tmp`" -gt "`wc -l < $HISTFILEMASTER_C`" ]] && mv $HISTFILEMASTER_DIR/hc.tmp $HISTFILEMASTER_C || cat $HISTFILE >>$HISTFILEMASTER_C;
    echo "" >$HISTFILE );
    set +vx
}
kill_jobs ()
{
    for i in `command jobs -p`;
    do
        kill -9 $i;
    done
}
l ()
{
    command ls -AhFp $AAC "$@"
}
la ()
{
    command du -kh * | awk '{print $2,$1}' | sort -n | tr ' ' "\t"
}
lin ()
{
    local L2 L1='__________________________________________________________________________';
    case ${1:-1} in
        0)
            echo -e "\n ${CC[0]}${L1}"
        ;;
        1)
            L2=`echo '                                                                          '`;
            echo -e "${CC[0]}|${CC[34]}${L2}${CC[0]}|"
        ;;
        2)
            echo -en "${CC[0]}|${CC[34]}";
            echo -en "${2:-1}" | sed -e :a -e 's/^.\{1,72\}$/ & /;ta' -e "s/\(.*\)/\1/";
            echo -e "${CC[0]} |"
        ;;
        3)
            echo -e "${CC[0]} ${L1} $R$X\n\n"
        ;;
    esac
}
ll ()
{
    command ls -lABls1c $AAC "$@"
}
locate1 ()
{
    [[ "$#" -gt "0" && $1 == *-h* ]] && ahelp $1 && echo "Usage: $FUNCNAME" && return 2;
    [[ "$#" -lt "1" ]] && echo "Usage: $FUNCNAME " 1>&2 && return 2;
    command locate "$@" | xargs -iFF stat -c %a\ %A\ \ A\ %x\ \ M\ %y\ \ C\ %z\ \ %N FF
}
make_ionice ()
{
    ahave ionice || return;
    local theclass thenicelevel thepid;
    thepid=${1:-$$};
    theclass=${2:-3};
    thenicelevel=${3:-7};
    pm "Making IONice class: $theclass, nicelevel: $thenicelevel, process: $thepid";
    pm 0 0;
    command ionice -c$theclass -n$thenicelevel -p $thepid;
    pd
}
make_nice ()
{
    ahave renice || return;
    local thenicelevel thepid;
    thepid=${1:-$$};
    thenicelevel=${2:-19};
    pm "Making process $thepid $thenicelevel nice.";
    pm 0 0;
    command renice $thenicelevel -p $theppid;
    pd
}
nh ()
{
    [[ "$#" -lt "1" ]] && echo "Usage: $FUNCNAME <command>" 1>&2 && return 2;
    nohup "$@" >&$N6 & echo
}
nicest ()
{
    [[ "$#" -gt "0" && $1 == *-h* ]] && ahelp $1 && echo "Usage: $FUNCNAME" && return 2;
    local ncmd icmd='';
    ahave ionice && icmd='ionice -c3 -n7';
    ahave nice && ncmd='nice -n 19';
    $icmd $ncmd "$@";
    ahave ionice
}
nob ()
{
    [[ $# -eq 0 ]] && exec grep -v ^$ | sed -e '/[^ \t]\{1,\}/!d'
}
nobb ()
{
    [[ $# -eq 0 ]] && exec sed -e 's/^[ \t]*//;s/[ \t]*$//' -e '/[^ \t]\{1,\}/!d'
}
nobl ()
{
    [[ $# -eq 0 ]] && exec sed -e 's/^[ \t]*//'
}
nobt ()
{
    [[ $# -eq 0 ]] && exec sed -e 's/[ \t]*$//'
}
pd ()
{
    echo -e "\n ${CC[15]} ${1:-DONE} $R\n\n"
}
pm ()
{
    [[ "$#" -lt "1" ]] && echo "Usage: $FUNCNAME " 1>&2 && return 2;
    local I=${1:-3};
    echo -en "$R\n";
    case ${2:-1} in
        0)
            echo -en "${CC[6]}-- $X$R"
        ;;
        1)
            echo -e "${CC[2]}>>> $X$I$R"
        ;;
        2)
            echo -en "${CC[4]} > $X$I$R"
        ;;
        3)
            echo -e "${CC[4]} :: $X$I$R"
        ;;
    esac
}
print_ascii_chart ()
{
    for i in `seq 0 256`;
    do
        echo -e "\\0$(( $i/64*100 + $i%64/8*10 + $i%8 ))";
    done
}
procinfo1 ()
{
    PI=($(strace -s1 procinfo -a 2>&1|sed -e '/^op/!d' -e '/pro/!d' -e '/= -1/d'|sed -e 's%o.*"/proc/\(.*\)".*% \1%g'));
    for i in ${PI[*]};
    do
        echo -e "\n---===[  /proc/$i  ]\n" && cat /proc/$i && echo -e "\n\n";
    done
}
ps ()
{
    [[ -z "$1" ]] && command ps -Hacl -F S -A f && return;
    command ps "$@"
}
pss ()
{
    local U PPS PL PX PXX UUS=($(command ps uax|awk '{print $1}'|command tail -n +2|sort|uniq));
    UL=$((${#UUS[@]} - 1));
    exec 6>&1;
    exec >~/proc.$$;
    ps aux | grep ${USER:0:3} | awk '{print $2}' | xargs -t -ipid cat /proc/pid/environ;
    for UX in $(seq 0 1 $UUS);
    do
        U=${UUS[$UX]};
        PPS=($(pgrep -u ${U}));
        PL=$((${#PPS[@]} - 1));
        for PX in $(seq 0 1 $PL);
        do
            PXX=${PPS[$PX]};
            echo -e "\n\n\n----- PROCESS ID: ${PXX} -----\n\n";
            cat /proc/${PXX}/cmdline 2>$N6 || echo;
            echo -e "\n\n";
            command tree -Csuflapi /proc/Q/${PXX};
        done;
    done;
    exec 1>&6 6>&-;
    cat ~/proc.$$ | more
}
psu ()
{
    command ps -Hcl -F S f -u ${1:-$USER}
}
pwd ()
{
    command pwd -LP "$@"
}
rmb ()
{
    [[ "$#" -lt "1" ]] && echo "Usage: $FUNCNAME dirnametoremove" 1>&2 && return 2;
    ( ( command rm -rf "$@" >&$N6 ) & )
}
set_window_title ()
{
    [[ "$#" -lt "1" ]] && echo "Usage: $FUNCNAME " 1>&2 && return 2;
    echo -n -e "\033]0;$*\007"
}
sleeper ()
{
    [[ "$#" -lt "1" ]] && echo "Usage: $FUNCNAME <process id>" 1>&2 && return 2;
    echo -en "\n${2:-.}";
    while `command ps -p $1 &>$N6`; do
        echo -n "${2:-.}";
        sleep ${3:-1};
    done;
    echo
}
stat1 ()
{
    local D=${1:-$PWD/*};
    stat -c %a\ %A\ \ A\ %x\ \ M\ %y\ \ C\ %z\ \ %N ${D} | sed -e 's/ [0-9:]\{8\}\.[0-9]\{9\} -[0-9]\+//g' | tr -d "\`\'" | sort -r
}
stat2 ()
{
    local D=${1:-$PWD/*};
    stat -c %a\ %A\ \ A\ %x\ \ M\ %y\ \ C\ %z\ \ %N ${D} | sed -e 's/\.[0-9]\{9\} -[0-9]\+//g' | tr -d "\`\'" | sort -r
}
tard ()
{
    [[ "$#" -lt "2" ]] && echo "Usage: $FUNCNAME dest-file (auto adds tgz" 1>&2 && return 2;
    yn "Tarring $2 to $1" && ( ( command tar -zcpf "$1" "$2" >&$N6 ) & )
}
test_colors ()
{
    echo -e "$R";
    ahave tput && tput sgr0;
    for ((i=0; i<=${#CC[@]} - 1; i++))
    do
        echo -e "${CC[$i]}[$i]\n$R";
    done
}
uniq2 ()
{
    [[ -f "$1" ]] && sed = $1 | sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'
}
uniqf ()
{
    [[ "$#" -lt "1" ]] && echo "Usage: $FUNCNAME " 1>&2 && return 2;
    awk '!($0 in a) {a[$0];print}' "$1"
}
uptime1 ()
{
    command uptime | sed '/.*,  \([0-9]*\) users,  load average: \(.*\)/!d; s//[ \2, \1 users ]/;q'
}
vless ()
{
    [[ $# -eq 0 ]] && command vim --cmd 'let no_plugin_maps = 1' -c 'runtime! macros/less.vim' -;
    [[ $# -eq 0 ]] || command vim --cmd 'let no_plugin_maps = 1' -c 'runtime! macros/less.vim' "$@"
}
wcdir ()
{
    [[ "$#" -gt "0" && $1 == *-h* ]] && ahelp $1 && echo "Usage: $FUNCNAME" && return 2;
    find ${1:-.} -mindepth 1 -type d -print0 | xargs -0 -iFF sh -c 'echo `find "FF"/ -type f 2>/dev/null|wc -l;echo "FF"`' | sort -n | sed -e 's/^\([0-9]*\) \(.*\)$/ \1\t\2/g'
}
yes_no ()
{
    [[ "$#" -lt "1" ]] && echo "Usage: $FUNCNAME Question" 1>&2 && return 2;
    local a YN=65;
    echo -en "${1:-Answer} [Y/n] ?";
    read -n 1 a;
    case $a in
        [yY])
            YN=0
        ;;
    esac;
    return $YN
}
yn ()
{
    [[ "$#" -lt "1" ]] && echo "Usage: $FUNCNAME " 1>&2 && return 2;
    local a YN=65;
    echo -en "\n ${CC[6]}@@ ${1:-Q} $R$X[y/N] ?$R";
    read -n 1 a;
    echo;
    case $a in
        [yY])
            echo -n "Y";
            YN=0
        ;;
        *)
            echo -n "N"
        ;;
    esac;
    return $YN
}

Custom bash_profile for Advanced Shell Users originally appeared on .

Today something great and something horrible happened within 5 minutes of each other. I got a letter from Chase and opened it in front of my computer as i am always prepared for the worst when dealing with scum like that. So I quickly scanned the friendly letter and almost exploded, it was a very polite letter happily informing me that my 1.9% APR on my credit card originally gotten from WaMu is from now on going to be 29.99%. That is not a typo.

ally bank

One of the main reasons I wanted to share this on my blog is because take a look at that website! Someone sure knows how to make money, there is alot to admire about this site for you marketers and developers alike. But seriously, getting a money-market account with no minimums, no monthly fees, no direct-deposit requirements, and none of all those other made-up pirate words is pretty unheard of. If I was a journalist I would want to know who it is that is behind this. But knowing how the banking industry is it’s probably their idea of a “lead-generating-tool” that they can later just buy outright and stick it to all the “leads”. It’s basically the same setup as PayPal and their optional money market account that I use, except unlike paypal you can send and receive money without them taking a big chunk. And who ever heard of such a thing as a no-penalty CD that doesn’t have hidden costs?

Have you heard congress wants to make it legally required to inform your blog readers if you are getting paid to promote a product?
Well don’t worry about me, I never have been paid and I literally just opened my account 20 minutes ago. I’m just anal about not having to pay a bank to let them lend it to others. ally vs Chase is the same thing as Open Source vs Microsoft.

Any other ideas?

Update: Best Free Online Banking originally appeared on .

My Setup

As this screenshot shows, I only use a handful of add-ons at a time. These buttons allow me to clear the DNS/Cookies/Cache for whichever site I’m on when I click it. Very very helpful for me as a web designer. The 4th button there is just a restart button. Other than those, Firebug, YSlow, LastPass, and Web Developer are the only ones I always use regularly.

I like the idea of the last.fm but it’s not as powerful as the site, which is awesome. Lately I’ve been listening to Kings of Leon Radio…

Slow Firefox

The more add-ons you have, disabled or not, the slower Firefox is. (unless you are running your profile folder in TMPFS). Also, bookmarks and settings like that have a performance hit. I have been using Firefox since it launched way back when, and I have always kept my bookmarks when moving to new machines and new installations.. So with over 5 thousand bookmarks I finally did some debugging and discovered that was a huge cause of Firefox acting slow. Now I am trying to migrate them all over to Google’s Gmarks, which knowing Google will be awesome eventually.

Multiple Firefox Installations, Sorta

The solution to all these problems is to use separate Firefox profiles which use separate folders to store your profile-specific extensions and settings. So I have profiles with upwards of 40 Addons installed and enabled, and another profile that is built for speed… It’s very very slow to be running Firebug and have multiple tabs open.. You can use the profilemanager to load them specific profiles from the command line. I personally use separate icons on my Windows Quick Launch that I just modified the shortcut pointing to firefox to also have the profile commandline. Note also that you can have multiple profiles open and running simultaneously.. This lets you do some crazy networking and other random hacks like having many Firefox instances running each of which is configured to use a separate SOCKS Proxy or network interface, so you can really open up those pipes for some intense txrx.

You can do some very powerful things with Firefox that most people are unaware of, if you are interested start with these:

• Setting up an extension development environment
• Firefox Command Line Options

About This Collection

Web Development Add-ons for Advanced Web Developers. I personally use these to work on Sites, Servers, WordPress, Javascript, PHP, CSS, XHTML, validation, page-loading, SEO, optimizing, and much more. Add this collection.

Created by:AskApache

Updated:September 22, 2009

Share this Collection

• Digg this!
• Post to Facebook
• Add to Delicious
• Post to MySpace
• Share on FriendFeed
• Post to Twitter

Firefox Add-ons for Web Developers originally appeared on .

Keep in mind that this blog and my research is only a hobby of mine, my job is primarily marketing and sales, so I’m not some licensed expert or anything, or even an unlicensed expert! haha. But it does bother me that those who are tech-savvy enough to run web-hosting companies are happily ripping people off. So this article details the main tools that are used to speed up and optimize your machine by delegating levels of priority to specific processes. Future articles will use these tools alot, so this is meant as an intro.

CPU and Disk I/O

As most of you are aware, there are 2 variables that determine any computer or programs speed. CPU and Disk I/O. CPU determines how fast you can process data, crunch numbers, etc. while disk I/O determines how fast your disks can read and write data to the hard-drive. Wouldn’t it be great if you could easily configure your server to give your httpd, php, and other processes both greater CPU processing and disk IO than your non-important processes like backup scripts, ftp daemons, etc.? We are talking about Linux in this article, so of course YES not only can you do that, you should!

RAM

RAM is like a hard-drive in that data is stored on it, and read/written to it. The difference is that RAM is somewhere around 30x faster than disk I/O, but the cost of that incredible speed is that the data stored on it is only temporary in the sense that it won’t be stored permanently, it is completely erased when your machine is rebooted. RAM is also expensive, and there is a limit to how much a server or machine can have due to hardware limits.

SWAP

SWAP takes off when you run out of RAM but you still want certain data to be read/write quickly. Basically when you start running out of RAM your machine starts supplementing RAM with SWAP storage. SWAP is usually a partition on a second hard-drive disk. There is an upper limit on how much I/O can occur on a disk at one time, and the more I/O takes place, the slower all I/O becomes, so SWAP works well on a separate hard-drive as it will have much faster I/O. On Windows they opted to copy the SWAP mechanism but instead use a file named pagefile.sys, and that is just one reason people in the know do not care for Windows.

CPU

So lets do this, think of your CPU (your processor) as having an amount of 100% processing available when not being used, 0% when its maxed out. CPU’s handle multiple processing tasks simultaneously, so what we will discuss in this article is how to specify HOW MUCH of that processing amount each of your programs (heretofore “processes”) are able to use. Yes, very very cool.

That is correct, you can easily configure your server to provide more of the available processing time to certain programs over others, like you can configure apache and php to utilize 50% of your CPU processing time by themselves, so that all other processes (proftpd, sshd, rsync, etc.) combined can only utilize 50%. The terminology is we can give certain specific processes (like php.cgi, httpd, fast-cgi.cgi) a specific priority, where -19 is the most priority, and +19 is the least amount of priority, or CPU processing time. I know it seems backwards..

The Tools

If you run Windows, you are in the right place… because the following advice will save your life: GET LINUX! Ok, now that that is out of the way, the following are the tools dicussed on this page. All of them are free, open-source, and wonderful. The basic idea of these tools is to control how much CPU is devoted to each process, and also how much Disk IO/Disk traffic is given to each process.

nice

run a program with modified scheduling priority

renice

alter priority of running processes

ionice

set or retrieve the I/O priority for a given pid or execute a new task with a given I/O priority.

iostat

Report Central Processing Unit (CPU) statistics and input/output statistics for devices and partitions.

ulimit

Ulimit provides control over the resources available to processes started by the shell, on systems that allow such control.

chrt

set or retrieve real-time scheduling parameters for a given pid or execute a new task under given scheduling parameters.

taskset

set or retrieve task CPU affinity for a given pid or execute a new task under a given affinity mask.

Part 1: Process Processes Faster

Ok so lets tackle figuring out how to give your response-intensive processes (like apache, php, ruby, perl, java) meaning a request to your server/machine requires a response. For instance, when you requested this page that you are reading at this very second, several things on my server had to happen for you to be able to read this.

First your computer sends out a request to see what server the www.askapache.com domain name is. DNS servers respond with my server IP, so for servers dedicated as nameservers, optimizing the DNS processes like bind would speed that up. Now that your computer knows how to reach my server it sends an HTTP GET request for this url. This request is received by the httpd process that is apache, and apache determines this url should be handled by my custom compiled php5.3.0 binary, because this page is WordPress generated. So the php binary loads up the WordPress /index.php file, which chain-loads several other php files, including wp-config.php containing my MySql database settings. Now php connects to my MySql Server to fetch this articles content, comments, title, tags, etc. and then generates the HTML and hands that back to Apache.

Finally, Apache generates a HTTP RESPONSE and sends the RESPONSE and CONTENT back to your Browser, which then in turn renders the page for your eyes with the necessary javascript, images, css, and other files included in the HTML response.

Too much Processing

Now you see why I’ve opted to write my own caching plugin that takes the php and mysql processes OUT of that equation. Both the php binary and the mysql instance consume CPU processing, and disk IO, to load all their library files, make various network requests and sockets, check permissions, and on and on. And that’s completely ok, the thing is, unless you configure these processes (Apache, PHP, MySQL) they will use the same amount of CPU processing that other processes use, other processes that have very little to do with you reading this sentence. Processes to run my mail server, my FTP server, my SSH server, my cronjobs, cleanup scripts, atd daemon, etc.. and they will get the same amount of CPU!

Another even simpler example is what got me to look into this myself. I wrote a shell script that created hourly, daily, weekly, and monthly backups for all of my websites and sql databases, and set it up to run by cronjob at those set intervals. Eventually I noticed my sites were slower, my php even slower, and sometimes I even saw 503 errors that my host throws up when my server is overloaded. The research that I pursued to prevent that from happening has been hugely eye-opening. What does a backup script do? Mine just created tar archives of all the files in my web root, then gzipped the tar archive saving to a backup server using scp (a file transfer using ssh). This resulted in the following huge problems that seem to have nothing to do with a faster server and speedier website, but they have everything to with it.

1. CPU Bottleneck #1 – tar and gzip use compression algorithms at a low level to create a compressed version, and all that compressing uses a whole lot of crunching – CPU processing
2. DISK IO Bottleneck – Tarring the whole web root directory was creating a ton of disk io, and remember the more disk io that is going on, the less is available for everything else.
3. CPU Bottleneck #2 – Using scp to send my backups was security-smart, but these huge archive files had to be encrypted and sent over the net.

Breaking Bottles

I apologize for being a little long-winded there, but I think it’s important to make sure everyone understands those basic concepts, which are foreign to most people. Once you understand what is causing the bottlenecks, then you can understand the solutions, which actually are incredibly simple and even a novice linux user can easily do. Besides, the net gets a little bit faster every time someone implements this.

nice

Nice allows you to run a program with modified scheduling priority which specifies how much CPU is devoted to a particular process. Run COMMAND with an adjusted niceness, which affects process scheduling. With no COMMAND, print the current niceness.

Nicenesses range from -20 (most favorable scheduling) to 19 (least favorable). -n, --adjustment=N – add integer N to the niceness (default 10). nice +19 tasks get a HZ-independent 1.5%. Running a nice +10 and a nice +11 task means the first will get 55% of the CPU, the other 45%.

nice usage

nice [OPTION] [COMMAND [ARG]...]
 
-n, --adjustment=ADJUST   increment priority by ADJUST first

Examples of nice

Using nice to download a file

nice -n 17 curl -q -v -A 'Mozilla/5.0' -L -O http://wordpress.org/latest.zip

Unzipping a file with nice

nice -n 17 unzip latest.zip

Nice way to build from source

nice -n 2 ./configure
nice -n 2 make
nice -n 2 make install

It is sometimes useful to run non-interactive programs with reduced priority.

$ nice factor `echo '2^9 - 1'|bc`
511: 7 73

Since nice prints the current priority, we can invoke it through itself to demonstrate how it works: The default behavior is to reduce priority by 10.

 $ nice nice
10
$ nice -n 10 nice
10

The ADJUSTMENT is relative to the current priority. The first nice invocation runs the second one at priority 10, and it in turn runs the final one at a priority lowered by 3 more.

$ nice nice -n 3 nice
13

Specifying a priority larger than 19 is the same as specifying 19.

$ nice -n 30 nice
19

Only a privileged user may run a process with higher priority.

$ nice -n -1 nice
nice: cannot set priority: Permission denied
$ sudo nice -n -1 nice
-1

The new scheduler in v2.6.23 addresses all three types of complaints:

To address the first complaint (of nice levels being not “punchy” enough), the scheduler was decoupled from ‘time slice’ and HZ concepts (and granularity was made a separate concept from nice levels) and thus it was possible to implement better and more consistent nice +19 support: with the new scheduler nice +19 tasks get a HZ-independent 1.5%, instead of the variable 3%-5%-9% range they got in the old scheduler.

To address the second complaint (of nice levels not being consistent), the new scheduler makes nice(1) have the same CPU utilization effect on tasks, regardless of their absolute nice levels. So on the new scheduler, running a nice +10 and a nice 11 task has the same CPU utilization “split” between them as running a nice -5 and a nice -4 task. (one will get 55% of the CPU, the other 45%.) That is why nice levels were changed to be “multiplicative” (or exponential) – that way it does not matter which nice level you start out from, the ‘relative result’ will always be the same.

The third complaint (of negative nice levels not being “punchy” enough and forcing audio apps to run under the more dangerous SCHED_FIFO scheduling policy) is addressed by the new scheduler almost automatically: stronger negative nice levels are an automatic side-effect of the recalibrated dynamic range of nice levels.

renice

Renice is similar to the nice command, but it lets you modify the nice of a currently running process. This is nice for shell scripts where you can add this to the top of the script to nicify the whole script to 19.

renice usage

renice priority [ [ -p ] pids ] [ [ -g ] pgrps ] [ [ -u ] users ]
 
-g      Force who parameters to be interpreted as process group ID's.
-u      Force the who parameters to be interpreted as user names.
-p      Resets the who interpretation to be (the default) process ID's.

Examples of renice

From the shell, changes the priority of the shell and all children to 19. From a shell script, does the same but only for the script and its children.

renice 19 -p $$

This runs renice without any output

renice 19 -p $$ &>/dev/null

10 gets more CPU than 19

renice 10 -p $$

change the priority of process ID’s 987 and 32, and all processes owned by users daemon and root.

renice +1 987 -u daemon root -p 32

Part 2: Optimizing Disk I/O

Linux Scheduling Policies

The scheduler is the kernel component that decides which runnable process will be executed by the CPU next. Each process has an associated scheduling policy and a static scheduling priority, sched_priority

Processes scheduled under one of the real-time policies (SCHED_FIFO, SCHED_RR) have a sched_priority value in the range 1 (low) to 99 (high). (As the numbers imply, real-time processes always have higher priority than normal processes.) The following “real-time” policies are also supported, for special time-critical applications that need precise control over the way in which runnable processes are selected for execution:

Currently, Linux supports the following “normal” (i.e., non-real-time) scheduling policies:

SCHED_OTHER: Default Linux time-sharing scheduling

The standard round-robin time-sharing policy

SCHED_BATCH: Scheduling batch processes

This policy is useful for workloads that are non-interactive, but do not want to lower their nice value, and for workloads that want a deterministic scheduling policy without interactivity causing extra preemptions (between the workload’s tasks).

SCHED_IDLE: Scheduling very low priority jobs

This policy is intended for running jobs at extremely low priority (lower even than a +19 nice value with the SCHED_OTHER or SCHED_BATCH policies)

SCHED_FIFO: First In-First Out scheduling

A first-in, first-out policy

SCHED_RR: Round Robin scheduling

A round-robin policy.

Scheduling Classes

IOPRIO_CLASS_RT

This is the realtime io class. The RT scheduling class is given first access to the disk, regardless of what else is going on in the system. Thus the RT class needs to be used with some care, as it can starve other processes. As with the best effort class, 8 priority levels are defined denoting how big a time slice a given process will receive on each scheduling window. This scheduling class is given higher priority than any other in the system, processes from this class are given first access to the disk every time. Thus it needs to be used with some care, one io RT process can starve the entire system. Within the RT class, there are 8 levels of class data that determine exactly how much time this process needs the disk for on each service. In the future this might change to be more directly mappable to performance, by passing in a wanted data rate instead.

IOPRIO_CLASS_BE

This is the best-effort scheduling class, which is the default for any process that hasn’t set a specific io priority. This is the default scheduling class for any process that hasn’t asked for a specific io priority. Programs inherit the CPU nice setting for io priorities. This class takes a priority argument from 0-7, with lower number being higher priority. Programs running at the same best effort priority are served in a round-robin fashion. The class data determines how much io bandwidth the process will get, it’s directly mappable to the cpu nice levels just more coarsely implemented. 0 is the highest BE prio level, 7 is the lowest. The mapping between cpu nice level and io nice level is determined as: io_nice = (cpu_nice + 20) / 5.

IOPRIO_CLASS_IDLE

This is the idle scheduling class, processes running at this level only get io time when no one else needs the disk. A program running with idle io priority will only get disk time when no other program has asked for disk io for a defined grace period. The impact of idle io processes on normal system activity should be zero. This scheduling class does not take a priority argument. The idle class has no class data, since it doesn’t really apply here.

ionice

ionice – get/set program io scheduling class and priority. This program sets the io scheduling class and priority for a program. Since v3 (aka CFQ Time Sliced) CFQ implements I/O nice levels similar to those of CPU scheduling. These nice levels are grouped in three scheduling classes each one containing one or more priority levels:

ionice usage

If no arguments or just -p is given, ionice will query the current io scheduling class and priority for that process.

ionice [-c] [-n] [-p] [COMMAND [ARG...]]

• -c – The scheduling class. 1 for real time, 2 for best-effort, 3 for idle.
• -n – The scheduling class data. This defines the class data, if the class accepts an argument. For real time and best-effort, 0-7 is valid data.
• -p – Pass in a process pid to change an already running process. If this argument is not given, ionice will run the listed program with the given parameters.

ionice Examples

Sets process with PID 89 as an idle io process.

ionice -c3 -p89

Runs ‘bash’ as a best-effort program with highest priority.

ionice -c2 -n0 bash

Returns the class and priority of the process with PID 89

ionice -p89

With the ionice command, you can set the IO priority for a process to one of three classes: Idle (3), Best Effort (2), and Real Time (1). The Idle class means that the process will only be able to read and write to the disk when all other processes are not using the disk. The Best Effort class is the default and has eight different priority levels from 0 (top priority) to 7 (lowest priority). The Real Time class results in the process having first access to the disk irregardless of other process and should never be used unless you know what you are doing.

If we wish to run the updatedb process in the background with an Idle IO class priority, we can run the following:

$ sudo date
$ sudo updatedb &
[1] 16324
$ sudo ionice -c3 -p16324

If we’d rather just lower the Best Effort class priority (defaults to 4) for the command so the process isn’t limited to idle IO periods, we can run the following:

$ sudo date
$ sudo updatedb &
[1] 16324
$ sudo ionice -c2 -n7 -p16324

Again, the Real Time class should not be used as it can prevent you from being able to interact with your system.

You may wonder where you can get the process ID if you don’t know it, can’t remember it, or didn’t start the process (an automatted script may have launched it). You can find process IDs with the ps command.

For example, if I had an updatedb program running in the background, and I wanted to find its process ID, I can run the following:

$ ps -C updatedb
PID TTY TIME CMD
4234 ? 00:00:42 updatedb

This tells me that the process’ process ID (PID) is 4234.

iostat

iostat Usage

iostat [ -c ] [ -d ] [ -N ] [ -n ] [ -h ] [ -k | -m ] [ -t ] [ -V ] [ -x ] [ -z ] [ <device> [...] | ALL ] [ -p [ <device> [,...] | ALL ] ] [ <interval> [ <count> ] ]

-c     The -c option is exclusive of the -d option and displays only the CPU usage report.
-d     The -d option is exclusive of the -c option and displays only the device utilization report.
-k     Display statistics in kilobytes per second instead of blocks per second.  Data displayed are valid only with kernels 2.4 and newer.
-m     Display statistics in megabytes per second instead of blocks or kilobytes per second.  Data displayed are valid only with kernels 2.4 and newer.
-n     Displays the NFS-directory statistic.  Data displayed are valid only with kernels 2.6.17 and newer.  This option is exclusive ot the -x option.
-h     Display the NFS report more human readable.
-p [ { device | ALL } ]   The  -p  option  is  exclusive  of  the -x option and displays statistics for block devices and all their partitions that are used by the system.
-t     Print the time for each report displayed.
-x     Display extended statistics.

iostat Examples

iostat -p ALL 2 1000
avg-cpu:  %user   %nice    %sys %iowait   %idle
            8.34    0.08    1.26    2.27   88.05

Display a single history since boot report for all CPU and Devices.

$ iostat

Display a continuous device report at two second intervals.

$ iostat -d 2

Display six reports at two second intervals for all devices.

$ iostat -d 2 6

Display six reports of extended statistics at two second intervals for devices hda and hdb.

$ iostat -x hda hdb 2 6

Display six reports at two second intervals for device sda and all its partitions (sda1, etc.)

$ iostat -p sda 2 6

Schedule Utils

These are the Linux scheduler utilities – schedutils for short. These programs take advantage of the scheduler family of syscalls that Linux implements across various kernels. These system calls implement interfaces for scheduler-related parameters such as CPU affinity and real-time attributes. The standard UNIX utilities do not provide support for these interfaces — thus this package.

The programs that are included in this package are chrt and taskset. Together with nice and renice (not included), they allow full control of process scheduling parameters. Suggestions for related utilities are welcome, although it is believed (barring new interfaces) that all scheduling interfaces are covered.

I’ve found that quite a few servers do not have this package installed, indicating to you that they might not know what they are doing. Here is how you can install this incredible package, for non-root users. Root users know how to do this, or they shouldn’t be root. Download and install in 1 line provided you have curl. Or just use the following commands.

mkdir -pv $HOME/{dist,source,bin,share/man/man1} && cd ~/dist && curl -O http://ftp.de.debian.org/debian/pool/main/s/schedutils/schedutils_1.5.0.orig.tar.gz && cd ~/source && tar -xvzf ~/dist/sch*z && cd sch* && sed -i -e 's,= /usr/local,=${HOME},g' Makefile && make && make install && make installdoc

mkdir -pv $HOME/{dist,source,bin,share/man/man1}
cd ~/dist && curl -O http://ftp.de.debian.org/debian/pool/main/s/schedutils/schedutils_1.5.0.orig.tar.gz
cd ~/source && tar -xvzf ~/dist/schedutils_1.5.0.orig.tar.gz
cd ~/source/schedutils-1.5.0 && sed -i -e 's,= /usr/local,=${HOME},g' Makefile
make || make -d && make install || make install -d && make installdoc || make installdoc -d

taskset

Taskset is used to set or retrieve the CPU affinity of a running process given its PID or to launch a new COMMAND with a given CPU affinity. CPU affinity is a scheduler property that “bonds” a process to a given set of CPUs on the system. The Linux scheduler will honor the given CPU affinity and the process will not run on any other CPUs. Note that the Linux scheduler also supports natural CPU affinity: the scheduler attempts to keep processes on the same CPU as long as practical for performance reasons. Therefore, forcing a specific CPU affinity is useful only in certain applications.

The CPU affinity is represented as a bitmask, with the lowest order bit corresponding to the first logical CPU and the highest order bit corresponding to the last logical CPU. Not all CPUs may exist on a given system but a mask may specify more CPUs than are present. A retrieved mask will reflect only the bits that correspond to CPUs physically on the system. If an invalid mask is given (i.e., one that corresponds to no valid CPUs on the current system) an error is returned. A user must possess CAP_SYS_NICE to change the CPU affinity of a process. Any user can retrieve the affinity mask.

taskset Usage

taskset [options] [mask | cpu-list] [pid | cmd [args...]]
 
-p, --pid            operate on existing given pid
-c, --cpu-list     display and specify cpus in list format

taskset-examples

The default behavior is to run a new command:

$ taskset 03 sshd -b 1024

You can retrieve the mask of an existing task or set it:

$ taskset -p 700
$ taskset -p 03 700

List format uses a comma-separated list instead of a mask:

$ taskset -pc 0,3,7-11 700

chrt

chrt sets or retrieves the real-time scheduling attributes of an existing PID or runs COMMAND with the given attributes. Both policy (one of SCHED_FIFO, SCHED_RR, or SCHED_OTHER) and priority can be set and retrieved. A user must possess CAP_SYS_NICE to change the scheduling attributes of a process. Any user can retrieve the scheduling information.

chrt Usage

chrt [options] [prio] [pid | cmd [args...]]
 
-p, --pid operate on an existing PID and do not launch a new task
-f, --fifo set scheduling policy to SCHED_FIFO
-m, --max show minimum and maximum valid priorities, then exit
-o, --other set policy scheduling policy to SCHED_OTHER
-r, --rr set scheduling policy to SCHED_RR (the default)

chrt Examples

The default behavior is to run a new command: chrt [prio] -- [command] [arguments]

You can also retrieve the real-time attributes of an existing task:

chrt -p [pid]

Or set them:

chrt -p [prio] [pid]

ulimit – get and set user limits

Ulimit provides control over the resources available to processes started by the shell, on systems that allow such control. One can set the resource limits of the shell using the built-in ulimit command. The shell’s resource limits are inherited by the processes that it creates to execute commands.

ulimit Usage

ulimit [-SHacdfilmnpqstuvx] [limit]

-S

use the `soft’ resource limit

-H

use the `hard’ resource limit

-a

all current limits are reported

-c

the maximum size of core files created

-d

the maximum size of a process’s data segment

-f

the maximum size of files created by the shell

-l

the maximum size a process may lock into memory

-m

the maximum resident set size

-n

the maximum number of open file descriptors

-p

the pipe buffer size

-s

the maximum stack size

-t

the maximum amount of cpu time in seconds

-u

the maximum number of user processes

-v

the size of virtual memory

If LIMIT is given, it is the new value of the specified resource; the special LIMIT values `soft’, `hard’, and `unlimited’ stand for the current soft limit, the current hard limit, and no limit, respectively. Otherwise, the current value of the specified resource is printed. If no option is given, then -f is assumed. Values are in 1024-byte increments, except for -t, which is in seconds, -p, which is in increments of 512 bytes, and -u, which is an unscaled number of processes.

RLIMIT_AS

The maximum size of the process’s virtual memory (address space) in bytes. This limit affects calls to brk(2), mmap(2) and mremap(2), which fail with the error ENOMEM upon exceeding this limit. Also automatic stack expansion will fail (and generate a SIGSEGV that kills the process if no alternate stack has been made available via sigaltstack(2)). Since the value is a long, on machines with a 32-bit long either this limit is at most 2 GiB, or this resource is unlimited.

RLIMIT_CORE

Maximum size of core file. When 0 no core dump files are created. When non-zero, larger dumps are truncated to this size.

RLIMIT_CPU CPU

time limit in seconds. When the process reaches the soft limit, it is sent a SIGXCPU signal. The default action for this signal is to terminate the process. However, the signal can be caught, and the handler can return control to the main program. If the process continues to consume CPU time, it will be sent SIGXCPU once per second until the hard limit is reached, at which time it is sent SIGKILL. (This latter point describes Linux 2.2 through 2.6 behavior. Implementations vary in how they treat processes which continue to consume CPU time after reaching the soft limit. Portable applications that need to catch this signal should perform an orderly termination upon first receipt of SIGXCPU.)

RLIMIT_DATA

The maximum size of the process’s data segment (initialized data, uninitialized data, and heap). This limit affects calls to brk(2) and sbrk(2), which fail with the error ENOMEM upon encountering the soft limit of this resource.

RLIMIT_FSIZE

The maximum size of files that the process may create. Attempts to extend a file beyond this limit result in delivery of a SIGXFSZ signal. By default, this signal terminates a process, but a process can catch this signal instead, in which case the relevant system call (e.g., write(2), truncate(2)) fails with the error EFBIG.

RLIMIT_LOCKS

(Early Linux 2.4 only) A limit on the combined number of flock(2) locks and fcntl(2) leases that this process may establish.

RLIMIT_MEMLOCK

The maximum number of bytes of memory that may be locked into RAM. In effect this limit is rounded down to the nearest multiple of the system page size. This limit affects mlock(2) and mlockall(2) and the mmap(2) MAP_LOCKED operation. Since Linux 2.6.9 it also affects the shmctl(2) SHM_LOCK operation, where it sets a maximum on the total bytes in shared memory segments (see shmget(2)) that may be locked by the real user ID of the calling process. The shmctl(2) SHM_LOCK locks are accounted for separately from the per-process memory locks established by mlock(2), mlockall(2), and mmap(2) MAP_LOCKED; a process can lock bytes up to this limit in each of these two categories. In Linux kernels before 2.6.9, this limit controlled the amount of memory that could be locked by a privileged process. Since Linux 2.6.9, no limits are placed on the amount of memory that a privileged process may lock, and this limit instead governs the amount of memory that an unprivileged process may lock.

RLIMIT_MSGQUEUE

(Since Linux 2.6.8) Specifies the limit on the number of bytes that can be allocated for POSIX message queues for the real user ID of the calling process. This limit is enforced for mq_open(3). Each message queue that the user creates counts (until it is removed) against this limit according to the formula: bytes = attr.mq_maxmsg * sizeof(struct msg_msg *) + attr.mq_maxmsg * attr.mq_msgsize where attr is the mq_attr structure specified as the fourth argument to mq_open(3). The first addend in the formula, which includes sizeof(struct msg_msg *) (4 bytes on Linux/i386), ensures that the user cannot create an unlimited number of zero-length messages (such messages nevertheless each consume some system memory for bookkeeping overhead).

RLIMIT_NICE

(since Linux 2.6.12, but see BUGS below) Specifies a ceiling to which the process’s nice value can be raised using setpriority(2) or nice(2). The actual ceiling for the nice value is calculated as 20 – rlim_cur. (This strangeness occurs because negative numbers cannot be specified as resource limit values, since they typically have special meanings. For example, RLIM_INFINITY typically is the same as -1.)

RLIMIT_NOFILE

Specifies a value one greater than the maximum file descriptor number that can be opened by this process. Attempts (open(2), pipe(2), dup(2), etc.) to exceed this limit yield the error EMFILE. (Historically, this limit was named RLIMIT_OFILE on BSD.)

RLIMIT_NPROC

The maximum number of processes (or, more precisely on Linux, threads) that can be created for the real user ID of the calling process. Upon encountering this limit, fork(2) fails with the error EAGAIN.

RLIMIT_RSS

Specifies the limit (in pages) of the process’s resident set (the number of virtual pages resident in RAM). This limit only has effect in Linux 2.4.x, x < 30, and there only affects calls to madvise(2) specifying MADV_WILLNEED.

RLIMIT_RTPRIO

(Since Linux 2.6.12, but see BUGS) Specifies a ceiling on the real-time priority that may be set for this process using sched_setscheduler(2) and sched_setparam(2).

RLIMIT_RTTIME

(Since Linux 2.6.25) Specifies a limit on the amount of CPU time that a process scheduled under a real-time scheduling policy may consume without making a blocking system call. For the purpose of this limit, each time a process makes a blocking system call, the count of its consumed CPU time is reset to zero. The CPU time count is not reset if the process continues trying to use the CPU but is preempted, its time slice expires, or it calls sched_yield(2). Upon reaching the soft limit, the process is sent a SIGXCPU signal. If the process catches or ignores this signal and continues consuming CPU time, then SIGXCPU will be generated once each second until the hard limit is reached, at which point the process is sent a SIGKILL signal. The intended use of this limit is to stop a runaway real-time process from locking up the system.

RLIMIT_SIGPENDING

(Since Linux 2.6.8) Specifies the limit on the number of signals that may be queued for the real user ID of the calling process. Both standard and real-time signals are counted for the purpose of checking this limit. However, the limit is only enforced for sigqueue(2); it is always possible to use kill(2) to queue one instance of any of the signals that are not already queued to the process.

RLIMIT_STACK

The maximum size of the process stack, in bytes. Upon reaching this limit, a SIGSEGV signal is generated. To handle this signal, a process must employ an alternate signal stack (sigaltstack(2)).

ulimit Examples

Turn off core dumps

ulimit -S -c 0

More Reading

• Please see the SYSSTAT Utilities Home for more performance monitoring tools like sar, sadf, mpstat, iostat, pidstat and sa tools.
• Multitasking from the Linux Command Line + Process Prioritization

Man Pages

1. sched_setscheduler
2. cpuset
3. signal
4. getrlimit
5. ulimit
6. ioprio_get
7. ioprio_set

Kernel Documentation

• information on schedstats (Linux Scheduler Statistics)
• real-time group scheduling
• How and why the scheduler’s nice levels are implemented
• information on scheduling domains
• goals, design and implementation of the Complete Fair Scheduler

Future Discussions:

IO Benchmarking: How, Why and With What

Optimizing Servers and Processes for Speed with ionice, nice, ulimit originally appeared on .

The first article was meant as a detailed and thorough introduction to speeding up Windows-based PC’s in a way that makes it easy to follow, without getting too specific. So make sure you read that first, and pay the most attention to freeing up RAM, CPU, and Disk IO speed by reducing the number of services and processes that are running, we will deal with defragmentation, hard-drive speed, Disk IO, Prefetching, and Pagefile/Registry Defragmentation now.

This article has some really really really great stuff for ya. It shows which tools (all but one completely free) are the best and you will use them for a long time, they are all very good. That is just a side benefit, as this article is really more of a step-by-step guide to optimizing your system that won’t have to be repeated for at least a year. The result of course is a much more responsive PC.

Whats New

After writing that article I continued my research and testing into the subject on my personal computers. I wanted to test out several additional programs and methods before I wrote about them for you guys, and I found a few really sweet additions that had a very big performance gain for all my computers, from my oldest and slowest PC’s to my new 4K power laptop. This article is primarily focused on optimizing your hard drive data and improving your Disk IO speed, and you will definately see an improvement in speed. It doesn’t get REALLY good until the defragmenting section..

1. Clean Up Hard Drive – Removing unneccessary files
2. Clean Registry – Fixing slow registry problems automatically
3. Ultimate Defragmenting – The best defrag method I use
4. Optimize Physical Hard Disk – Final step that cleans and heals your physical disk

Clean Up Hard Drive

The first step is to clean up all the extra, temporary, and unneccessary files cluttering your hard-drive. The reason is because we will be defragmenting your hard-drive like its never been defragged before, then we are going to go over every single bit and byte of your hard-drive to optimize the physical sectors and storage of your data.. Also we will be running a check of your registry and cleaning out bad links and other slow errors, so get it as clutter-free as possible. DON’T use Windows built-in folder compression, it makes defragmentation worse… DO use 7-zip or winrar to create a solid archive file of any misc directories with a bunch of files… You only need one program to clean your system.

CCleaner

CCleaner is a Small, Fast and Free software that removes unused and temporary files from your system and allows Windows to run faster, more efficiently and gives you more hard disk space. I’ve now been using it for several months and love it. As well as cleaning up old files and settings left by standard Windows components, CCleaner also cleans temporary files and recent file lists for many applications. Including: Firefox, Opera, Safari, Media Player, eMule, Kazaa, Google Toolbar, Netscape, Microsoft Office, Nero, Adobe Acrobat Reader, WinRAR, WinAce, WinZip and more… Google Chrome, Opera, Safari, etc..

• Recycle Bin, Clipboard
• Windows Temporary files, Windows Log files, Chkdsk file fragments
• Recent Documents (on the Start Menu), Run history (on the Start Menu)
• Windows XP Search Assistant history, old Prefetch data, Windows memory dumps after crashes

Download CCleaner

Registry Cleaning

The registry is Windows biggest mistake, (although I’m sure they like it), and basically holds all the information for your programs and Windows. Things like the size of your windows, recent file lists, icon files for different icons, etc.. I’ve never seen a computer that didn’t have some registry issues, so this needs to be cleaned and may have a huge impact on your speed. Real quickly, here are some programs to backup and restore your registry, optimize and defrag your registry, and finally search and clean any errors in your registry.

CCleaner

Yup! CCleaner also takes care of most of the performance issues of your registry. It’s very safe and fast. CCleaner uses an advanced Registry Cleaner to check for problems and inconsistencies. It checks the following:

• ClassIDs, ProgIDs, Application Paths, Icons
• Uninstallers, Shared DLLs, Fonts, Help File references
• File Extensions, ActiveX Controls, Invalid Shortcuts and more…

Download CCleaner

ERUNT – The Emergency Recovery Utility NT

ERUNT is a Registry Backup and Restore utility for Windows NT/2000/2003/XP. I use this to backup my registry automatically or on command.. Ive used it for years and it’s always good to backup before you do anything.

# Here's the command I use (only for advanced users familiar with autoback)
"%ProgramFiles%\ERUNT\AUTOBACK.EXE" %SystemRoot%\ERDNT\#Date# sysreg curuser otherusers /noconfirmdelete /noprogresswindow /days:45 /alwayscreate

Download ERUNT · (Details)

NTGREGOPT – NT Registry Optimizer

NTGREGOPT is a Registry Optimization tool for Windows NT/2000/2003/XP/Vista that minimizes the size of your registry files by simply compacting the registry hives to the minimum size possible.

Registry files in an NT-based system can become fragmented over time, occupying more space on your hard disk than necessary and decreasing overall performance. You should use the NTREGOPT utility regularly, but especially after installing or uninstalling a program, to minimize the size of the registry files and optimize registry access. The program works by recreating each registry hive “from scratch”, thus removing any slack space that may be left from previously modified or deleted keys. It does NOT change the contents of the registry in any way, nor does it physically defrag the registry files on the drive. I recommend using this once every couple weeks. I scheduled it to run automatically.

Download NTGREGOPT · (Details)

Ultimate Defragmenting

I say “Ultimate Defragmenting” because this is the result of a lot of testing of all the various defragmenting software out there, reading a lot of documentation, and running benchmarking to find the fastest results. This is a mix of several individual defragmenting steps combined for a once-a-year ultimate defragmenting session. This is what I use today, and although it’s altogether a long process, each step you’ll add a new tool or skill that you can use by itself from here on out.

ATTENTION: While running MyDefrag/JkDefrag, SpinRite, and UltraDefrag your computer can get very hot and that is very not cool. I set my laptop on a coke can and pointed a small desk fan at it which kept it very very cool, so do what you can to minimize heat during these programs.

PageDefrag

PageDefrag uses advanced techniques to provide you what commercial defragmenters cannot: the ability for you to see how fragmented your paging files and Registry hives are, and to defragment them. In addition, it defragments event log files and Windows 2000/XP hibernation files (where system memory is saved when you hibernate a laptop). One of the limitations of the Windows NT/2000 defragmentation interface is that it is not possible to defragment files that are open for exclusive access. Thus, standard defragmentation programs can neither show you how fragmented your paging files or Registry hives are, nor defragment them. Paging and Registry file fragmentation can be one of the leading causes of performance degradation related to file fragmentation in a system.

I personally keep this enabled for every boot, as it only takes a few seconds after the first time it’s run.

Download PageDefrag

MyDefrag

JkDefrag is a disk defragmenter and optimizer for Windows 2000/2003/XP/Vista/2008/X64. Completely automatic and very easy to use, fast, low overhead, with several optimization strategies, and can handle floppies, USB disks, memory sticks, and anything else that looks like a disk to Windows. Included are a Windows version, a commandline version (for scheduling by the task scheduler or for use from administrator scripts), a screensaver version, a DLL library (for use from programming languages), versions for Windows X64, and the complete sources. (Frequently Asked Questions)

After trying out dozens of degragmenting programs, this is my favorite. I utilize the cool screensaver function and just run sometimes when I’m calling it a day.

Tips and tricks

• Many users start looking for defragmentation/optimization programs when their computer becomes slow. The main reason for a slow computer is a full harddisk. A full harddisk is slow because the distance between files is greater than on a fresh practically empty harddisk. Deleting half the data on a full disk will just about double the speed. The more free diskspace, the faster your computer will be.
• Buy a second harddisk (for example an USB harddisk) and move little used stuff from your primary harddisk to that secondary harddisk. The second disk can also be used for backing up the primary disk.
• When buying a new computer, buy the biggest harddisk you can afford. Investing in a bigger harddisk gives more speed-per-dollar than investing in a faster CPU or investing in more memory.
• Cleanup old junk from your harddisk before running MyDefrag. You can clean Windows files with for example “Start -> Programs -> Accessories -> System Tools -> Disk Cleanup”, or with something like the freeware CCleaner program.
• Reboot before running MyDefrag. This will release files that are in use, so they can be defragmented and optimized.
• Boot into Windows safe mode by pressing F8 when booting, and then run MyDefrag. It will be slower because the Windows disk cache is off in safe mode, but MyDefrag will be able to process (a few) more files.
• Stop your real time virus scanner before running MyDefrag. Virus scanners check all disk activity, making defragmentation and optimization very slow.
• Move the swap file to another volume, reboot, defragment, and move the swap file back. If you don’t have a second volume then temporarily make the swap file small, for example 100Mb.
• Package unused files with a packager such as 7-zip. The packagefile not only takes less harddisk space, but will also defragment and optimize much faster than the individual files. Note: This does not apply to Windows NTFS compression, which will actually make defragmentation and optimization slower.
• The first partition on a harddisk is significantly faster than other partitions. Try to use other partitions only for data that is used less often, such as music, movies, archives, backups, logfiles.
• If you have 2 physical harddisks of the same speed, then place the pagefile on the first partition of the second harddisk.

The way I recommend is to run MyDefrag at the highest level of defragmentation once, which took my fastest PC almost 30 hours. Once that is done you can just run it normally in 20 minutes or so.. This software also has the best defrag information I’ve found to date, so check out the documentation on the site.

Download MyDefrag

UltraDefrag

UltraDefrag is a powerful Open Source Defragmentation tool for Windows Platform. It is very fast, because the defragmenting is done via the kernel-mode driver. There are three interface available : Graphical, Console and Native. I personally like the MyDefrag more because I think it does a better job, but I also use the UltraDefrag tool because it has one very important feature like the PageDefrag tool. It has a native version. That means it can run before Windows loads up by utilizing the bootexecute, the same place that windows chkdsk runs at boot. It also can takeover for Windows builtin prefetcher, to speed up the loading of frequently used programs, which I’ll explain a bit later.

I set ultradefrag up after the MyDefrag 30hour defrag completes, to run at boot and control the prefetching, then I erase any prefetch files currently saved and reboot which lets it defrag the system.

erase /Q "%SYSTEMROOT%\Prefetch\*.*"

Once both MyDefrag and UltraDefrag have run THEN I finally login to windows and don’t open any programs to let the windows OS files get optimized by not doing anything at all for 5 minutes.. Then I reboot and log back in and this time I don’t do anything for 30 minutes and reboot. Finally I log back in and this time I instantly load up 10 of my most frequently used programs, (DreamWeaver, Photoshop, Firefox, Chrome, Notepad2, Thunderbird, Internet Explorer, and a few others) and once they are all loaded I don’t do anything for an hour. Then I reboot and repeat that same process.

This may seem odd or made up but I do my research and this allows your prefetched files to be optimized, including your boot prefetch files. Once that is done I reboot and run all the defrags again. Then I reboot and am ready for the last step.

Download UltraDefrag

Optimize Physical Hard Disk

Now at this point the system is defragged and optimized as much as I can get it, but the last step is to run a program to go over every single bit on our hard-drive-disk to keep the drive clean and healthy and its too technical for me to understand, I just know its amazingly cool and I noticed a big change right away.

HD Tune

HD Tune is a fantastic little utility that you can use to benchmark the DISK IO speed of your various drives, internal and external, fixed and USB, firewire, etc.. Other than using it to determine your fastest drives for moving your program files and temps to, I am just including it in this article because it is an awesome program that you will love.

Download HD Tune

SpinRite

This is the last step in this guide, and was the one thing that surprised me the most in terms of how much of a speed improvement I noticed after using it. SpinRite is the most capable, thorough, and reliable utility that has ever been created for the long term maintenance, recovery, and repair of mass storage systems. SpinRite is not a drive defragmenter. SpinRite operates with the drive’s built-in intelligence to reassign and relocate defective sectors without creating file system fragments. Thus, running SpinRite does not create fragments, but neither does it eliminate any that may exist before it was run. Unlike any other disk utility, SpinRite interfaces directly to the hard disk system’s hardware, rather than working through the system’s operating system or BIOS. FAQ.

The way that we use SpinRite in this article is method #4, Drive Maintenance mode, which reads and writes and verifies every single sector and area of your hard drives, and improves the health of your hard drive a lot. Even on my new 4K dell power laptop, this had a noticeable improvement on speed. After running this 20+ hours for my fastest PC, I rebooted, defragged with jkdefrag, and that is the end of this article.

In case you don’t already know . . . What is SpinRite?

SpinRite is a stand-alone DOS application that specializes in the recovery of marginally or completely unreadable hard and floppy disk data, and in the lifetime maintenance of PC mass storage devices. It earned its stripes many years ago by introducing the concept of non-destructive low-level reformatting and sector interleave optimization. Since then its capabilities have continued to broaden until it has become the premiere tool for disk data recovery and magnetic mass storage drive maintenance. Written in assembly language, SpinRite still performs as well on a clunky old 4.77 megahertz PC/XT as on a screaming 333 megahertz Pentium II.

While SpinRite 6.0 is running, you can toggle through seven displays:

Purchase and Download SpinRite

More Reading

• PCNet File Catch – SpinRite 6.0
• SpinRite 6.0 for Linux Users
• Anticipate Drive Problems Early with SpinRite v6.0
• Black Viper – Windows XP Super Tweaks
• Black Viper – Optimize your Services
• Advanced Windows Shell Tutorial
• Batch Files (Scripts) in Windows
• File System Heirarchy
• Steve Gibson discussing Defragmenting

Windows Optimization – Intense Part II originally appeared on .

Note: I spent no time on readability, this is primarily a read the code and figure it out article.. This is for advanced users looking for a reference or discussion and for those of you looking to advance. Feedback would be great if you make it that far..

For a better handle on the way I like to structure web site directories, see Optimize a Website for Speed, Security, and Easy Management but note it is a bit outdated compared to what I’m doing now. I don’t have the luxury of using only one type of server, or hosting provider anymore, so I have been working towards making things even more portable in order to move from host to host from server to server without issues i.e. my portable .bash_profile.

So I’ve been basically experimenting various ways to accomplish that and thought I would share what I am currently doing for my benefit and hopefully get some input. All of my WP installs run the development version, and one main idea with my setups is that upgrading is automated. So I really keep the WordPress install clean and use plugins and wp-config.php to do all the customization.

• Portability – Hands-free upgrades and easy to move
• Security – Additional security and protection
• Speed – Less CPU and Disk I/O
• Customization – All my favorite customizations

wp-config.php

These are the main settings I use.. Seriously this is more like an interactive article, because to understand it you will need to do some code grepping. You may want to grab a jolt.

ASKAPACHE_ROOT

The ASKAPACHE_ROOT variable is just a better way for me to be able to include and access all the different files in my site tree. For instance, in my non-wp php files, I can do this:

!defined('ASKAPACHE_ROOT') && require $_SERVER['DOCUMENT_ROOT'] . '/wp-config.php';
include(ASKAPACHE_ROOT . '/includes/custom-download.inc.php');

ASKAPACHE_LOCK

This is one of my all-time favorite hacks, that I think is one of the most useful methods I employ as a web developer. This allows me to use far-future-expire headers for optimum caching, while still forcing browsers to re-validate every day or so automatically, or forcing them to re-validate whenever I change the suffix. This takes advantage of the mod_rewrite trick that I use on EVERY site I run, definately worth learning. Because I practice best-practice web-standards, for every web site I create a single css file and javascript file, which I then add to the template like:

<link rel="stylesheet" type="text/css" media="all" href="http://static.askapache.com/c/apache-0&lt;?php echo ASKAPACHE_LOCK?&gt;.css" />
<script src="http://static.askapache.com/j/apache-0&lt;?php echo ASKAPACHE_LOCK;?&gt;.js" type="text/javascript"></script>

<?php
/**
 * The base configurations of the WordPress.
 *
 * This file has the following configurations: MySQL settings, Table Prefix,
 * Secret Keys, WordPress Language, and ABSPATH. You can find more information by
 * visiting {@link http://codex.wordpress.org/Editing_wp-config.php Editing
 * wp-config.php} Codex page. You can get the MySQL settings from your web host.
 *
 * This file is used by the wp-config.php creation script during the
 * installation. You don't have to use the web site, you can just copy this file
 * to "wp-config.php" and fill in the values.
 *
 * @package WordPress
 */
/* http://codex.wordpress.org/Editing_wp-config.php */
 
/** /home/liet/askapache.com */
!defined('ASKAPACHE_ROOT') && define('ASKAPACHE_ROOT', str_replace('/public_html','', $_SERVER['DOCUMENT_ROOT']));
 
/** The 008 at the end is for manual tweaking.  time() returns seconds since '00:00:00 1970-01-01 UTC'. */
// http://www.askapache.com/htaccess/mod_rewrite-fix-for-caching-updated-files.html
!defined('ASKAPACHE_LOCK') && define(ASKAPACHE_LOCK', substr(time(),0,5).'008'); // 12533001
 
/** absolute path to the Wordpress directory */
!defined('ABSPATH') && define('ABSPATH', ASKAPACHE_ROOT .'/public_html/');
 
/**
 * WP_SITEURL, defined since WordPress Version 2.2, allows the WordPress address (URL) to be defined. The valued defined is the address where your WordPress core files reside.
 * It should include the http:// part too. Do not put a slash "/" at the end.
 * Setting this value in wp-config.php overrides the wp_options table value for siteurl and disables the WordPress address (URL) field in the Administration > Settings > General panel.
 */
!defined('WP_SITEURL') && define('WP_SITEURL', 'http://'.$_SERVER['SERVER_NAME']);
 
/**
 * WP_HOME is another wp-config.php option added in WordPress Version 2.2. Similar to WP_SITEURL,
 * WP_HOME overrides the wp_options table value for home but does not change it permanently.
 * home is the address you want people to type in their browser to reach your WordPress blog. It should include the http:// part. Also, do not put a slash "/" at the end.
 */
!defined('WP_HOME') && define('WP_HOME', WP_SITEURL);
 
/** no trailing slash, full paths only */
!defined('WP_CONTENT_DIR') && define( 'WP_CONTENT_DIR', ABSPATH . 'wp-content' );
 
// full url - WP_CONTENT_DIR is defined further up
!defined('WP_CONTENT_URL') && define( 'WP_CONTENT_URL', WP_SITEURL . '/wp-content');
 
/** Allows for the plugins directory to be moved from the default location. @since 2.6.0 */
// full path, no trailing slash
!defined('WP_PLUGIN_DIR') && define( 'WP_PLUGIN_DIR', WP_CONTENT_DIR . '/plugins' );
 
/** Allows for the plugins directory to be moved from the default location. @since 2.6.0 */
// full url, no trailing slash
!defined('WP_PLUGIN_URL') && define( 'WP_PLUGIN_URL', WP_CONTENT_URL . '/plugins' );
 
/** Allows for the plugins directory to be moved from the default location. @since 2.1.0 */
// Relative to ABSPATH.  For back compat.
//!defined('PLUGINDIR') && define( 'PLUGINDIR', 'wp-content/plugins' );
 
/** Number of autosaves to save. TRUE is default and enables post revisions, FALSE disables revisions completely. */
!defined('WP_POST_REVISIONS') && define('WP_POST_REVISIONS', 150);
 
/* ini_set('memory_limit', WP_MEMORY_LIMIT); */
!defined('WP_MEMORY_LIMIT') && define('WP_MEMORY_LIMIT', '64M');
 
/** Only check at this interval for new messages. Default is 5min */
/** @since 2.9  */
!defined('WP_MAIL_INTERVAL') && define('WP_MAIL_INTERVAL', 3600); // 1 hour
 
/** Saves updated post values to post from edit window every x seconds. (default 60)
 * When editing a post, WordPress uses Ajax to auto-save revisions to the post as you edit. You may want to increase this setting for longer delays in between auto-saves, or decrease the setting to make sure you never lose changes.
 * @since 2.5.0 */
!defined( 'AUTOSAVE_INTERVAL' ) && define( 'AUTOSAVE_INTERVAL', 60 );
 
/** @since 2.9.0  */
/** Permanently deletes posts, pages, attachments, and comments which have been in the trash for EMPTY_TRASH_DAYS. */
!defined( 'EMPTY_TRASH_DAYS' ) && define( 'EMPTY_TRASH_DAYS', 300 );

Debugging WordPress

One of my secrets for getting really good at this stuff is to master debugging. There is really not ever a time when I am working on a site that I don’t have color-highlighted logs scrolling automatically in an ssh window. It’s really almost impossible to fix problems with wordpress or do any kind of advanced anything without being able to view debugging info. At first I relied heavily on a custom php.ini being available on the server, but after having to deal with many hosts who don’t allow php.ini files I now rely completely on setting values using ini_set for ultimate portability. Detailed towards the end of this article and is also included in this wp-config.php

/**#@+
 * DEBUGGING STUFF
 */
/** display of notices during development. if false, error_reporting is E_ERROR | E_WARNING | E_PARSE | E_USER_ERROR | E_USER_WARNING | E_RECOVERABLE_ERROR otherwise E_ALL */
!defined('WP_DEBUG') && define('WP_DEBUG', false);
 
/** The SAVEQUERIES definition saves the database queries to a array and that array can be displayed to help analyze those queries.
 *  The information saves each query, what function called it, and how long that query took to execute.  */
!defined('SAVE_QUERIES') && define('SAVE_QUERIES', WP_DEBUG);
 
!defined('ACTION_DEBUG') && define('ACTION_DEBUG', WP_DEBUG);
 
/** This will allow you to edit the scriptname.dev.js files in the wp-includes/js and wp-admin/js directories.  */
!defined('SCRIPT_DEBUG') && define('SCRIPT_DEBUG', WP_DEBUG);

 
/** Add define('WP_DEBUG_LOG', true); to enable php debug logging to WP_CONTENT_DIR/debug.log */
//!defined('WP_DEBUG_LOG') && define('WP_DEBUG_LOG', true);
 
/** This determines whether errors should be printed to the screen as part of the output or if they should be hidden from the user.
 *  Add define('WP_DEBUG_DISPLAY', false); to wp-config.php to use the globally configured setting for display_errors and not force it to On */
!defined('WP_DEBUG_DISPLAY') && define('WP_DEBUG_DISPLAY', false);

Ultimate Security Tweaks

Well, ultimate for WP’s built-in keys and password functions, this is all for wp-config.php keep in mind. This is a very neccessary and recommended step, and is one of the only things I modify for each new installation.

Security KEYS

If like me you are familiar with password-cracking software like John the ripper, rainbow hash tables, l0pht-crack, etc.. then you will like to know that you can specify your own keys and salts for the encryption used by WP. They are AUTH_KEY, AUTH_SALT, SECURE_AUTH_KEY, SECURE_AUTH_SALT, LOGGED_IN_KEY, LOGGED_IN_SALT, NONCE_KEY, NONCE_SALT, SECRET_KEY and SECRET_SALT.

A random and long key gives you better encryption, and exponentially increasing that is using a random and long salt for the encryption. Encryptions with known salts are incredibly easy to decrypt compared to encryptions with secure salts, because the salt + key individually need to be guessed in order to find a matching hash, vs. just the key if the salt is known. See: Locating weak passwords.

A secret key is a hashing salt which makes your site harder to hack and access harder to crack by adding random elements to the password.

In simple terms, a secret key is a password with elements that make it harder to generate enough options to break through your security barriers. A password like “password” or “test” is simple and easily broken. A random, unpredictable password such as “88a7da62429ba6ad3cb3c76a09641fc” takes years to come up with the right combination.

For more information on the technical background and breakdown of secret keys and secure passwords, see:

• WordPress Support Forum – HOWTO: Set up secret keys in WordPress 2.6+
• Wikipedia’s explanation of Password Cracking

I like to use the WordPress.org secret-key service 4 times. That’s because for each key and salt I like to do: (1 key from api +random keyboard input+1 key from api).

/**#@+
 * Authentication Unique Keys.
 *
 * Change these to different unique phrases!
 * You can generate these using the {@link https://api.wordpress.org/secret-key/1.1/ WordPress.org secret-key service}
 * You can change these at any point in time to invalidate all existing cookies.
 * This will force all users to have to log in again.
 *
 * @since 2.6.0
 *
 * Get salt to add to hashes to help prevent attacks.
 *
 * The secret key is located in two places: the database in case the secret key
 * isn't defined in the second place, which is in the wp-config.php file. If you
 * are going to set the secret key, then you must do so in the wp-config.php
 * file.
 *
 * The secret key in the database is randomly generated and will be appended to
 * the secret key that is in wp-config.php file in some instances. It is
 * important to have the secret key defined or changed in wp-config.php.
 *
 * If you have installed WordPress 2.5 or later, then you will have the
 * SECRET_KEY defined in the wp-config.php already. You will want to change the
 * value in it because hackers will know what it is. If you have upgraded to
 * WordPress 2.5 or later version from a version before WordPress 2.5, then you
 * should add the constant to your wp-config.php file.
 *
 * Below is an example of how the SECRET_KEY constant is defined with a value.
 * You must not copy the below example and paste into your wp-config.php. If you
 * need an example, then you can have a
 * {@link https://api.wordpress.org/secret-key/1.1/ secret key created} for you.
 *
 * Salting passwords helps against tools which has stored hashed values of
 * common dictionary strings. The added values makes it harder to crack if given
 * salt string is not weak.
 *
 * @since 2.5
 * @link https://api.wordpress.org/secret-key/1.1/ Create a Secret Key for wp-config.php
 *
 * @return string Salt value from either 'SECRET_KEY' or 'secret' option
 */
define('AUTH_KEY',        'jflkhaskljdfhkljasdhflkjashd;flkjhas;djfh;kajshdflkjashdlfkjhasdlkfhal?p[B+GR{@>{Yq`c|LnG;dvq#| %OA_cbBSU6,rICC1o/c)-|');
define('SECURE_AUTH_KEY', 'jflkhaskljdfhkljasdhflkjashd;flkjhas;djfh;kajshdflkjashdlfkjhasdlkfhal?Vp[Bb15baar8&R-r<[T|?(xhJJABGq+Ux+U$)-Hltp/');
define('LOGGED_IN_KEY',   'jflkhaskljdfhkljasdhflkjashd;flkjhas;djfh;kajshdflkjashdlfkjhasdlkfhal?Vp[B<5n6DG|YWnJ9tY2!M1L)`{-$LW~~Ia%.uCbn!P. 41o2$Z$4');
define('NONCE_KEY',       'jflkhaskljdfhkljasdhflkjashd;flkjhas;djfh;kajshdflkjashdlfkjhasdlkfhal?Vp[Bgu<wM*zewR0.{+m:bmrB?wj!B,4]Wo+4 Avk ApR-D?E');
define('SECRET_KEY',     'jflkhaskljdfhkljasdhflkjashd;flkjhas;djfh;kajshdflkjashdlfkjhasdlkfhal?Vp[B52ugH6muE9r4._iZwoYKUybrqLPpv|d Xr+|yrqhUE');
 
define('AUTH_SALT',        '123423190847olqkfhladhfsldshafasdfasdf09a7f-90a87df98adfyapoiyaf9asd8f70a9s8d7f908a7sdf97W4qCdm~Ky%+%~PPa5b YEmDI%U[W!-B');
define('SECURE_AUTH_SALT', '123423190847olqkfhladhfsldshafasdfasdf09a7f-90a87df98adfyapoiyaf9asd8f70a9s8d7f908a7sdf97W4qCdmad/7o6.AU3%9o-|Kqm]+eUqr-n~:ag');
define('LOGGED_IN_SALT',   '123423190847olqkfhladhfsldshafasdfasdf09a7f-90a87df98adfyapoiyaf9asd8f70a9s8d7f908a7sdf97W4qCdmsLiCv@KJ{#wd(?qe(KcH3!');
define('NONCE_SALT',       '123423190847olqkfhladhfsldshafasdfasdf09a7f-90a87df98adfyapoiyaf9asd8f70a9s8d7f908a7sdf97W4qCdmG9>+wm 2)bS0Pd_+1rx0brX]ND8|');
define('SECRET_SALT',      '123423190847olqkfhladhfsldshafasdfasdf09a7f-90a87df98adfyapoiyaf9asd8f70a9s8d7f908a7sdf97W4qCdm2<>))U|sty)+4vpWooKls/^[vN');
/**#@-*/

Using SSL for Admin and Login

SSL is kinda required from my point of view, it is just way to easy to sniff data off the wire otherwise. At least with SSL you force them to use tools like burpsuite, paros proxy, webscarab, etc..

/** @since 2.6.0  */
!defined('FORCE_SSL_ADMIN') && define('FORCE_SSL_ADMIN', true);
 
/** @since 2.6.0  */
!defined('FORCE_SSL_LOGIN') && define('FORCE_SSL_LOGIN', true);

Mod_Rewrite to Force SSL

This is pretty cool, it forces non-https for all urls except for /wp-admin and wp-login.php, which both require https. It also checks for the logged_in_cookie, and if that is present in the request then it doesn't force non-https. Kinda confusing if you don't have a mod_rewrite cheatsheet.

RewriteCond %{THE_REQUEST} ^$ [OR]
RewriteCond %{REQUEST_URI} ^/(wp-admin|wp-login\.php).*$ [NC,OR]
RewriteCond %{HTTP_COOKIE} ^.*wp_li_sadfsdfasdf11b361cdsdfasdfasd=.*$ [NC]
RewriteRule .* - [S=1]
 
RewriteCond %{HTTPS} =on [OR]
RewriteCond %{HTTP_HOST} !^www\.askapache\.com$ [NC]
RewriteRule .* http://%{SERVER_NAME}%{REQUEST_URI} [R=301,L]
 
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /(wp-admin/.*|wp-login\.php.*)\ HTTP/ [NC]
RewriteCond %{HTTPS} !=on
RewriteRule .* https://%{SERVER_NAME}%{REQUEST_URI} [R=301,L]

File System Permissions

You can get a basic and solid intro on file permissions by reading: Changing File Permissions, or you can check out some of my file permission research.

/** The permissions as octal number, usually 0644 for files, 0755 for dirs.
 *  http://codex.wordpress.org/Changing_File_Permissions
 *  if ( !$wp_filesystem->mkdir($remote_destination, FS_CHMOD_DIR) )
 */
!defined('FS_CHMOD_DIR') && define('FS_CHMOD_DIR', (0755 & ~ umask()));
!defined('FS_CHMOD_FILE') && define('FS_CHMOD_FILE', (0644 & ~ umask()));
/**#@-*/
 
/** Define the timeouts for the connections. Only available after the construct is called to allow for per-transport overriding of the default. */
//stream_set_timeout( $stream, FS_TIMEOUT );
//!defined('FS_TIMEOUT') && define('FS_TIMEOUT', 30);
 
//$this->link = @ftp_connect($this->options['hostname'], $this->options['port'], FS_CONNECT_TIMEOUT);
//!defined('FS_CONNECT_TIMEOUT') && define('FS_CONNECT_TIMEOUT', 30);
 
// function get_filesystem_method($args = array(), $context = false) {
//  $method = defined('FS_METHOD') ? FS_METHOD : false; //Please ensure that this is either 'direct', 'ssh', 'ftpext' or 'ftpsockets'
//!defined('FS_METHOD') && define('FS_METHOD', 'direct');
 
/** These methods for the WordPress core, plugin, and theme upgrades try to determine the WordPress path, as reported by PHP, but symlink trickery can sometimes
 * 'muck this up' so if you know the paths to the various folders on the server, as seen via your FTP user, you can manually define them in the wp-config.php file.
 * FS_METHOD forces the filesystem method. It should only be "direct", "ssh", "ftpext", or "ftpsockets".
 * FTP_BASE is the full path to the "base" folder of the WordPress installation.
 * FTP_CONTENT_DIR is the full path to the wp-content folder of the WordPress installation.
 * FTP_PLUGIN_DIR is the full path to the plugins folder of the WordPress installation.
 * FTP_PUBKEY is the full path to your SSH public key.
 * FTP_PRIKEY is the full path to your SSH private key.
 * FTP_USER is either user FTP or SSH username. Most likely these are the same, but use the appropriate one for the type of update you wish to do.
 * FTP_PASS is the password for the username entered for FTP_USER. If you are using SSH public key authentication this can be omitted.
 * FTP_HOST is the hostname:port combination for your SSH/FTP server. The standard FTP port is 21 and the standard SSH port is 22.
 */
//define('FS_METHOD', 'ftpext');
//define('FTP_BASE', '/path/to/wordpress/');
//define('FTP_CONTENT_DIR', '/path/to/wordpress/wp-content/');
//define('FTP_PLUGIN_DIR ', '/path/to/wordpress/wp-content/plugins/');
//define('FTP_PUBKEY', '/home/username/.ssh/id_rsa.pub');
//define('FTP_PRIKEY', '/home/username/.ssh/id_rsa');
//define('FTP_USER', 'username');
//define('FTP_PASS', 'password');
//define('FTP_HOST', 'ftp.example.org:21');
 
/**
 * Block requests through the proxy.
 *
 * Those who are behind a proxy and want to prevent access to certain hosts may do so. This will
 * prevent plugins from working and core functionality, if you don't include api.wordpress.org.
 *
 * You block external URL requests by defining WP_HTTP_BLOCK_EXTERNAL in your wp-config.php file
 * and this will only allow localhost and your blog to make requests.
 * The constant WP_ACCESSIBLE_HOSTS will allow additional hosts to go through for requests. The format of the
 * WP_ACCESSIBLE_HOSTS constant is a comma separated list of hostnames to allow.
 *
 * @since 2.8.0
 * @link http://core.trac.wordpress.org/ticket/8927 Allow preventing external requests.
/** @since 2.9  */
//!defined('WP_HTTP_BLOCK_EXTERNAL') && define( 'WP_HTTP_BLOCK_EXTERNAL', false );
 
/*
 * The constant WP_ACCESSIBLE_HOSTS will allow additional hosts to go through for requests. The format of the
 * WP_ACCESSIBLE_HOSTS constant is a comma separated list of hostnames to allow.
 *
 * @since 2.8.0
 * @link http://core.trac.wordpress.org/ticket/8927 Allow preventing external requests.
 * $accessible_hosts = preg_split('|,\s*|', WP_ACCESSIBLE_HOSTS);
 * return !in_array( $check['host'], $accessible_hosts ); //Inverse logic, If its in the array, then we can't access it.
 */
//!defined('WP_ACCESSIBLE_HOSTS') && define( 'WP_ACCESSIBLE_HOSTS', 'askapache.com,askapache.org' );

Cookies!

There’s always a little comfort in having non-default cookies for security (against auto-bots), and using shorter names also means smaller HTTP Packets.

The $cookie_hash is my hack to get around the fact that COOKIEHASH isn’t definable in wp-config.

/**#@+
 * COOKIES
 * Used to guarantee unique hash cookies @since 1.5 */
$cookie_hash=md5(WP_SITEURL);
 
/** Set a cookie now to see if they are supported by the browser.
 * setcookie(TEST_COOKIE, 'WP Cookie check', 0, COOKIEPATH, COOKIE_DOMAIN);
 * @since 2.3.0 */
!defined('TEST_COOKIE') && define('TEST_COOKIE', 'wp_tc');
 
/* @since 2.6.0 */
!defined('LOGGED_IN_COOKIE') && define('LOGGED_IN_COOKIE', 'wp_li_' . $cookie_hash);
 
/* @since 2.6.0 */
!defined('SECURE_AUTH_COOKIE') && define('SECURE_AUTH_COOKIE', 'wp_sa_' . $cookie_hash);
 
/* @since 2.5.0 */
!defined('AUTH_COOKIE') && define('AUTH_COOKIE', 'wp_a_' . $cookie_hash);
 
/* @since 2.0.0 */
!defined('PASS_COOKIE') && define('PASS_COOKIE', 'wp_p_' . $cookie_hash);
 
/* @since 2.0.0 */
!defined('USER_COOKIE') && define('USER_COOKIE', 'wp_u_' . $cookie_hash);
 
/* ok unset this var, its not needed as COOKIEHASH will have this value, but is not definable in wp-config.php */
unset($cookie_hash);
 
/** @since 1.2.0 */
!defined('COOKIEPATH') && define('COOKIEPATH', preg_replace('|https?://[^/]+|i', '', WP_HOME . '/' ) );
 
/** @since 1.5.0 */
!defined('SITECOOKIEPATH') && define('SITECOOKIEPATH', preg_replace('|https?://[^/]+|i', '', WP_SITEURL . '/' ) );
 
/** @since 2.6.0 */
!defined('ADMIN_COOKIE_PATH') && define( 'ADMIN_COOKIE_PATH', SITECOOKIEPATH . 'wp-admin' );
 
/** @since 2.6.0 */
!defined('PLUGINS_COOKIE_PATH') && define( 'PLUGINS_COOKIE_PATH', preg_replace('|https?://[^/]+|i', '', WP_PLUGIN_URL)  );
 
/** @since 2.0.0 */
!defined('COOKIE_DOMAIN') && define('COOKIE_DOMAIN', $_SERVER['SERVER_NAME']);

/**
  * The WP_CACHE setting, if true, includes the wp-content/advanced-cache.php script, when executing wp-settings.php.
  * For an advanced caching plugin to use, static because you would only want one
  * if ( defined('WP_CACHE') )@include WP_CONTENT_DIR . '/advanced-cache.php';
  */
!defined('WP_CACHE') && define('WP_CACHE', true);
 
/** WordPress Localized Language, defaults to en_US.
 *
 * Change this to localize WordPress.  A corresponding MO file for the chosen
 * language must be installed to wp-content/languages. For example, install
 * de.mo to wp-content/languages and set WPLANG to 'de' to enable German
 * language support. */
!defined('WPLANG') && define ('WPLANG', 'en_US');
 
/** Stores the location of the language directory. First looks for language folder in WP_CONTENT_DIR
 *   and uses that folder if it exists. Or it uses the "languages" folder in WPINC. @since 2.1.0 */
//!defined('WP_LANG_DIR') && define('WP_LANG_DIR', ABSPATH . WPINC . '/languages');
 
/** LANGDIR defines what directory the WPLANG .mo file resides. If LANGDIR is not defined WordPress looks first to wp-content/languages and then wp-includes/languages for the .mo defined by WPLANG file.  Old static relative path maintained for limited backwards compatibility - won't work in some cases*/
//!defined('LANGDIR') && define('LANGDIR', 'wp-content/languages');
 
/** Stores the location of the WordPress directory of functions, classes, and core content. @since 1.0.0 */
//!defined('WPINC') && define('WPINC', 'wp-includes');

WPMU Stuff

I personally don’t use.

/** Allows for the mu-plugins directory to be moved from the default location. @since 2.8.0 */
//!defined('WPMU_PLUGIN_DIR') && define( 'WPMU_PLUGIN_DIR', WP_CONTENT_DIR . '/mu-plugins' ); // full path, no trailing slash
 
/** Allows for the mu-plugins directory to be moved from the default location. @since 2.8.0 */
//!defined('WPMU_PLUGIN_URL') && define( 'WPMU_PLUGIN_URL', WP_CONTENT_URL . '/mu-plugins' ); // full url, no trailing slash
 
/** Allows for the mu-plugins directory to be moved from the default location. @since 2.8.0 */
//!defined( 'MUPLUGINDIR' ) && define( 'MUPLUGINDIR', 'wp-content/mu-plugins' ); // Relative to ABSPATH.  For back compat.

WordPress Database

This is usually the only thing I have to manually edit when creating a new site, unless I just use the same DB and modify the $table_prefix, (farther down). I run everything I possibly can in UTF-8, but if you don’t already know alot about character sets, wow it is one of the most confusing things so you may want to save learning about that topic for another day. Otherwise the following are helpful (and show how confusing character sets are!)

• Character Sets and Collations MySQL Support
• Converting Database Character Sets
• UTF-8 character sets (UTF-8)

If you ever setup WP to use the builtin membership features, make sure you learn about the CUSTOM_USER_TABLE and CUSTOM_USER_META_TABLE constants, I’ve found them very helpful.

/**#@+
 * MySQL settings
 */
/** The name of the database for WordPress */
define('DB_NAME', 'askapachewpblog75');
 
/** The username to access the database */
define('DB_USER', 'askapache245d');
 
/** The password for the username to access the database */
define('DB_PASSWORD', 'asdfklj2340');
 
/** The hostname to connect to the database at */
define('DB_HOST', 'mysql.askapache.com');
 
/** The charset of the database */
define('DB_CHARSET', 'utf8');
 
/** The collation of the database */
define('DB_COLLATE', 'utf8_general_ci');

$table_prefix

The $table_prefix is the value placed in the front of your database tables. Change the value if you want to use something other than wp_ for your database prefix. Typically this is changed if you are installing multiple WordPress blogs in the same database, and also for enhanced security.

Its a safe and good idea to change this value pre-installation to add more security to your WordPress blog. Exploits attempted against your WordPress blog by malicious crackers often are built with the premise that your blog uses the prefix wp_, by changing the value you mitigate some attack vectors.

/**
 * WordPress Database Table prefix.
 *
 * You can have multiple installations in one database if you give each a unique
 * prefix. Only numbers, letters, and underscores please!
 */
$table_prefix  = 'ar15_';
 
/** CUSTOM_USER_TABLE and CUSTOM_USER_META_TABLE are used to designated that the user and usermeta tables normally utilized by WordPress are not used, instead these values/tables are used to store your user information. */
//!defined('CUSTOM_USER_TABLE') && define('CUSTOM_USER_TABLE', $table_prefix . 'my_users');
//!defined('CUSTOM_USER_META_TABLE') && define('CUSTOM_USER_META_TABLE', $table_prefix . 'my_usermeta');

Setup PHP Ini Settings

 
/** Turns the output of errors on or off, you really never want this on, you should only view errors by reading the log file. */
ini_set('display_errors', WP_DEBUG_DISPLAY);
 
/** Tells whether script error messages should be logged to the server's error log or error_log. */
ini_set('log_errors', 'On');
 
/** http://us.php.net/manual/en/timezones.php */
ini_set('date.timezone', 'America/Indianapolis');
 
/** Where to log php errors */
ini_set('error_log', ASKAPACHE_ROOT . '/logs/php_error.log');
 
/** Set the memory limit, otherwise defaults to '32M' */
ini_set('memory_limit', WP_MEMORY_LIMIT);

Sessions are slow

So I only use sessions when I have a specific use… In this case I need sessions only when one of the tools in the /online-tools/ directory is being used. And that is for the captcha image. In the future I won’t ever use sessions.

if(preg_match( '#^/online-tools/#',$_SERVER['REQUEST_URI'])) session_start();

Include Custom Files

Sure you could use the my-hacks.php that WP allows, or you can just stick your functions in your TEMPLATEPATH/functions.php file, but they are executed only after the wp-settings.php file, which may be too late for your file.

In the past I’ve also used the auto_prepend_file settings to run my script before anything (index.php) but I ran into some issues on different hosts, and it wasn’t as portable.

This is useful because you can have a file with globally available functions that you can use in non-WP areas as well as WP areas. I am moving away from this more and more as I learn more about classes and build plugins instead for portability.

include_once ASKAPACHE_ROOT . '/includes/myfunctions.inc';
 
/** Sets up WordPress vars and included files. */
require_once(ABSPATH . 'wp-settings.php');
?>

Some Useful PHP

I am constantly trying to make my sites and code more portable, so I am using plugins alot more to accomplish things that I use to do with separate php. Here are some examples of minimal php.

add_filter("the_generator", create_function('$a','return "";'));
add_filter('the_content', create_function('$a', 'return ((is_feed())? $a."<p><a href=\"".get_permalink()."\">".get_the_title()."</a> originally appeared on ".get_bloginfo("name").".</p>" : $a);'), 99999);
add_filter('excerpt_length', create_function('$a', 'return 300;'),99);
add_filter('excerpt_more', create_function('$a', 'return "&hellip;";'),99);
add_action( 'wp_head', create_function('$a','echo "<link rel=\"pingback\" href=\"'.get_bloginfo('pingback_url').'\" />\n";'), 95 );
add_action( 'wp_head', create_function('$a','echo "<link rel=\"schema.rss\" href=\"http://purl.org/rss/1.0/\" />\n";'), 96 );
add_action( 'wp_head', create_function('$a','echo "<link rel=\"schema.rel\" href=\"http://purl.org/vocab/relationship/\" />\n";'), 97 );
add_action( 'wp_head', create_function('$a','echo "<link rel=\"meta\" type=\"application/rdf+xml\" href=\"/foaf.rdf\" />\n";'), 98 );
add_action( 'wp_head', create_function('$a','echo "<link href=\"/favicon.ico\" rel=\"shortcut icon\" type=\"image/x-icon\" />\n";'), 99 );

Debugging Note

If you read this far than you probably know how important debugging is, but I sometimes like to stick the best tips deep in my articles to make sure only YOU find it. GRTFM isn’t used on this site, it’s mostly a requirement because my writing can get pretty bad.. The point, debugging is more than a crucial requirement if you want to do anything cool. Don’t worry I got you.. check my AskApache Debug Viewer Plugin from the official WP site. It’s pretty close to providing as verbose amount of information that I could possibly figure out how to get out of php, probably more than you have ever seen at least, I focused on quantity. I use it all the time on new installs as there is no setup required and it tells me advanced information about the setup of the server, hacker code for sure.

Here’s a quick function to see set global vars, I just think this is interesting code.

function askapache_global_debug(){
  global $_GET,$_POST,$_COOKIE,$_SESSION,$_ENV,$_FILES,$_SERVER,$_REQUEST,$HTTP_POST_FILES,$HTTP_POST_VARS,$HTTP_SERVER_VARS,$HTTP_RAW_POST_DATA,$HTTP_GET_VARS,$HTTP_COOKIE_VARS,$HTTP_ENV_VARS;
  $gv=create_function('$n','global $$n; ob_start(); if ( is_array($$n) && sizeof($$n)>0 && print("[{$n}]\n") ) print_r($$n);return ob_get_clean();');
  foreach (array('_GET','_POST','_COOKIE','_SESSION','_ENV','_FILES','_SERVER','_REQUEST','HTTP_POST_FILES','HTTP_POST_VARS','HTTP_SERVER_VARS','HTTP_RAW_POST_DATA','HTTP_GET_VARS','HTTP_COOKIE_VARS','HTTP_ENV_VARS') as $k)echo $gv($k);
  print_r(get_defined_constants());
}

Also check the WordPress Codex page: Editing wp-config.php and Perishable Press’s: Stupid WordPress Tricks

Advanced WordPress wp-config.php Tweaks originally appeared on .

The Worst Kind of People

Spammers, and Leechers. They operate like this: Let’s say you have some mp3 files on a server, and SOMEWHERE on the web there is a link to that mp3 file’s location. This includes in javascript files, css files, robots.txt files, the spammers and leechers robots check all those files looking for the type of link they are looking for. Then they try to request that file usually utilizing a number of different types of requests to get access to the file. Then they use it for personal gain, at your peril.

Some robots perform valuable services for the world wide web community, and other leeching programming is pretty cool, so not all these activities are perpetrated by nefarious spammers.

Blocking by IP COOKIE

Ok so if a link exists to your file, it is going to be requested by a robot eventually, so the way to defeat them is by doing something on your site that modifies the way a user would request it. Robots for the most part are not javascript-capable, so the most-common advanced method is to set a cookie using javascript, and then we can check for that cookie in the request for the file using mod_rewrite.

So if your site sets a cookie named fspammers, and furthermore gives that cookie a value of 445, then this is what the request sent by an HTTP Client like Firefox looks like.

GET /hotlink/lovefreedom.mp3 HTTP/1.1
Host: static.askapache.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET CLR 3.5.30729)
Accept: image/png,image/*;q=0.8,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.askapache.com/wordpress/seo-in-wordpress.html
Cookie: fspammers=455

Mod_Rewrite HTTP Headers

The mod_rewrite module has access to ALL the HTTP Headers sent in a request, so for each of the HTTP Headers in the request example above, we can use mod_rewrite to validate.

Mod_Rewrite .htaccess Example

Finally, now that everyone is on the same page about what is really going on, here is the .htaccess code that blocks any requests for anything in the /hotlink/ folder.

Here are the triggers this code blocks access based on.

1. Cookie: Checks if fspammers cookie is present, and that it has the value of 445.
2. HTTP Protocol: Checks if HTTP 1.1 is being used (many robots use 1.0)
3. Host: Checks that the HOST Requested was static.askapache.com
4. Referer: Checks for Referring site is static.askapache.com or www.askapache.com

RewriteEngine On
RewriteBase /
 
RewriteCond %{HTTP_COOKIE} !^.*fspammers=445.*$ [NC,OR]
RewriteCond %{THE_REQUEST} !^[A-Z]{3,9}\ /(.*)\ HTTP/1\.1 [NC,OR]
RewriteCond %{HTTP_HOST} !^z\.askapache\.com$ [NC,OR]
RewriteCond %{HTTP_REFERER} !^http://(www|z)\.askapache\.com.*$ [NC]
RewriteRule ^hotlink/.*$ - [F]</p>

Protecting Files with Advanced Mod_Rewrite Anti-Hotlinking originally appeared on .