Lately I been using x86_64 Fedora Linux. There are requirements for me execute some 32 bits binaries as well as compiling to produce 32 bits binaries.

First of all, I do not aware of any great different for me based on user experience while migrating from 32 bits to 64 bits Fedora. Maybe this is because the fedora repository is in the good shape, where everything that I had downloaded by using yum, will works perfectly.

Until one day I downloaded some legacy projects from the 3rd party website, that piece of software comes with 32 bits libs which needs 32 bits compilation. I started to realized that the binaries that I had compiled in my 64 bits environment is not working at 32 bits OS even though I am using the legacy version of gcc 3.4 compiler.

Fedora x86_64 allows you to run 32 bits software given that the dynamic libs is installed properly. I realized that my fedora does have two directories for lib, /lib and another one is /lib64. When I am searching for a particular lib using yum, let say libxml2, I realized that yum is actually returns me both 32 bits and 64 bits result.

yum search libxml2
...
libxml2.i386 : Library providing XML and HTML support
libxml2.x86_64 : Library providing XML and HTML support
...

By default if I execute “yum install libxml2″, yum will install the 64 bits one for me. In my case I want the 32 bits glibc, what I have to do is to specify the exact package name.

yum install libxml2.i386

After the installation, now I can run the 32 bits application that requires libxml2.

How about compiling 32 bits application in 64 bits operating system?

First of all, I need to make sure I had installed all the development packages of the 32 bits libs required. Let takes libxml2 as the example, I need to yum ‘libxml2-devel.i386′.

Next tell gcc that I am going to compile the codes into 32 bit binaries by specify the -m32 CFLAGS. Assume I wrote a simple hello.c.

gcc -m32 -o hello{,.c} -lxml2

Now I need to produce the binary that can be run under Red Hat es4. I know the default version of gcc for RHES4 is 3.4 which I need gcc34 in my fedora. ( I need to yum another package ‘compat-gcc-34.x86_64′ for that) . With that now I can compile the codes at my x86_64 fedora to produce binary that can run at RHES4.

gcc34 -m32 -o hello{,.c} -lxml2

I had wrote a post shares the idea how we can mount a ftp host, the same way we can mount a ssh point using sshfs.

With sshfs, we can mount a remote directory through ssh, the command line works like this:

sshfs user-name@host-ip:remote-dir mount-point

Mount-point is simply a directory which we can trigger mkdir to create one. If we want to mount the user’s home directory at the remote server, we can leave the remote-dir as blank. For examples I want to connect to 192.168.1.1, I can do this: ( Assume I created a folder call ssh_mount )

sshfs surface@192.168.1.1: ssh_mount

As simple as that, we had done mounting the ssh server. But bear in mind, it is not the same like you ssh to the server. We are just mounting a remote filesystem, we are still working under the local system environment.

Below is the sample of INI configuration: config.ini

#
# GENERAL section contains generic configuration
#
[GENERAL]
IP=192.168.1.4
Port=2143

#
# LOG section contains logging settings
#
[LOG]
LogLevel=5
LogPath=/var/log
LogType=Info Trace Error Debug

I created few python function use to serialize the INI configuration into dictionary. But there are some assumption to make. Assume that the comment of INI is start with char ‘#’. python dictionary do not takes in comments, because comment is not configuration. But what if after I change the configurations in dictionary and want to serialize back to INI file with the original comments stayed in as well? Yes you can remain the comments with the set of python functions I wrote.

Lets look at what function I have.
grep (str_list,pattern,neg=False): is my internal function to filter element in string list (first param).

ParseConfig (name=None): put the INI config filename as param, dictionary object will be returned.

ConfigTemplate (name=None): The construct the INI structure in another dictionary, storing the Section Name as well as your comments.

ConfigWrite (template=[],config=[],filename=”"): This function is to serialize back the python dictionary back to INI file. First param takes in the template dictionary object, second takes in configuration dictionary object.

Lets us look at all functions definition as well as the examples: ParseConfig.py

#/usr/bin/env python
import sys,os,datetime,time,re,string
from StringIO import StringIO

GREP_NEGATIVE=True

def grep (str_list,pattern,neg=False):
    """grep() will return list of match case.
Param: str_list - must be a list of string, each element in list is a new line
                  can use .split('\n') to obtain list from mutiple line of string
Param: pattern - re pattern can be composed by re.compile()"""
    sres=[] #empty list
    for s in str_list:
        r=re.search(pattern,s,re.VERBOSE)
        if r and neg==False:
            sres.append(s)
        elif (not r) and neg==True:
            sres.append(s)

    return sres

def ParseConfig (name=None):
    global GREP_NEGATIVE
    """ Parsing config file to this function, will serialized to dictionary
@name  = filename
@return = dictionary"""
    configdict={}
    loglist=[]

    while True:
        try:
            f = open(name,"r")
        except IOError:
            print "Err ",name,"not exist"
            return None
            pass
        else:
            break

    try:
        for line in f:
            #strip all the whitespaces and write all lines to list.
            loglist.append(line.strip(string.whitespace))
    finally:
        f.close()

    # filter out the comment
    loglist=grep(loglist,"^\#",GREP_NEGATIVE)

    # make a dictionary
    xL=[] #current L
    itemdict={}
    for item in loglist:
        L=grep([item],"^\[.*\]$") # section
        if L:
            xL=[L[0].lstrip("[").rstrip("]")]
            itemdict={}
            #print xL
        else: # items?
            if len(xL)>0:
               if item:
                    #split index and values.
                    iL=item.split("=")
                    #replace all '\t' to " " in values
                    iL[1]=string.replace(iL[1],"\t"," ")
                    # split the values into list
                    iiL=iL[1].strip(string.whitespace).split(" ")
                    real_list=[]
                    for item in iiL:
                        # read each value in the list, if got something append to the real_list
                        if len(item.strip(string.whitespace)) is not 0:
                            real_list.append(item.strip(string.whitespace))
                    # create another dict into the section
                    itemdict[iL[0].strip(string.whitespace)]=real_list
                    configdict[xL[0]]=itemdict
    return configdict
pass

def ConfigTemplate (name=None):
    try:
        f=open(name,"r")
    except IOError:
        return None

    loglist=[]
    try:
        for line in f:
            #strip all the whitespaces and write all lines to list.
            theline = line.strip(string.whitespace)
            loglist.append(theline)
    finally:
        f.close()

    loglist=grep(loglist,"^\#|^\[")
    if len(loglist) is 0:
        loglist = None
    return loglist

def ConfigWrite (template=[],config=[],filename=""):
    global DebugDupConfigWrite

    if len(template) is 0 or len(config) is 0 or filename is "":
        return False

    try:
        f = open(filename,"w")
        for line in template:
            r=re.search("^\#",line,re.VERBOSE)
            if r:
                f.write(line + "\n")
            j=re.search("^\[",line,re.VERBOSE)
            if j:
                f.write(line + "\n")
                # get all section line up
                section_name = line.strip(string.whitespace).lstrip("[").rstrip("]")
                if config.has_key(section_name):
                    sec_dict = config[section_name]
                    #
                    element_list=[]
                    for element in sec_dict:
                        element_list.append(element)
                    element_list.sort()
                    for element in element_list:
                        # index = values....
                        #print element, sec_dict[element]
                        f.write(element + " = " + "\t".join(sec_dict[element]) + "\n")
                    f.write("\n")
                else:
                    pass
                    #f.write("CAN'T FIND THE ELEMENTS in section[%s]! DATA CORRUPTED.\n" % section_name)
                pass # end if j

            pass # end for line
        f.close()
        return True

    except:
        return False

# sample calling
myDict = ParseConfig ("config.ini")
template = ConfigTemplate ("config.ini")
ConfigWrite(template,myDict,"test.ini")

print "COMPLETE DICTIONARY"
print myDict

print "ALL TAGS"
for c in myDict:
    print " ",c

print ""
print "ALL ITEM IN 'GENERAL'"
for d in myDict["GENERAL"]:
    print " ",d

print ""
print "ALL ITEM IN 'LOG'"
for d in myDict["LOG"]:
    print " ",d

print ""
print "DIRECT ACCESS 'LOG's LOGTYPE'"
print myDict["LOG"]["LogType"]

print "SO, YOU CAN SEE THAT IS THE LIST, IT ITERATE the LIST"
for e in myDict["LOG"]["LogType"]:
    print " ",e

print "DIRECT ACCESS 'LOG's LOGTYPE 2nd value'"
print "second value is ",myDict["LOG"]["LogType"][1]

I hope my example is straight forward for you to understand how to use it, wish you enjoy python scripting :)

SCons is a software construction tool (build tool, or make tool) implemented in Python, that uses Python scripts as “configuration files” for software builds. Based on the design that won the Software Carpentry build tool competition, SCons solves a number of problems associated with other build tools, especially including the classic and ubiquitous Make itself.

In its evolution, SCons will be more general than a make replacement, it will be a Gnu Build System replacement. The Gnu Build System (GBS) is also known as the set of Autotools (autoconf, automake, autoheader, etc…).

Quote From http://www.scons.org/wiki/TheBigPicture

So scons is just a makefile replacement? Which claims to be simple and feature rich? Let me construct a simple but sufficient scons configuration file (SConstruct), and show you how you can compile your c++ projects. Then you make the judgement whether to go for it or not.

#filename: SConstruct
env = Environment(
CXX = 'g++34',
CCFLAGS = ['-g','-O3','-Wall','-DPOSIX_C_SOURCE=199506L','-D_REENTRANT','-DLINT_ARGS','-DLINUXOS'],
LIBS = ['pthread', 'c','nsl','rt' ,'lbestd'],
LIBPATH = ['lib'],
CPPPATH = ['.','lib/inc'],
LINKFLAGS = ['-mt']
)

env.Program(
'bin/msgQ',
['Main.cc', 'MainState.cc','MainFunc.cc',
'MyLog.cc','MyQueue.cc','MyTcp.cc',
'TcpSvr.cc']
)

To compile the source codes, simply do this:

scons

To clear the objects and binaries, simply do this:

scons -c

Let me briefly explain the items in the SConstruct file.

By default, scons does define a simple compilation environment for you, if you write helloworld.cc, you just need to write a line in SConstruct.

Program('helloworld.cc')

But the case above, I overwrite some environment variable’s value for my needs to build a complex application.

CXX is the variable defines you c++ compiler. My case, I change it to g++34.

You can check the default value by just insert a python print statement in your SConstruct file.

env = Environment()
print "CXX = ", env['CXX']

CCFLAGS defines compilation flags.
LIBS defines library to link.
LIBPATH defines additional library paths besides the system default.
CPPPATH defines include file paths.
LINKFLAGS defines compiler linker flags

After constructed the environment, just compile your binary by calling Program function.

scons have more features compiles java codes too and importantly it is cross platform. To know more? Check out the scons man pages ans well as user manual.

python -m SimpleHTTPServer 9914

9914 is the port number I choose to host my web, assume my IP is 192.168.1.1, in firefox you can ask your friend to access this url: http://192.168.1.1:9914.

It is easy isn’t it? You do NOT even need to run it as root, just run it as normal user but make sure your firewall allow the chosen port number.

A complex combo, I have constructed one last time in post what is your 10 common linux commands?

history | awk '{CMD[$2]++;count++;}END { for (a in CMD)print CMD[a] " " CMD[a]/count*100 "% " a;}' |  \
grep -v "./" | \
column -c3 -s " " -t | \
sort -nr | nl |  head -n10

But sometimes we tend to misuse them, I found someone likes to do this:

cat foobar.txt | grep "barfoo"

Actually grep alone is enough to do the search, we do not need cat or meow for that. As I show him this:

grep "barfoo" foobar.txt

Another example is list the names of all files under a given directory that match a particular string. Let say I wanna obtain a list of files that have string “foobar”.

find /opt/src/kmess -type f | xargs grep -l "foobar"

Hey! find is cool but grep alone is enough to do the job.

grep -lr "foobar" /opt/src/kmess

Try to goes through the grep manual, you will find out that grep is more powerful than what you expect.

for i in range(0,10):
    print "0x0001 << %d = 0x%04X" % ( i , 0x1 << i)

The result is

0x0001 << 0 = 0x0001
0x0001 << 1 = 0x0002
0x0001 << 2 = 0x0004
0x0001 << 3 = 0x0008
0x0001 << 4 = 0x0010
0x0001 << 5 = 0x0020
0x0001 << 6 = 0x0040
0x0001 << 7 = 0x0080
0x0001 << 8 = 0x0100
0x0001 << 9 = 0x0200

The important part in the example is this :

0x1 << i

Let say I wanna shift the bits of a value rightwards 3 times, I will do this:

value = 102
print value >> 3

The result will be 12.

To know more about what is bit shifting, check out wikipedia.

Have Fun!

Well, how to do it?

The idea is simple, the first instance of the python script will open a file, and lock it. Therefore when the consequent instances try to lock the file, it will fails.

The code is simple too.

import fcntl
def lockFile(lockfile):
    fp = open(lockfile, 'w')
    try:
        fcntl.lockf(fp, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except IOError:
        return False

    return True

The function will try to lock the file specified by me, if it success, return True, else return False.

From any point of my codes, I’ll do

if not lockFile(".lock.pod"):
        sys.exit(0)

Interesting?

For example, I have a module that extract raw values into logs based on the specified interval. The result will looks like this (test.dat):

##date time slot_1  slot_2  slot_3  slot_4  slot_5  slot_6  slot_7  slot_8  slot_9  slot_10
06/02/2009 09:16:49 16  6  15  5  14  4  17  3  14  4
06/02/2009 09:16:54 16  8  13  3  15  4  17  4  15  5
06/02/2009 09:16:59 18  8  11  5  15  6  14  3  16  5
06/02/2009 09:17:04 19  6  12  5  18  7  15  5  18  5
06/02/2009 09:17:09 19  7  14  6  16  6  16  6  18  4
06/02/2009 09:17:14 21  7  17  5  19  7  16  4  20  5
06/02/2009 09:17:19 20  5  19  6  17  5  15  3  17  3
06/02/2009 09:17:24 20  6  21  5  18  4  14  4  18  4
06/02/2009 09:17:29 17  5  19  4  16  4  15  4  17  2

gnuplot allows us to create a template describe how our graph outlook. To generate the graph, we just have to execute the template, just like executing bash script.

This is a sample of a gnuplot template that I use to construct a plot graph, (test1.pg)

#!/usr/bin/gnuplot
reset
set terminal png

set xdata time
set timefmt "%d/%m/%Y %H:%M:%S"
set format x "%H:%M"
set xlabel "time"

set ylabel "total actives"
set yrange [0:31]

set title "M7YC Performance per time"
set key reverse Left outside
set grid

set style data linespoints

plot "test.dat" using 1:3 title "slot 1", \
"" using 1:4 title "slot 2", \
"" using 1:5 title "slot 3", \
"" using 1:6 title "slot 4", \
"" using 1:7 title "slot 5", \
"" using 1:8 title "slot 6", \
"" using 1:9 title "slot 7", \
"" using 1:10 title "slot 8", \
"" using 1:11 title "slot 9", \
"" using 1:12 title "slot 10"
#

To allow the template to be executable, the first line of the template must add the SHABANG ‘#!/usr/bin/gnuplot’, similar to bash script. And also remember to perform chmod to your template.

chmod +x test1.pg

I would like to generate graph that shows the performance value over time, therefore I have to read datetime into my X axis and performance value into my Y axis. First two columns of my data file are datetime, I need to set the xdata as time, the define the time format in the raw data files and at last define the x format to appear in my graph.

set xdata time
set timefmt "%d/%m/%Y %H:%M:%S"
set format x "%H:%M"

I would like to display my keys outside the graphs as well as enable the grid to ease the reading.

set key reverse Left outside
set grid

The plot graph I intended to generate consist of lines and dots.

set style data linespoints

The important part is how to define the source of my raw data, how many items I wanted to plot into the graph, as well as giving each item a title. The plot format may vary based on different graph format, but for my case it is

plot [raw_data] using [x value's column in data file]:[y value's column in data files] title [item's name], ...

The plot portion can be shorten into this:

plot "test.dat" u 1:3 t "slot 1", \
"" u 1:4 t "slot 2", \
"" u 1:5 t "slot 3", \
"" u 1:6 t "slot 4", \
"" u 1:7 t "slot 5", \
"" u 1:8 t "slot 6", \
"" u 1:9 t "slot 7", \
"" u 1:10 t "slot 8", \
"" u 1:11 t "slot 9", \
"" u 1:12 t "slot 10"

Execute the template and redirect the binary stream to construct a png file.

./test1.pg > test1.png

test1.png

gnuplot is far more powerful than what I have illustrate here, you may want to checkout the official website for more info.

P.S. I am using gnuplot v4.2 while writing this.

str = "Hello, "
...
str = str + "my name is "
...
str = str + Name
print str

In python, we have string stream (StringIO) that will behave like file stream, you can construct your string like this:

str=StringIO()
...
str.write("Hello, ")
...
str.write("my name is ")
...
str.write(Name)
print str.getvalue()

The same way, you can construct your binary bytes with StringIO and write it into file once you are done.

bin=StringIO()
bin.write("/x5F/x5F%c" % 0xFF)
...
file = open ("my.bin","wb")
file.write(bin.getvalue())
file.close()