David Latham

Configuring MySQL on Redhat Linux while utilising network (iSCSI) storage.

2011-08-31T22:12:00.006+12:00

Challenges:
MySQL only allows a single location to be specified as the default data store for all databases.

Consider the following typical scenario:

A database server running one instance of MySQL has its datastore located at /mysql/databases.
This server will contain multiple databases for multiple applications.
The database tables are a mixture of InnoDB and MYISAM storage engines.
The storage is provided by a SAN and separate storage volumes (iSCSI targets) are provisioned for each database.

In this scenario, you would want all your MYISAM files (*.MYI, *.MYD, *.frm) and all your InnoDB table spaces to be contained on the same iSCSI volume on a per-database basis.

Solution:

This is easily achieved by mounting the iSCSI target on a sub-folder of /mysql/databases. For example:

Mount iqn.openfiler.com:2006-06:blog-db-storage on /mysql/databases/blog

Here is a sequence of steps that need to be carried out to make this work.

in /etc/my.cnf set datastore=/mysql/databases
in /etc/my.cnf set innodb_file_per_table (http://dev.mysql.com/doc/refman/5.0/en/innodb-multiple-tablespaces.html)
Configure SELINUX and user/group rights appropriately for MySQL. There is heaps on this on the internet but please comment if you would like me to add some detail here.
Start the mysqld service
configure the mysql root password if necessary
login to mysql as root
create the blog database (create database blog;) This creates the blog directory in /mysql/databases/blog At this point there is nothing inside it.
log out of mysql.
Mount your iscsi storage volume on /mysql/databases/blog
Configure your system so that the iscsi storage volume has a persistence and is always mounted on this location. Redhat have a great tutorial on how to do this. Basically it involves checking the WID of your iscsi target and then applying that to a udev rule that creates a consistently named block device under /dev
Now go ahead and create your tables. All the MYISAM and InnoDB tables will be created inside the /mysql/databases/blog directory.

Code / Scripts:

I am happy to provide scripts or code if people want. The following links should get you started.

Links:

MUST FOLLOW PROCESS...

2011-08-29T21:18:00.007+12:00

MUST FOLLOW PROCESS.... MUST FOLLOW PROCESS...

Starting with the premise that we need our customers to be happy with us, what do we need to do as an organisation to keep them that way?

We are working through steps to improve our customer satisfaction at work by carefully considering how our daily work routines, processes and actions impact our customers.

Consider this fairly typical example of how an organisation might respond to a customer request:

[NOTE: This is a little bit (A LOT) satirical and not meant to in anyway be a real representation of an actual process. It's designed to highlight how strict adherence to process can lead to unhappy customers.]

A customer needs a new type of widget. Your organisation sells gadgets that are very similar to widgets so they approach you to design and build one for them. They are willing to pay for the service but they need this new widget to work for them.

Luckily your organisation has a process to handle this type of request. It's called Customer Request Acceptance Process. It is a sequence of actions scraped out of a managerial text book written in the early 1900's.

So the sales rep in your organisation finds the appropriate template and begins...

The sales rep spends 2 hours working out how best to express his customer's requirements on the template. The template is designed to be all encompassing providing options for everything. Clearly (from looking at the template) anything not covered by it's fields are not valid for submission. [Templates have their place but be careful of over detailing them - they get confusing. A good place to start is a subject and details box. Leave the rest to the technical guys to nut out.]
The working group that handle inbound requests receive the completed form and instantly find gaps in information. They make assumptions about some of the more obvious omissions and request clarification from the sales rep on the others. [Leave the mind reading to the professionals]
The sales rep is very confident that he understands 90% of the questions and answers them straight away and on the other 10% he seeks clarification. [More mind reading...]
Finally the requirements are gathered (from the sales rep and via a series of mini assumptions about unimportant details)
A quote is hastily written up and the sales rep informed. The sales rep sells the solution to the customer. [the solution has to be sold??? I thought the customer asked for it in the first place. Be careful! If you have to **sell** the solution to a customer request then it's probably not what they wanted.]
The tech team receive the go-ahead and begin work on the widget. The widget is FANTASTIC. It's so damn clever, some individuals are already talking about it to their contacts in an effort to drum up even more revenue. [Woooah there Silver, Does it work for the customer? Has anyone asked them?]
The widget is delivered more or less on schedule. [Good work guys - go eat pizza and have a celebration.]
The customer finds that it sort-of does what it's supposed to do but they don't complain because they have come to expect this sort of thing from the organisation.[What??? This is acceptable to you Mr. Customer? Find another provider already!]
Your organisation sends the customer a survey asking how happy they are and soon it becomes apparent that they are not. [By sending a survey out you send a message saying that you need to know what's wrong before you can fix it. If you don't fix it your customer will assume your are being cynical about the whole survey thing and not bother answering it.]

My point with this extremely obvious example is that:

a happy customer is very simply a customer that you talk to all the time!

So while your organisation might have a process and tools to manage that process, it's useless if you don't know what your customer really wants.

Some customers are not too good at telling us what they really want.
Most of the time, I suggest that we are not placing the appropriate individuals in our organisations together. IE: Customer end user of WIDGET next to the provider technical person designing the WIDGET. It makes sense that these two should work closely together.

Providers should develop fast prototypes and be less precious about the their solutions in case they end up not being fit for purpose. Every time a mistake is corrected, your product is improved.

It is impossible to gather all the requirements in one go. Requirements have a way of evolving as the design takes shape.

No amount of technology based tools to manage your processes will help you if your processes do not include the customer at every step.

Every step in your process that does not include the customer results in your product diverging further from the customer's requirements.

We should all evaluate all the time how what we do impacts the customer. If you don't know how your job impacts your customers, you should make it your number one concern to find out. Customers pay your salary. Not your boss or your manager.

REBLOG!!! Sticky bit, suid and guid

2011-07-21T22:59:00.002+12:00

I have to reblog this article in it's full just in case the original author removes it.

Here is the original.
http://bashscript.blogspot.com/2010/03/unixlinux-advanced-file-permissions.html

Here in plain text to ** FORCE ** it into my rubbish layout.

UNIX/Linux Advanced File Permissions - SUID,SGID and Sticky Bit

After you have worked for a while with Linux you discover probably
that there is much more to file permissions than just the "rwx" bits.
When you look around in your file system you will see "s" and "t"

$ ls -ld /tmp
drwxrwxrwt 29 root root 36864 Mar 21 19:49 /tmp

$ which passwd
/usr/bin/passwd

$ ls -l /usr/bin/passwd
-rwsr-xr-x 1 root root 22984 Jan 6 2007 /usr/bin/passwd

What is this "s" and "t" bit? The vector of permission bits is really
4 * 3 bits long. Yes there are 12 permission bits,not just 9.The first
three bits are special and are frequently zero. And you almost always
learn about the trailing 9 bits first.Some people stop there and never
learn those first three bits.

The forth permission bit is used only when a special mode of a file
needs to be set. It has the value 4 for SUID, 2 for SGID and 1 for the
sticky bit. The other 3 bits have their usual significance.

Here we will discuss about the 3 special attributes other than the
common read/write/execute:

1.Set-User-Id (SUID)
2.Set-Group-Id (SGID)
3.Sticky Bit

Set-User_Id (SUID): Power for a Moment:

By default, when a user executes a file, the process which results in
this execution has the same permissions as those of the user. In fact,
the process inherits his default group and user identification.

If you set the SUID attribute on an executable file, the process res-
ulting in its execution doesn't use the user's identification but the
user identification of the file owner.

The SUID mechanism,invented by Dennis Ritchie,is a potential security
hazard. It lets a user acquire hidden powers by running such a file
owned by root.

$ ls -l /etc/passwd /etc/shadow /usr/bin/passwd
-rw-r--r-- 1 root root 2232 Mar 15 00:26 /etc/passwd
-r-------- 1 root root 1447 Mar 19 19:01 /etc/shadow

The listing shows that passwd is readable by all, but shadow is unre-
adable by group and others. When a user running the program belongs to
one of these two categories (probably, others), so access fails in the
read test on shadow. suppose normal user wants to change his password,
How can he do that? He can do that by running /usr/bin/passwd. Many
UNIX/Linux programs have a special permission mode that lets users
update sensitive system files –like /etc/shadow --something they can't
do directly with an editor. This is true of the passwd program.

$ ls -l /usr/bin/passwd
-rwsr-xr-x 1 root root 22984 Jan 6 2007 /usr/bin/passwd

The s letter in the user category of the permission field represents a
special mode known as the set-user-id (SUID). This mode lets a process
have the privileges of the owner of the file during the instance of
the program. Thus when a non privileged user executes passwd, the eff-
ective UID of the process is not the user's, but of root's – the owner
of the program. This SUID privilege is then used by passwd to edit
/etc/shadow.

What is effective user-id:

Every process really has two user IDs: the effective user ID and the
real user ID. (Of course, there's also an effective group ID and real
group ID.Just about everything that's true about user IDs is also true
about group IDs) Most of the time,the kernel checks only the effective
user ID. For example, if a process tries to open a file, the kernel
checks the effective user ID when deciding whether to let the process
access the file.

Save the following script under the name reids.pl and make it
executable (chmod 755 reids.pl).

#!/usr/bin/perl
# print real UID
print "Real UID: $<\n";
# print real GID
print "Real GID: $(\n";
# print effective UID
print "Effective UID: $>\n";
# print effective GID
print "Effective GID: $)\n";

check file permissions:

$ ls -l reids.pl
-rwxr-xr-x 1 venu venu 203 Mar 24 10:40 reids.pl

Note: For security reasons the s-bit works only when used on binaries
(compiled code) and not on scripts (an exception are perl scripts).
Scripts,i.e. programs that cannot be executed by the kernel directory
but need an interpreter such as the Bourne shell or Java,can have
their setuid bit set, but it doesn't have any effect. There are some
platforms that honor the s bits even on scripts ( some System V vari-
ants, for example), but most systems don't because it has proven such
a security headache - most interpreters simply aren't written with
much security in mind. Set the SUID bit on shell script is useless,
that's why I am using perl script here.

When you run the script you will see that the process that runs it
gets your user-ID and your group-ID:

$ ./reids.pl
Real UID: 500
Real GID: 500 500
Effective UID: 500
Effective GID: 500 500

Note: If you get an error like this:
Can't do setuid (cannot exec sperl)

In Debian install perl-suid using following command:
apt-get install perl-suid

In Centos install perl-suidperl using following command:
yum install perl-suidperl

Now change owner ship to another user (Do it as an administrator).

# chown king /home/venu/reids.pl
# ls -l /home/venu/reids.pl
-rwxr-xr-x 1 king venu 203 Mar 24 10:40 /home/venu/reids.pl

Now run the script again.

$ ./reids.pl
Real UID: 500
Real GID: 500 500
Effective UID: 500
Effective GID: 500 500

What you observed, the output of the program depends only on the user
that runs it and not the one who owns the file.

How to assign SUID permission:

The SUID for any file can be set (mostly by the superuser) with a
special syntax of the chmod command. This syntax uses the character s
as the permission. Now add SUID permission to the script reids.pl :

# chmod u+s /home/venu/reids.pl (Do it from root account)

Now return from the super user mode to the usual non privileged mode.

$ ls -l reids.pl
-rwsr-xr-x 1 king venu 203 Mar 24 10:40 reids.pl

To assign SUID in an absolute manner, simply prefix 4 to whatever
octal string you would otherwise use (like 4755 instead of 755).

The file reids.pl is owned by king and has the s-bit set where norma-
lly the x is for the owner of the file. This causes the file to be
executed under the user-ID of the user that owns the file rather than
the user that executes the file. If venu runs the program then this
looks as follows:

$ perl reids.pl
Real UID: 500
Real GID: 500 500
Effective UID: 503
Effective GID: 500 500

Effective user id of process is 503, this is not the venu's , but of
king's - the owner of the program. As you can see this is a very powe-
rful feature especially if root owns the file with s-bit set. Any user
can then do things that normally only root can do.

Caution: When you write a SUID program then you must make sure that
it can only be used for the purpose that you intended it to be used.
As administrator, you must keep track of all SUID programs owned by
root that a user may try to create or copy. The find command easily
locate them:

# find /home -perm -4000 -print | mail root

The extra octal bit (4) signifies the SUID mode, but find treats the
"–" before 4000 as representing any other permissions.

Set-Group_Id (SGID):

The set-group-id (SGID) is similar to SUID except that a program with
SGID set allows the user to have the same power as the group which
owns the program. The SGID bit is 2,and some typical examples could be
chmod g+s reids.pl or chmod 2755 reids.pl.
You can remove SGID bit using following commands:

$ chmod g-s reids.pl
$ chmod 755 reids.pl (Absolute manner)

It is really useful in case you have a real multi-user setup where
users access each others files. As a single homeuser I haven't really
found a lot of use for SGID. But the basic concept is the same as the
SUID,Similar to SUID, SGID also grants privileges and access rights to
the process running the command, but instead of receiving those of the
file's owner it receives those of the file's group. In other words,the
process group owner will be set to the file's group.

I explain it with an example. I have created two user accounts king
and venu with same home directory project. king belongs to king and
development groups, venu belongs to venu and development groups.

# groups king venu
king : king development
venu : venu development

venu's default group is venu and king's default group is king.

Login as king and create reids.pl file again and make it executable
(using chmod 755 reids.pl) .

$ id
uid=503(king) gid=503(king) groups=501(development),503(king)
$ ls -l reids.pl
-rwxr-xr-x 1 king development 203 Mar 25 19:00 reids.pl

Now login as venu and run the program:

$ id
uid=501(venu) gid=504(venu) groups=501(development),504(venu)
$ perl reids.pl
Real UID: 501
Real GID: 504 504 501
Effective UID: 501
Effective GID: 504 504 501

The effective GID of the process is the venu's,but not of the king's
-the owner of the program.

Now login as king and assign SGID bit to reids.pl program:

$ chmod 2755 reids.pl; ls -l reids.pl
-rwxr-sr-x 1 king development 203 Mar 25 19:00 reids.pl

Now login as venu and run the reids.pl program:

$ perl reids.pl
Real UID: 501
Real GID: 504 504 501
Effective UID: 501
Effective GID: 501 504 501

Real GID and Effective GID are different,here Effective GID is the
king's - the owner of the program.

Set SGID on a directory:

When SGID is set on a directory it has a special meaning. Files crea-
ted in a directory with SGID set will inherit the same group ownership
as the directory itself,not the group of the user who created the file.
If the SGID is not set the file's group ownership corresponds to the
user's default group.

In order to set the SGID on a directory or to remove it, use the
following commands:

$ chmod g+s directory or $ chmod 2755 directory
$ chmod g-s directory or $ chmod 755 directory

As I mentioned earlier venu and king's home directory is same that is
/home/project. I changed group ownership of /home/project directory
to development.

# ls -ld /home/project/
drwxrwxr-x 16 root development 4096 Mar 26 00:22 /home/project/

Now login as king and create a temp file.

$ whoami
king
$ pwd
/home/project/
$ touch temp; ls -l temp
-rw-r--r-- 1 king king 0 Mar 26 12:34 temp

You can see from the ls output that the group owner for project is
development, and that the SGID bit has not been set on the directory
yet. When king creates a file in project, the group for the file is
king (king's primary gid).

Set SGID bit on project directory. For that login as administrator
and set SGID bit using following command:

# chmod g+s /home/project/
# ls -ld /home/project/
drwxrwsr-x 15 root development 4096 Mar 26 12:34 /home/project/

From the ls output above, you know the SGID bit is set because of the
s in the third position of the group permission set,which replaces the
x in the group permissions.

Now login as king and create temp2 file.

$ whoami
king
$ touch temp2; ls -l temp2
-rw-r--r-- 1 king development 0 Mar 26 13:49 temp2

Notice the group ownership for temp2 file. It inherits group permiss-
ion from the parent directory.

Enabling SGID on a directory is extremely useful when you have a
group of users with different primary groups working on the same set
of files.

For system security reasons it is not a good idea to set many
program's set user or group ID bits any more than necessary,since this
can allow an unauthorized user privileges in sensitive system areas.If
the program has a flaw that allows the user to break out of the inten-
ded use of the program, then the system can be compromised.

Sticky bit:

The sticky bit(also called the saved text bit) is the last permission
bit remaining to be discussed. It applies to both regular files and
directories. When applied to a regular file, it ensures that the text
image of a program with the bit set is permanently kept in the swap
area so that it can be reloaded quickly when the program's turn to use
the CPU arrives. Previously, it made sense to have this bit set for
programs like vi and emacs. Today,machines with ultra-fast disk drives
and lots of cheap memory don't need this bit for ordinary files and
that is also useless.

However, the sticky bit become a useful security feature when used
with a directory. The UNIX/Linux system allows users to create files
in /tmp, but none can delete files not owned by him. That's possible
because sticky bit set for /tmp directory.

The /tmp directory is typically world-writable and looks like this
in a listing:

# ls -ld /tmp
drwxrwxrwt 32 root root 36864 Mar 27 12:38 /tmp

Everyone can read,write and access the directory.The t indicates that
only the user (root and owner of the directory,of course) that created
a file in this directory can delete that file.

In order to set or to remove the sticky bit, use the following
commands:

$ chmod +t directory or $ chmod 1754 directory
$ chmod -t directory or $ chmod 754 directory

Note: 754 permissions for a directory are powerful enough to guard
your directories from intruders with malicious intentions, that's why
I used 754 as default,if yow want you can change it.

Example:

I logged in as king and created a temp file.

$ whoami
king
$ pwd
/home/project/
$ touch temp; ls -l
-rw-r--r-- 1 king king 0 Mar 27 13:44 temp

Now logged in as venu and try to delete temp file.

$ whoami
venu
$ rm temp
rm: remove write-protected regular empty file `temp'? Y
$ ls temp
ls: temp: No such file or directory

So what happened? venu deleted file owned by king.

Assign sticky bit to the project directory.As a owner of the directory
or administrator.

# chmod +t /home/project
# ls -ld /home/project/
drwxrwxr-t 15 root development 4096 Mar 27 13:46 /home/project/

From the ls output above, you know the sticky bit is set because of
the t in the third position of the other permission set,which replaces
the x in the other permissions.

Now repeat same steps again,then you get the following message:

$ whoami
venu
$ ls -l temp
-rw-r--r-- 1 king king 0 Mar 27 17:36 temp
$ rm temp
rm: remove write-protected regular empty file `temp'? y
rm: cannot remove `temp': Operation not permitted

Observation: Login as normal user and create a file.
[venu@localhost ~]$ touch sample
[venu@localhost ~]$ ls -l sample
-rw-rw-r-- 1 venu venu 0 Dec 21 03:41 sample

Now change permissions to 644

[venu@localhost ~]$ chmod 644 sample
[venu@localhost ~]$ ls -l sample
-rw-r--r-- 1 venu venu 0 Dec 21 03:41 sample

Now assign SUID permission.

[venu@localhost ~]$ chmod u+s sample
[venu@localhost ~]$ ls -l sample
-rwSr--r-- 1 venu venu 0 Dec 21 03:41 sample

After setting SUID, if you see 'S' then it means that the file has no
executable permissions for that user.

Now remove SUID permission and change permissions to 744. Then assign
SUID permission. You should see a smaller 's' in the executable permi-
ssion position.

[venu@localhost ~]$ chmod u-s sample
[venu@localhost ~]$ chmod 744 sample
[venu@localhost ~]$ chmod u+s sample
[venu@localhost ~]$ ls -l sample
-rwsr--r-- 1 venu venu 0 Dec 21 03:41 sample

Same is applicable for SGID and Stickybit.

Posted by venu k at 10:51 AM
Labels: Article

Switch Google Desktop Search to search desktop by default

2011-03-01T12:14:00.002+13:00

I recently installed the latest version of Google Desktop. I found, however, that when searching either from the sidebar searchbox or the Ctrl Ctrl popup box, GD would always select Search Web by default.

This can now be changed using a small registry edit on Windows XP it is:
HKEY_CURRENT_USER\Software\Google\Google Desktop\Deskbar\ change search_type to "Search Desktop"

That's it. Hopefully this will save some of you some time.

Some programming guidlines

2011-01-05T17:35:00.003+13:00

This is a very simple post. I found these principals very useful in solving a problem I have with a particular project I am currently working on...

http://en.wikipedia.org/wiki/Unix_philosophy

1.Small is beautiful.
2.Make each program do one thing well.
3.Build a prototype as soon as possible.
4.Choose portability over efficiency.
5.Store data in flat text files.
6.Use software leverage to your advantage.
7.Use shell scripts to increase leverage and portability.
8.Avoid captive user interfaces.
9.Make every program a filter.

Have a read. It's well worth remembering these guidelines. Even if you have read them before, it might be useful to refresh your memmory.

Python Basic HTTPS webservice

2010-12-15T03:18:00.001+13:00

The basic webservice module in my previous posting does not support https or ssl encryption.

Here is how I finally managed to work it out:


# Basic https web server

import socket, ssl

host = ''
port = 8080

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind((host, port))
sock.listen(1)

while 1:
    osock, caddr = sock.accept()
    csock = ssl.wrap_socket(osock,
                                server_side=True,
                                certfile="servercert.cer",
                                keyfile="serverkey.key",
                                ssl_version=ssl.PROTOCOL_SSLv23)
                                
    cfile = csock.makefile('rw', 0)

    # Protocol exchange - read request
    msg = "GO AWAY!"
    while 1:
        line = cfile.readline().strip()
        if line == "GET /echotest HTTP/1.1":
            msg = "ECHOTEST OK"
        if line == '':
            cfile.write("HTTP/1.0 200 OK\n\n")
            cfile.write("<head><title>Eh?</title></head>")
            cfile.write("<h1>%s</h1>" % msg)
            cfile.close()
            csock.close()
            osock.close()
            break

Compile Pana (Amarok 1.4 clone) on Fedora 14 x86_64

2010-11-08T17:36:00.005+13:00

After meeting all the various dependencies, I had to make one small actual code change to the source code. I would have liked to submit it on the Pana website but could not find a way to do that.

Anyway here is the patch.


[dave@fedora junk]$ diff [source tree]/pana/src/osd.h [source tree]/pana/src/osd.h.original
40c40
<         void show( const QString &text, QImage newImage = QImage() );
---
>         void show( const QString &text, QImage newImage = QImage::QImage() );

Python webservice that executes local commands

2010-10-14T21:04:00.007+13:00

There are a few different options when it comes to managing server-side scripting on a web site. Usually folks use php or perl and even python in many occasions.

This blog post is about using python to execute code locally on the server in response to http GET requests.

So far you are thinking so what? You are already crafting your comment and it is saying something like, "Google mod_python" or "Google mod_perl". You are right, the best way to do CGI is via mod_perl, mod_php or mod_perl. The problem is user access and chroot.

Apache will execute server side scripts as the user / group defined in the main httpd.conf. In my case: apache / apache.
Apache will also assume a document root of /var/www/ for scripts (on a Centos 5.5 box) even if the userdir module is in use.

My problem was: How to get apache to execute scripts as dave:dave on doc root = /home/dave/. It was critical to get this working because the scripts in question interact with the .gnupg/pubkeyring and .gnupg/seckeyring files under /home/dave/.gnupg/.

Basically, I was making some kind of web based PGP key server. A web based gui for remote users to manage keys.

In the end I settled for python and the BaseHTTPServer.

First of all a simple class that will accept a shell command, execute it and return the stdout.


import popen2

class MyShellCommand:
 
 """execute a command, capture stdout and return it."""
 def __callShellCmd(self, cmd):
  
  stdout, stdin = popen2.popen2(command)
  data = ""
  while True:
   c = stdout.read(1)
   if( c ):
    data += c
   else:
    break
  return data
 
 """concrete example"""
 def getPublicGPGKey(self, keyid):
  
  """TODO: Add logic to validate key id..."""
  command = "gpg -a --export '%s'" % keyid
  return self.__callShellCmd(command)

Now that we have a utility to retrieve public keys from the gpg keyring, lets call it from a webservice that is owned and operated by user:group, dave:dave.


import MyShellCommand
import time
import BaseHTTPServer, cgi

"""Configure host ip and port to listen on.  Use high port for non root users."""
HOST_NAME = '127.0.0.1'
PORT_NUMBER = 8080 

class MyHandler(BaseHTTPServer.BaseHTTPRequestHandler):

 def do_HEAD(s):
  s.send_response(200)
  s.send_header("Content-type", "text/html")
  s.end_headers()

 def do_GET(s):
  s.send_response(200)
  s.send_header("Content-type", "text/html")
  s.end_headers()

  """Get the path and find the parameters"""
  path, query_string = s.path.split('?', 1)
  params = dict(cgi.parse_qsl(query_string))
  
  """create a shell object"""
  shell = MyShellCommand()
  
  """Validate the call being made"""
  if path == '/publickey':
   """Validate the parameters"""
   if params.has_key('id'):
    s.wfile.write('%s' % shell.getPublicGPGKey(params['id']))
    
   """I dont like descriptive errors."""
   else: s.wfile.write('An error occurred.')
  else: s.wfile.write('An error occurred.')



if __name__ == '__main__':
 server_class = BaseHTTPServer.HTTPServer
 httpd = server_class((HOST_NAME, PORT_NUMBER), MyHandler)
 print time.asctime(), "Server Starts - %s:%s" % (HOST_NAME, PORT_NUMBER)
 try:
  httpd.serve_forever()
 except KeyboardInterrupt:
  pass
 httpd.server_close()
 print time.asctime(), "Server Stops - %s:%s" % (HOST_NAME, PORT_NUMBER)

If you execute the above script, you will have a working webservice that responds nicely to only one specific set of GET data. Call it with a URL like this:


http://127.0.0.1:8080/publickey?id=mykeyname

Use PHP to perform an LDAP Bind to Windows Active Directory for User Authentication

2010-10-01T23:18:00.003+13:00

This example shows some basic LDAP bind lookups on a Windows 2003 active directory server.

The sequence is:

Connect
Bind using privileged user
search for 'dn' of supplied credentials
re-bind using this dn and password of supplied credentials
unbind

The per-requisite is that php_ldap modules are loaded and compiled. These are fairly standard now days. In fedora it was just a matter of executing:


# yum install php-ldap

Anyway: Here is the example. It's farily well commented so should be a simple matter to make work in your own environment.


<?php
// example.php
//
// David Latham @ 2010
// david-latham.blogspot.com
//
// This code cannot be executed on the same server as AD is installed on!!!
//
// Active Directory has an Organizational Unit of "_Test_Users" with 3 users loaded
// into it.
// user1  | User One
// user2  | User Two
//    ldapbind  | ldap bind
//
// execute with php -q example.php
    
// credentials to test
$user='user2@example.local';
// user='user2'; //Use this one to test binding without an email address.
$pass='2wsx#EDC';

// AD server details
$ldap_server = "192.168.122.231";
$dn = "OU=_Test_Users,DC=example,DC=local";

// Connect
$ad = ldap_connect($ldap_server);  //Must be a real active directory server
     //(WIN2003 used in this example)

// Set some variables
ldap_set_option($ad, LDAP_OPT_PROTOCOL_VERSION, 3);
ldap_set_option($ad, LDAP_OPT_REFERRALS, 0);

// Bind to the ldap directory.
// Here we are binding using a fixed user who has privileges to bind to the
// active directory.  This user also has a fixed password.  It can not change
// otherwise this script will break.


$bd = @ldap_bind($ad,"ldap user","pa$$word")
 or die("Couldn't bind to AD!");

// Search the directory for a distinguished name (dn) using the supplied
// credentials.
//
// If $user is an email address then use 'mail' otherwise use 'sAMAccountName'

if( strpos($user, '@') > 0 ) {
 $field='mail';
} else {
 $field='sAMAccountName';
}
echo "Searching AD using ...".$field."\n";

$result = @ldap_search($ad, $dn, "(".$field."=".$user.")", array('dn'));

// Create result set
$entries = @ldap_get_entries($ad, $result);

// Show an array of result
print_r($entries);

// If we have a result then we should 're-bind' using the destinguished name found
// in the search.
if ( $entries[0] ) {
 if(@ldap_bind($ad, $entries[0]['dn'], $pass)) {
  // If bind was successful then show some data.  This is user
  // authentication success.
  echo "found: ".$entries[0]['dn'];
 } else {
  // Bind failed so this would equate to user authentication failure.
  echo "Failed to bind.";
 }
} else { 
 // Failure to search would indicate a problem with your $dn variable.
 echo "failed to search";
}

//never forget to unbind!
ldap_unbind($ad);

?>

the result might look something like this:


[dave@fedora php-ldap]$ php -q example.php 
Searching AD using ...mail
Array
(
    [count] => 1
    [0] => Array
        (
            [count] => 0
            [dn] => CN=user2,OU=_Test_Users,DC=example,DC=local
        )

)
found: CN=user2,OU=_Test_Users,DC=example,DC=local

Have fun authenticating your users against your work active directory. You will:

Please your security manager

Avoid the headache of managing passwords password compliance

Write safer applications in terms of locking them down

Feel good about yourself

Images from web in python gui

2010-06-16T21:06:00.002+12:00

I have an application that I use to monitor data produced by a webservice. This webservice produces a new image ( a .png graph to be precise ) every 15 seconds. Basically I have four cron jobs that execute every minute. Three of these are configured to sleep for 15s, 30s and 45s respectivley. So that's one way to make cron do stuff more often than once a minute.

Anyway, I have been using a web browser and some fancy ajax type functions to continuously display this graph and some other JSON data from the webserver every 15 seconds forever. We have a group of people who have the job of watching my graphs and data among a whole bunch of other stuff in case they might provide some alert to a problem.

Anyway, I have been wondering, of late, about the reliablility of using a web browser for monitoring. The web browser is not refreshing itself and so I am a little worried that it just kind of get's stuck at times. I think it looses the AJAXIAN plot so to speak. This is why I started making a python gui using the following python modules:

httplib
json
urrlib
wx
logging

I will only show three classes here for reference purposes.

There is also a threading module required. This bit of handy code is at the bottom and I must apologies to the author of this code as I failed to take a note of his name / URL and so can not reference the source.

This is the Web client class that I use to retrieve the http / JSON content and the image content from my webserver.


#!/usr/bin/env python

import httplib
import urllib

class HTTP_Client(httplib.HTTPConnection):
 
 def __init__(self, url):
  httplib.HTTPConnection.__init__(self, url)
  
 
 def getPage(self, page):
  self.request("GET", "/%s" % page)
  r = self.getresponse()
  data = r.read()
  if r.status == 200 and r.reason == "OK":
   return urllib.base64(data)
  else:
   return "E Fetching %s: %s %s" % (page, r.status, r.reason)

if __name__ == "__main__":
 hc = HttpClient("www.python.org")
 print hc.getPage("parrot.spam")
 print hc.getPage("index.html")

This is an example of how to display an image on a panel using the wx gui toolkit. I think this is a mostly coding by google but it's not a big leap to make it useful. I believe my real program uses the _Image.SetBitmap(_Bitmap) method to update the bitmap.


#!/usr/bin/env python

import urllib
import wx
import cStringIO

class Panel1(wx.Panel):
 """ class Panel1 creates a panel with an image on it, inherits wx.Panel"""
 def __init__(self, parent, id):
  wx.Panel.__init__(self, parent, id)
  try:
   url = "http://www.photoshopsupport.com/photoshop-blog/07/2008/photoshop-blog/istockphoto-free-photo-credits.jpg"
   data = urllib.urlopen(url).read()
   stream = cStringIO.StringIO(data)
   bmp = wx.BitmapFromImage( wx.ImageFromStream( stream ) )
   wx.StaticBitmap(self, -1, bmp, (5,5))
  except IOError:
   print "Image file %s is not found" % url
   raise SystemExit

app = wx.PySimpleApp()
frame1 = wx.Frame(None, -1, "An image on a panel", size=(400,300))
Panel1(frame1, -1)
frame1.Show(1)
app.MainLoop()

This is the logger object. It sets up a logger and a formatter as well as a few actual logging methods. These just expose the internal methods of the logging module provided in default python.


#!/usr/bin/env python
import logging
import logging.handlers

class CDLogger(object):

    def __init__(self):
        self.log = logging.getLogger('cdBurner')
        self.log.setLevel(logging.INFO)

        hand = logging.FileHandler("cdburner.log", 'a')
        form = logging.Formatter("%(asctime)s %(levelname)-10s %(message)s")

        hand.setFormatter(form)
        self.log.addHandler(hand)

    def info(self, message):
        self.log.info(message)

    def debug(self, message):
        self.log.debug(message)
        
    def warning(self, message):
        self.log.warning(message)

    def error(self, message):
        self.log.error(message)

Here is the threading module that is used to execute the timer.


   Code on my laptop so I need to get it from the laptop.  Will do this later.

Amarok Script to Pana Script converter

2010-04-21T13:30:00.004+12:00

I use Pana now in favour of Amarok 1.4. Pana is under active development and includes a fix for the cover fetcher that stopped working some time ago.

The next step was to find a way to port the amarok scripts over to Pana. This is easily acheived ( for replay_gain ) anyway with this script:

Source: http://air.elementfx.com/phpBB3/viewtopic.php?f=5&t=3#p3


#! /bin/bash
FILE=$1
FILENAME=`echo ${FILE##*/}`
mkdirerr=$(mktemp)
mkdir $FILENAME".tmp" 2> $mkdirerr
if [ -n "`cat $mkdirerr`" ]; then
  cat $mkdirerr
  exit
fi
cd $FILENAME".tmp"
echo "Extracting files to $FILENAME.tmp/ ..."
tar jxvf $FILE &> /dev/null

currentdir=`pwd`

for (( i=0 ; $i < `find  . -depth -type d | wc -l` ; i=$(( $i+1 )) ));
do
  directory[$i]=`find  . -depth -type d | head -$(( $i+1 )) | tail -1`
done


for (( i=0 ; $i < ${#directory[@]} ; i=$(( $i+1 )) )); do
  cd ${directory[$i]}
  for file in `find  . -depth -maxdepth 1 -type f ` ; do
      echo "Changing "$file
      NEWNAME=`echo $file | sed 's/amarok/pana/g'`
 
      sed -i 's/amarok/pana/g' $file
      sed -i 's/Amarok/Pana/g' $file
      
      if [ "$file" != "$NEWNAME" ]; then
 echo "Renaming file $file to $NEWNAME"
 mv $file $NEWNAME
      fi
  done

  cd $currentdir
  if [ -z "`echo ${directory[$i]%/*} | grep 'amarok'`" ] && [ "${directory[$i]}" != "." ]; then 
    
    NEWDIRNAME=`echo ${directory[$i]##*/}`
    NEWDIRNAME=`echo ${NEWDIRNAME} | sed 's/amarok/pana/g'`
    NEWDIRNAME=`echo ${NEWDIRNAME} | sed 's/Amarok/Pana/g'`
    NEWDIRNAME=${directory[$i]%/*}"/"$NEWDIRNAME
    echo "Renaming directory ${directory[$i]} to ${NEWDIRNAME}"
    mv ${directory[$i]} ${NEWDIRNAME}
  fi
  echo "Finished folder ${directory[$i]}"
  echo ""
done

 


NEWFILENAME=$FILENAME
NEWFILENAME=`echo ${NEWFILENAME} | sed 's/amarok/pana/g'`
NEWFILENAME=`echo ${NEWFILENAME} | sed 's/Amarok/Pana/g'`

echo "Recompressing files into $NEWFILENAME"
tar jcvf $NEWFILENAME `ls` &> /dev/null
mv $NEWFILENAME ..
cd ..

echo "rm -f -r ${FILENAME}.tmp/"
# rm -f -r "${FILENAME}.tmp/"

Once you have done all this, then just import the new Pana version of your script using the Pana Script Manager.

Howto Install Sun JDK with EE into Ubuntu 10.4

2010-03-30T19:51:00.004+13:00

This tutorial is for Ubuntu 10.4 Beta. 10.4 full release is only next month. Well here you go.

Download Java for linux from the Oracle Sun website.

Download the 32bit version of libstdc++5 from here:

http://packages.debian.org/lenny/amd64/libstdc++5/download

Make sure you get the 32 bit version. It would seem that the installer for Java requires that you have a 32bit version of this library installed. I think this is for the installer GUI only.

Extract the library like this:

dpkg-deb -x libstdc++5_3.3.6-18_i386.deb libstdc++5

Copy the library into /usr/lib32

sudo cp libstdc++5/usr/lib/libstdc++.so.5.0.7 /usr/lib32/

Create a simlink to the file in /usr/lib32/

sudo ln -s libstdc++.so.5.0.7 libstdc++.so.5

Now you can install java like this:

./java_ee_sdk-5_08-jdk-6u18-linux.bin

All done!

VirtualBox - AMD-V and KVM

2010-03-28T17:31:00.002+13:00

I usually use the KVM on my linux host for Virtualisation. It's default with Fedora and I know it. Usually ...

Today I thought I would try the Ubuntu 10.4 Beta 1 on VirtualBox. I installed the latest VirtualBox ( version 3.1.6 ) and then had a crack at installing the Ubuntu 10.4 64bit. Of course an error came up saying that while AMD-v was enabled, it was not operational. After much googleling about, I found that the KVM modules were causing VirtualBox issues with access.

So a quick "rmmod kvm_amd" and "rmmod kvm" fixed it.

Next time I want to use KVM, I will need to re-enable these with, I suppose, a quick, "insmod kvm_amd" and "insmod kvm"

Freeview ( DVB-T ) Linux / Fedora in New Zealand.

2010-03-03T22:38:00.005+13:00

I dont have anything fancy other than a fairly good CPU. Its an AMD Athlon(tm) 64 X2 Dual Core Processor 4000+.

I have an avermedia A800 DVB-T tuner from www.nicegear.co.nz

I DO NOT have a VDPAU enabled graphics card which can do on board H.264 decoding for me. That will be soon though. Need to save a bit of cash first.

I am in New Zealand so there are some special areas of concern that needed to be addressed.

Here is a very brief rundown of how i get free view working in my linux.

**** COMMENT ****
Here is a great website. I use this now for post processing.
http://quadpoint.org/projects/simplerip#audio_and_video_bitrates

On my fedora system i have to change -oac copy to -oac pcm and of course I still use my compiled mplayer and the -lavdopts fast:threads=2 option on the 3 lines.

Steps to get Mplayer working.


********************************************************************************
******************** ~./mplayer/channels.conf **********************************
********************************************************************************

TV ONE:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_AUTO:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_16:HIERARCHY_NONE:250+100:300:1200
TV2:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_AUTO:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_16:HIERARCHY_NONE:251+101:301+1010:1201
TVNZ 6:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_AUTO:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_16:HIERARCHY_NONE:252+102:302+1020:1202
TVNZ 7:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_AUTO:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_16:HIERARCHY_NONE:253+103:303+1025:1203
TV3:570000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_AUTO:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_16:HIERARCHY_NONE:450+300:410:1300
C4:570000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_AUTO:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_16:HIERARCHY_NONE:451+301:401:1301
TV3 PLUS1:570000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_AUTO:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_16:HIERARCHY_NONE:452+302:402:1302
Maori Television:666000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_AUTO:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_16:HIERARCHY_NONE:550+200:600:1400
Parliament TV :666000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_AUTO:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_16:HIERARCHY_NONE:551+201:601:1401
Test Channel:666000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_AUTO:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_16:HIERARCHY_NONE:552+202:602:1402
ChineseTV:666000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_AUTO:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_16:HIERARCHY_NONE:553+203:603:1403
PRIME:666000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_AUTO:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_16:HIERARCHY_NONE:554+204:604:1404
Reserved 6KSD:666000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_AUTO:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_16:HIERARCHY_NONE:0:605:1405
Freeview | HD:666000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_AUTO:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_16:HIERARCHY_NONE:556+206:606:1406
Radio NZ National:666000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_AUTO:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_16:HIERARCHY_NONE:0+220:620:2000
Radio NZ Concert:666000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_AUTO:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_16:HIERARCHY_NONE:0+221:621:2001
BaseFM:666000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_AUTO:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_16:HIERARCHY_NONE:559+209:609:2002
tvCentral:666000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_AUTO:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_16:HIERARCHY_NONE:560+210:610:1408

********************************************************************************
******************** The DEVEL FILES I INSTALLED ( RELEVANT ONES ) *************
********************************************************************************

Feb 03 21:38:08 Installed: alsa-lib-devel-1.0.22-2.fc12.x86_64
Feb 04 14:32:30 Installed: xorg-x11-drv-nvidia-devel-190.53-1.fc12.x86_64
Feb 04 14:37:13 Installed: gstreamer-devel-0.10.25.1-3.fc12.x86_64
Feb 04 14:37:15 Installed: gstreamer-plugins-base-devel-0.10.25.1-2.fc12.x86_64
Feb 04 14:37:16 Installed: gstreamer-plugins-bad-devel-0.10.17-2.fc12.x86_64
Feb 04 14:37:17 Installed: gstreamer-plugins-good-devel-0.10.17-4.fc12.x86_64
Feb 04 14:42:59 Installed: libvdpau-devel-0.3-1.fc12.x86_64
Feb 04 15:02:07 Installed: x264-devel-0.0.0-0.26.20091026gitec46ace7.fc12.x86_64
Feb 04 15:52:22 Installed: libXvMC-devel-1.0.4-8.fc12.x86_64
Feb 18 17:46:24 Installed: kernel-devel-2.6.31.12-174.2.19.fc12.x86_64
Feb 23 21:53:46 Installed: lame-devel-3.98.2-3.fc11.x86_64
Feb 23 21:53:47 Installed: xvidcore-devel-1.2.1-3.fc12.x86_64


Make sure you also have the relevant libraries that go with these development
packages.


********************************************************************************
******************* COMPILE MPLAYER WITH MULTI THREADING ***********************
********************************************************************************

grab mplayer svn
svn checkout svn://svn.mplayerhq.hu/mplayer/trunk mplayer

now get ffmpeg-mt
git clone git://gitorious.org/ffmpeg/ffmpeg-mt.git

replace the libav* folders in mplayer-svn with the libav* folders in ffmpeg-mt

cd mplayer
./configure
*** FOR 64 BIT FEDORA use ./configure --libdir=/usr/lib64

*** WHEN this completes have a look to see which modules are compiled in.  The
*** video out, video codecs, audio out and audio codecs will be listed.
*** If you are missing any you need or think you might need then install the
*** relevant library and -devel packages and run ./configure again.

make

********************************************************************************
******************* RESULTS FROM MY ./configure ********************************
********************************************************************************

  Enabled optional drivers:
    Input: dvdnav(internal) ftp pvr tv-v4l2 tv-v4l tv libdvdcss(internal) dvdread(internal) vcd dvb network 
    Codecs: xvid libavcodec(internal) real xanim faad2(internal) libmpeg2(internal) mp3lib(internal) tremor(internal) 
    Audio output: alsa oss v4l2 mpegpes(dvb) 
    Video output: v4l2 pnm jpeg matrixview opengl mpegpes(dvb) fbdev xvidix cvidix vdpau xv x11 xover yuv4mpeg md5sum tga 

  Disabled optional drivers:
    Input: vstream radio tv-dshow live555 nemesi cddb cdda smb 
    Codecs: libschroedinger libdirac x264 libdv libopencore_amrwb libopencore_amrnb qtx win32 faac musepack libdca liba52 libtheora speex toolame twolame libmad liblzo gif OpenJPEG 
    Audio output: sun openal jack pulse nas esd arts ivtv dxr2 sdl 
    Video output: zr zr2 ivtv dxr3 dxr2 sdl vesa gif89a svga caca aa ggi xmga mga winvidix 3dfx dga xvmc dfbmga directfb bl xvr100 tdfx_vid wii s3fb tdfxfb 

********************************************************************************
******************* Here is a link to a person who did it in mandriva **********
********************************************************************************
http://blog.linuxbox.co.nz/2008/08/hauppauge-wintv-hvr900-on-mandriva.html

********************************************************************************
******************* Various mplayer commands ***********************************
********************************************************************************

RECORD A CHANNEL INDEFINATELY ( KILL OR CTRL+C TO END THE THING )
#!/bin/bash
/home/dave/devel/mplayer-mt/mplayer_svn/mplayer -dvbin file=/home/dave/.mplayer/channels.conf -dumpfile /home/dave/Videos/$1.ts -dumpstream dvb://"$2"


ENCODE A RECORDING
#!/bin/bash
/home/dave/devel/mplayer-mt/mplayer_svn/mencoder -oac lavc -ovc lavc -of mpeg -mpegopts format=xsvcd -vf scale=480:576,harddup -srate 44100 -af lavcresample=44100 -lavcopts vcodec=mpeg2video:mbd=2:keyint=15:vrc_buf_size=917:vrc_minrate=600:vbitrate=2500:vrc_maxrate=2500:acodec=mp2:abitrate=224:aspect=16/9 -ofps 25 -lavdopts fast:threads=4 -mc 10 -o /data/$1.mpg -quiet $1


WATCH A RECORDING.  THIS IS THE RAW STREAM CAPTURED DURING BY THE RECORD A CHANNEL SCRIPT ABOVE.

#!/bin/bash
/home/dave/devel/mplayer-mt/mplayer_svn/mplayer -vc ffh264 -vo gl -lavdopts fast:threads=4 -vf pp=fd -mc 10 -framedrop -cache 10240 -delay -0.3 /home/dave/Videos/$1


WATCH TV STRAIGHT.  THIS IS FINE FOR ALL CHANNELS EXCEPT TV3 WHICH IS AT FULL HD AND MY CPU DOESNT DECODE IT FAST ENOUGH.  EVEN IN MULTI THREADING.

#!/bin/bash
/home/dave/devel/mplayer-mt/mplayer_svn/mplayer -vo gl -lavdopts fast:threads=2 -mc 10 -framedrop -vf pp=fd -cache 10240 -delay -0.3 dvb://"$1"

So have fun.

chomping isql and date conversion

2010-01-08T00:11:00.003+13:00

Today a colleague was working on retrieving data from a Windows SQL Server database from within linux. He needed a bash script that would execute a query against the database and dump the results in comma delimited format.

The SQL query he wrote was rather long and so was broken up into new lines and indented to ease editing and readability. This, however, caused a snag in isql.

isql is a commandline tool that ships with linux ODBC that can be used to connect to an ODBC DSN and execute sql commands as well as execute batch files in unattended mode.

isql parses batch files ( essentially SQL scripts ) in such a way as that each line is treated as a single command. It does not recognise the GO statement or any other command terminator.

Our workaround was to chomp the script prior to piping it into isql. So here we go:

Here is an example script.


#--- mySqlScript.sql ---

SELECT   TOP 100 *
FROM     myTable t
WHERE    t.created_at between '2009-01-01' and '2009-12-31'
         AND t.enabled = 1

As you can tell the script is formatted in a nice and legible way which would break in isql batchmode.

The fix is rather simple but also elegant because we can keep the script in it's original format and remove the newlines ( chomp ) just before it is sent to isql.

Here is the commandline that does just that.


[dave@fedora junk]$ cat mySqlScript.sql | perl -ne 'chomp; print unless /^$/'
SELECT   TOP 100 *FROM     myTable tWHERE    t.created_at between '2009-01-01' and '2009-12-31'         AND t.enabled = 1[dave@fedora junk]$

So the final step is to pipe all that into an isql command.


[dave@fedora junk]$ cat mySqlScript.sql | perl -ne 'chomp; print unless /^$/' | isql -b -d, my_dsn userName passW0rd

Now the above script failed on my computer because (a) I do not have linux odbc installed at home and (b) I do not have a dsn called my_dsn to connect to.
The chomping, however, does work!

An excellent PHP script to handle time difference

2009-12-23T23:34:00.002+13:00

I am working on a project that involves displaying the time difference between a start and end time on a website.

The following script is lifted directly from: http://aidanlister.com/2004/04/making-time-periods-readable/


/**
 * A function for making time periods readable
 *
 * @author      Aidan Lister 
 * @version     2.0.0
 * @link        http://aidanlister.com/2004/04/making-time-periods-readable/
 * @param       int     number of seconds elapsed
 * @param       string  which time periods to display
 * @param       bool    whether to show zero time periods
 */
function time_duration($seconds, $use = null, $zeros = false)
{
    // Define time periods
    $periods = array (
        'years'     => 31556926,
        'Months'    => 2629743,
        'weeks'     => 604800,
        'days'      => 86400,
        'hours'     => 3600,
        'minutes'   => 60,
        'seconds'   => 1
        );

    // Break into periods
    $seconds = (float) $seconds;
    foreach ($periods as $period => $value) {
        if ($use && strpos($use, $period[0]) === false) {
            continue;
        }
        $count = floor($seconds / $value);
        if ($count == 0 && !$zeros) {
            continue;
        }
        $segments[strtolower($period)] = $count;
        $seconds = $seconds % $value;
    }

    // Build the string
    foreach ($segments as $key => $value) {
        $segment_name = substr($key, 0, -1);
        $segment = $value . ' ' . $segment_name;
        if ($value != 1) {
            $segment .= 's';
        }
        $array[] = $segment;
    }

    $str = implode(', ', $array);
    return $str;
}

Things to do with an iPod and Fedora

2009-12-08T21:10:00.002+13:00

My iPod Nano 8gb. ( not the fancy new one with the video ) works perfectly well inside fedora so long as you bear in mind the following basic points:

The thing doesn't play ogg files so if you have any of those on your system, they can not be synced.
Depending on the application you use, you end up with podcasts being saved under music.
Sometimes you get images sometimes you don't.
Albums with Various Artists do not appear under compilations. Instead your Artists menu shows a long list of artists most of which only have one track.

Applications that ship with fedora and support the iPod:

gtk-pod ( manage your ipod )
gpodder ( download podcasts etc - these actually go into the podcasts menu on the device. )
Amarok
Rythmbox

Other music applications I have played with that support the iPod:

Songbird 2

I should not here that once again, I have found that Amarok 1.4 is still the best player out there. I had to install it using the F11 binaries ( just search for them on google ) and a couple of symlinks:

ln -s /usr/lib64/libssl.so.10 libssl.so.8
ln -s /usr/lib64/libcrypt.so.10 libcrypt.so.8

So there you go!

Bash Plus Spaces In Filenames Equals Headache

2009-11-25T21:47:00.001+13:00

OR NOT...

Check this out:

http://www.cyberciti.biz/tips/handling-filenames-with-spaces-in-bash.html

Understanding Redhat Kickstart

2009-11-21T21:06:00.008+13:00

Kickstart provides linux system administrators a method for installing linux on pc's in unattended mode. A properly formatted Kickstart configuration will perform all tasks such as setting the root password, the networking interfaces, the disk partitioning scheme as well as the installation of packages.

I needed to build a clustered reverse proxy the other day to test if it could be used for SSL termination. I was looking for a way to aggregate multiple webservers behind a proxy server so that they could have their SSL terminate at the proxy, then the HTTP traffic passed through a Level 7 ( application layer ) scanner to test for sql injection and the like.

So I fired up my trusty Virtualbox and built a standard Centos 5.4 server. I copied the CD into the ftp directory and started NFS with an export to that FTP directory. This server was to serve as an install NFS server for the 2 proxy servers as well as a yum repository for them. That's why I placed the CD files in the FTP location.

So here is my example proxy diagram. It is very similar to the basic openOffice.org version in my previous post. I needed to make sure that the proxy servers had apache installed on them for the reverse proxie's and heartbeat. I ended up manually installing heartbeat. I had my reasons... :)

The trick making a successful kickstart file is to take one that is generated for you by anaconda. Anaconda is the software that installs your new Redhat based system. After a manual install is complete a copy of the kickstart file for that installation is stored in: /root/anaconda-ks.cfg

I simply copied this over to /var/www/html/ks.cfg and began editing it with vi. Here is the result of that editing...

[root@server html]# cat ks.cfg
# Kickstart file automatically generated by anaconda.

install
nfs --server 192.168.56.254 --dir=/var/ftp/pub/server
lang en_NZ.UTF-8
keyboard us
network --device eth0 --bootproto dhcp --nameserver 192.168.56.1
rootpw --iscrypted $1$ewn======YEAH RIGHT!==2hBDFz1
firewall --enabled --port=22:tcp
authconfig --enableshadow --enablemd5
selinux --enforcing
timezone --utc Pacific/Auckland
bootloader --location=mbr --driveorder=hda
zerombr
clearpart --all
part / --fstype ext3 --size=0 --grow --ondisk=hda
%packages
@base
@core
@editors
@server-cfg
@text-internet
@web-server
heartbeat
keyutils
trousers
fipscheck
device-mapper-multipath
-cups
-bluez-utils

This file does the following things in order:

Does an INSTALL and not an UPGRADE
Defines the location of the NFS share from which to install from.
Sets the default language
Sets the default keyboard layout
Sets up the network interface
Specifies the root password ( i have masked mine here )
Turns on the firewall and allows port 22 for SSH through it.
Configures authconfig to allow shadow passwords and md5 encryption.
Turns SELINUX on and sets it to ENFORCING mode.
Sets the timezone.
Defines where the bootloader will be installed. Master Boot Record on the disk HDA
Zeros the master boot record
Clears all partitions
Defines one partition mounted on / ( root ), filetype ext3, 0 min size configured to grow to extents of disk and on disk HDA
Sets up packages. Package Groups are preceeded with an @ symbol, individual packages are named normally one per line and packages to exclude are preceeded with a - symbol.

There is one very useful feature left out and that is POST INSTALLATION SCRIPTS. This is a script that will be executed after the system has installed and might used for creating default users or determining which services to turn on and off after an install.

All going well the above VERY BASIC kickstart file should result in a completly automated install, assuming your NFS share is available and the machine can connect to a DHCP server and be assigned an IP address.

Try it out some time. It's quite fun to watch.

Scheduled Tasks on HA Pairs in Windows

2009-08-02T21:41:00.005+12:00

Sometimes you might find yourself deciding that a Scheduled task is required for some report or file transfer on a production system. The production system, however, is part of a high availability (HA) pair. The particular method of dealing with HA in this particular ( hypothetical of course ) case works by moving an IP Address called a Virtual IP address between the servers of the HA Pair. Whichever server holds the VIP is the current node.

Your application needs to know if it is executing on the current node before starting.

Wrap your application in a very simple snippet of Comand Line script that:

Looks for the VIP in ipconfig.
If no errors found then jump to the section of your script that launches your application or another script to do the task.
If an error is found then jump to the section of your script that notifies the log.

@echo off
REM #Testing FIND in IPCONFIG
SET VIPTHATWORKS="192.168.122.1"
SET VIPTHATFAILS="192.168.122.127"



ipconfig /all | find %VIPTHATWORKS%
if ERRORLEVEL = 1 goto VIP_NOT_FOUND

REM #We are here becuase the find returned a result.
REM #It is safe to execute the rest of the application.
echo "VIP Found. - Continuing with application..."

REM #Pause for user input.
pause
goto END_SCRIPT

:VIP_NOT_FOUND
REM #This part of the script is where you would handle any
REM #error logging or other admin related
echo "Could not find a VIP. - Exiting"

REM #Pause for user input.
pause
goto END_SCRIPT

:END_SCRIPT
echo "end of script reached."

Things to do with an iPod

2009-08-01T16:08:00.002+12:00

I have had my iPod 8gb Nano ( Video ) for about 4 months now and I have been having heaps of fun with it. I know it's not cool like an iPod Touch or iPhone but it is still plenty powerful.

I have tried using it with Linux and although it is supported in Amarok ( I am still on Amarok 1.4 due to lack of Fedora packages for the new version which actually has replay-gain for Xine included ) I have found that the support for pod casts is a little limited. For example: I like to download all the pod casts in a feed at once so that I can catch up to the current feed. So I stick with iTunes. This means sticking with a windows pc or if you are lucky to own one, a mac.

Initially I spent heaps of time building the perfect play list on my iTunes and moving that over to the iPod. This was fun but only for a short while. The main problem being that no matter how many weeks of music you have on your iPod, you still end up with the feeling that you have heard it all before.

I was struck by the "Only options is to rebuild windows" bug recently. You might know that as the, "My Computer is getting slower and slower and doesn't shut down properly anymore" bug. Stupidly, I had no backup of my iTunes library so I lost the whole thing. About 200 CDs worth. Nice...

What was initially looking like a bad thing for me has really turned into a blessing in disguise. A friend suggested I try an audio book and pointed me to www.podiobooks.com. Its a great resource for free audio books, usually read by the Author. If you sign up you can subscribe to audio books which are exposed via an XML feed which is completely compatible with iTunes. Just select how you want the feed to be released to you: once a day, once every 3 days etc. and then select the Subscribe Via iTunes button. If the book is complete, ie: All episodes are available, then you can click the release all episodes now button and then subscribe to the feed provided. All episodes will be loaded into your iTunes library ready for you to click the "Get All" button.

All in all I seriously recommend Podiobooks. I suggest you start off with, "Infected" by Scott Sigler. Its a dark / fantasy kind of book with horror content but a riveting story. I was hooked within 5 minutes and found myself looking forward to my next bus trip to work so I could find out more about Perry Dawcey's plight...

Happy listening.

Rip from WAV to MP3 and add ID3 tags -- All with Perl

2009-07-31T22:50:00.002+12:00

Ok - so I know I have said before that I am not much of a perl fan. ( yes it's safe, I am wearing my flame proof suit. )

I will, however, say now; I am coming around. It would seem that I am starting to see the light. I needed to extract some files from a CD I have and convert them to MP3. The CD was not made with extracting to mp3 in mind if you get my drift. It's ok, because the CD is my copy and I want to listen to it on my iPod. Regular extracting on iTunes was not working out for me. So I rolled my chair over to my trusty Linux Box and perl, id3, cdparanoia and lame.

First thing I did was use cdparanoia to extract all the tracks. This went swimmingly leaving me with a folder full of files called 01.cdparanoia.wav or something just as useless. The files played ok but I needed to name them all. This particular CD was not listed on Music Brainz so I couldn't do anything fancy with FreeCDDB or whatever they call that now days.

* shudder * - I had to manually edit 17 filenames. I left the .wav extensions of course. I thought that I might need to get the tags later so I decided to choose a naming scheme for my file names.

TRACK-TITLE_ARTIST.wav
01-My Song Title_My Artist Name.wav

and so on.

Then I wrote a little script that:

Loops through each file in a folder with the .wav extension
Saves a variable with the file name but with .mp3 instead of .wav
Regexes ( new word ) out the various parts of the file name into $track, $title, $artist ( thanks to the nice naming scheme this was easy. )
Uses Lame to convert the file. Lame now days is pretty cool.
Uses id3v2 to add the tags extracted from the file name. I can use amarok to add the album titles etc once it is imported into Amarok.

Here is the script:

#!/usr/bin/env perl

@files = <*.wav>;

foreach $file (@files) {
 
 $newfile = $file;
 $newfile =~ s/\.wav/\.mp3/;
 
 
 $file =~ m/^(.*)-(.*)_(.*).wav/;
  
 $track = $1;
 $title = $2;
 $artist = $3;
 
 system("lame", "--preset", "cd" ,"$file" ,"$newfile");
 system("id3v2", "--artist", "$artist", "$newfile");
 system("id3v2", "--song", "$title", "$newfile");
 system("id3v2", "--track", "$track", "$newfile");

}

Real simple to connect to MSSQL from Perl

2009-07-30T22:09:00.004+12:00

Again, this took me a little while but here I am. Connecting to a database ( SQL SERVER 2008 ) from a Windows Server 2008 VM I own ( thanks TECH ED 2008 ) and Active State Perl.

Install Scite cos it's a nice editor and you are away laughing...

#!/usr/bin/perl

use DBI;

my $DSN = 'driver={SQL Server};Server=localhost; database=AdventureWorks;TrustedConnection=Yes'; 
my $dbh = DBI->connect("dbi:ODBC:$DSN") or die "$DBI::errstr\n";

my $sth = $dbh->prepare('select top 10 * from Sales.vSalesPerson')
    or die "Couldn't prepare statement: " . $dbh->errstr;

$sth->execute();

while( @data = $sth->fetchrow_array())
{
    foreach(@data) {
        print "[$_]";
    }
    print "\n\n";
}

$sth->finish;
$dbh->disconnect;

Results:


C:\Users\dave\Documents>perl perl1.pl
[268][][Stephen][Y][Jiang][][North American Sales Manager][238-555-0197][stephen
0@adventure-works.com][0][2427 Notre Dame Ave.][][Redmond][Washington][98052][Un
ited States][][][][677558.4653][.0000]

[275][][Michael][G][Blythe][][Sales Representative][257-555-0154][michael9@adven
ture-works.com][1][8154 Via Mexico][][Detroit][Michigan][48226][United States][N
ortheast][North America][300000.0000][4557045.0459][1750406.4785]

[276][][Linda][C][Mitchell][][Sales Representative][883-555-0116][linda3@adventu
re-works.com][0][2487 Riverside Drive][][Nevada][Utah][84407][United States][Sou
thwest][North America][250000.0000][5200475.2313][1439156.0291]

[277][][Jillian][][Carson][][Sales Representative][517-555-0117][jillian0@advent
ure-works.com][1][80 Sunview Terrace][][Duluth][Minnesota][55802][United States]
[Central][North America][250000.0000][3857163.6332][1997186.2037]

[278][][Garrett][R][Vargas][][Sales Representative][922-555-0165][garrett1@adven
ture-works.com][0][10203 Acorn Avenue][][Calgary][Alberta][T2P 2G8][Canada][Cana
da][North America][250000.0000][1764938.9859][1620276.8966]

[279][][Tsvi][Michael][Reiter][][Sales Representative][664-555-0112][tsvi0@adven
ture-works.com][1][8291 Crossbow Way][][Memphis][Tennessee][38103][United States
][Southeast][North America][300000.0000][2811012.7151][1849640.9418]

[280][][Pamela][O][Ansman-Wolfe][][Sales Representative][340-555-0193][pamela0@a
dventure-works.com][1][636 Vine Hill Way][][Portland][Oregon][97205][United Stat
es][Northwest][North America][250000.0000][.0000][1927059.1780]

[281][][Shu][K][Ito][][Sales Representative][330-555-0120][shu0@adventure-works.
com][2][5725 Glaze Drive][][San Francisco][California][94109][United States][Sou
thwest][North America][250000.0000][3018725.4858][2073505.9999]

[282][][JosΘ][Edvaldo][Saraiva][][Sales Representative][185-555-0169][josΘ1@adve
nture-works.com][0][9100 Sheppard Avenue North][][Ottawa][Ontario][K4B 1T7][Cana
da][Canada][North America][250000.0000][3189356.2465][2038234.6549]

[283][][David][R][Campbell][][Sales Representative][740-555-0182][david8@adventu
re-works.com][0][2284 Azalea Avenue][][Bellevue][Washington][98004][United State
s][Northwest][North America][250000.0000][3587378.4257][1371635.3158]


C:\Users\dave\Documents>

Basic Perl file parser - useful for working with logs

2009-07-26T23:39:00.002+12:00

Really I just need to remember how to do this:


#!/usr/bin/env perl
#yum log file
use warnings;
use strict;

open FILE, "/var/log/yum.log" or die $!;
while (<FILE>) {
 /^(.+?)\s(.+?)\s(.+?)\s(.+?)\s(.+?)\s/;
 my ($month,$day,$time,$action,$package) = ($1,$2,$3,$4,$5);
 if( $action =~ /Installed/ ) {
  my $suffix = 'th';
  if( $day == 1 ){ $suffix = 'st'; }
  if( $day == 21 ){ $suffix = 'st'; }
  if( $day == 31 ){ $suffix = 'st'; }
  if( $day == 2 ){ $suffix = 'nd'; }
  if( $day == 22 ){ $suffix = 'nd'; }
  if( $day == 3 ){ $suffix = 'rd'; }
  if( $day == 23 ){ $suffix = 'rd'; }
  
  print "$day$suffix of $month at $time\n"
 }
}
close FILE

Joining multiple PDF documents with Gostscript.

2009-07-26T20:22:00.008+12:00

This really works.

I downloaded all the chapters of beginning perl as published here: http://www.perl.org/books/beginning-perl/

To do it in a painless way I first used curl to grab the html.

curl http://www.perl.org/books/beginning-perl/ -o bp.txt

Then I used grep to grab the hyperlinks to pdf files and piped that into another file.

cat bp.txt | grep -o "<a.*pdf"> bp2.txt

Then I used sed to clean up a little bit.

cat bp2.txt | sed "s/^<a href=\"//g"> bp3.txt

Then I used wget to download each pdf.

for line in `cat bp3.txt`; do wget $line; done

So now I had a whole bunch of pdf files that together make up one book. A quick hunt on google produced this guy.
http://www.linux.com/news/software/applications/8229-putting-together-pdf-files

And I was away laughing.


gs -dBATCH -dNOPAUSE -q -sDEVICE=pdfwrite -sOutputFile=beginning-perl.pdf 3145_Intro.pdf 3145_Chap*.pdf 3145_App*.pdf 3145_Index.pdf