Stev.Org

Python - Doesn't do real for loops

james@stev.org — Fri, 24 Feb 2012 07:00:00 +0000

Previous I wrote about python and its weakness for 2d arrays. This has now got me thinking a little more out of the box.

I have also discovered that python doesn't actually support the tradition for loop. It supports a for each loop instead which has issues when attempts to run quite large loops. This is a simple example of one of these issues which you can only ever discover when it is too late normally resulting in a crash of some kind.

To start with lets look at the simple loop.

>>> for i in range(0, 10):
...  print i
...
0
[cut]
9
>>>

It does of course print 0 - 9 as output. But look at how it is constructed. It uses the range function and range does't produce a number range. It produces a list of numbers from the specified paramaters on the function. So whats the big deal?

Well if you need to run a "big" for loop in python you simply cannot use it ... Well not without using lots and lots of memory. Not to mention that the entire list has to be created before the first for loop runs. Which is just going to make it dog slow to begin with.

So to get python to use around 170MB of memory (in a for loop) simply write

>>> for i in range(0, 10000000):
...     print i

But wait... 10 million? That a very big number. Or is it? It is not uncommon for large data sets to grow to 10 millions records. This would be exactly the sort of place you might want to use python as well.

So I guess the alternative to this is to role your own for loop using a while loop. Which I have had todo for various other things for variable stride lengths across array's. Though it kinda defets the entire purpose of there being a for loop in the first place.

As a bonus surprise while playing with this feature it probably also open's up a whole new range of security holes if a loop counter is supplied from end user input. This would lead to using massive amounts of memory. The real surprise I was greated with the following from the linux kernel who's out of memory killer could have also picked apache or a database engine to kill in its place.

[5537286.542945] Out of memory: kill process 14166 (python) score 230114 or a child
[5537286.543168] Killed process 14166 (python)

But I guess thats python's way of doing things!

Python - 2d Array's don't work.

james@stev.org — Wed, 22 Feb 2012 20:25:00 +0000

If you have been working with python you will notice that 2d array's just don't work. This might come as a surprise since almost every other modern programming language supports 2d array's. What might even come as a bigger surprise is that pythin doesn't support array's at all in its language. It uses lists to support this functionality. It might look like an array but it is actually a list.

This can be showen with the following. Note the error about the list.

arr = []
arr[0] = 1


Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: list assignment index out of range

So once you stop thinking about trying to create and array in pythin (which is easy since they don't exist) and start thinking about creating a list of items things become much easyier to understand why your 2d arrays of arr = [][] just doesnt work. As you have just attempts to create 2 lists and set a variable with them.

So to create a single array list is easy and we can do so using the following code.

>>> arr = range(0,5)
>>> print arr
[0, 1, 2, 3, 4]

>>> arr[0] = 1
>>> print arr
[1, 1, 2, 3, 4]

I actually find this really ugly. Since we just created a list of incrementing number and the array still needs to be set to zero. So to create a real array preset to zero you need to use the following.

>>> arr = []
>>> for i in range(0, 5):
...  arr.append(0)
...
>>> print arr
[0, 0, 0, 0, 0]

So moving onto the 2d array and you do exactly the same thing. Except you create a list and put it in the list item. Like this

>>> arr = []
>>> for i in range(0, 5):
...  x = []
...  for j in range(0, 5):
...   x.append(0)
...  arr.append(x)
...
>>> print arr
[
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0]
]

Now you can read / write to the items by using an array like syntax or arr[0][0] .. arr[4][4].

There is another alternative support to using 2d arrays in python by using a single list and then calulating the offsets. However for the particular problem I was trying to solve it really didn't help since python doesn't appear to support proper for loops either!

I guess thats what you get with python!

C - get home dir location in linux

james@stev.org — Mon, 20 Feb 2012 08:00:00 +0000

This is a short example of how to get the home directory location in linux in a C program. There is really a few choices here and depending on what your requirements are you might want to use one or other methods. The first method can be changed by the user. The second method cannot. However if the first method fails you should drop faill back on the other method.

The simple method is to pull the enviroment variable "HOME"

The slightly more complex method is to read it from the system user database.

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <pwd.h>
#include <sys/types.h>


int main(int argc, char **argv) {


        char *homedir = getenv("HOME");

        if (homedir != NULL) {
                printf("Home dir in enviroment");
                printf("%s\n", homedir);
        }

        uid_t uid = getuid();
        struct passwd *pw = getpwuid(uid);

        if (pw == NULL) {
                printf("Failed\n");
                exit(EXIT_FAILURE);
        }

        printf("%s\n", pw->pw_dir);

        return 0;
}

C - ip address validation

james@stev.org — Fri, 17 Feb 2012 08:00:00 +0000

A short example to show how to check and ip address is actually an ip address in C. I have seen a number of time that people provide solution using reg'ex and various other methods to attempt to do this. However there is a standard function that will test for you.

The function is called inet_pton will be able to tell you if a string is acutally an ip address or not. Typically the function also has another use which is to convert an ip address into its network byte ordered integer format.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include <arpa/inet.h>

int main(int argc, char **argv) {
    unsigned long ip = 0;

    if (argc < 2) {
        printf("Usage: %s <ipv4>\n", argv[0]);
        exit(EXIT_FAILURE);
    }

    if (0 == inet_pton(AF_INET, argv[1], &ip)) {
        printf("Failed\n");
        exit(EXIT_FAILURE);
    }

    printf("IP: %ld\n", ip);
    return 0;
}

C - Get current ip address of an interface

james@stev.org — Wed, 15 Feb 2012 08:00:00 +0000

This is a short example to retrive an ip address from an interface in linux. It is typically used to be able to bind a port to a specific address on a machine.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <unistd.h>

#include <arpa/inet.h>
#include <net/if.h>

#include <sys/ioctl.h>
#include <sys/types.h>
#include <sys/socket.h>

int main(int argc, char **argv) {
    int sock = socket(PF_INET, SOCK_DGRAM, 0);
    struct ifreq req;

    if (argc < 2) {
        fprintf(stderr, "Usage: %s <interfacename>\n", argv[0]);
        exit(EXIT_FAILURE);
    }

    if (sock < 0) {
        perror("socket");
        exit(EXIT_FAILURE);
    }

    memset(&req, 0, sizeof(req));
    strncpy(req.ifr_name, argv[1], IF_NAMESIZE - 1);

    if (ioctl(sock, SIOCGIFADDR, &req) < 0) {
        perror("ioctl");
        exit(EXIT_FAILURE);
    }

    printf("%s\n", inet_ntoa(((struct sockaddr_in *)&req.ifr_addr)->sin_addr));


    close(sock);

    exit(EXIT_SUCCESS);
    return 0;
}

Read / Write a pid for a file in linux

james@stev.org — Mon, 13 Feb 2012 08:00:00 +0000

This is a short example to show you how to read and write a process id to a file in linux.

First the write.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/types.h>

int main(int argc, char **argv) {
    pid_t pid = getpid();

    if (argc < 2) {
        printf("Usage: %s <filename>\n", argv[0]);
        exit(EXIT_FAILURE);
    }

    FILE *fp = fopen(argv[1], "w");
    if (!fp) {
        perror("fopen");
        exit(EXIT_FAILURE);
    }

    fprintf(fp, "%d\n", pid);

    fclose(fp);

    return EXIT_SUCCESS;
}

Then the read

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/types.h>

int main(int argc, char **argv) {
    pid_t pid = 0;

    if (argc < 2) {
        printf("Usage: %s <filename>\n", argv[0]);
        exit(EXIT_FAILURE);
    }

    FILE *fp = fopen(argv[1], "r");
    if (!fp) {
        perror("fopen");
        exit(EXIT_FAILURE);
    }

    if (fscanf(fp, "%d\n", &pid) == 1) {
        printf("PID: %d\n", pid);
    } else {
        printf("Failed to Read PID\n");
    }

    fclose(fp);

    return EXIT_SUCCESS;
}

Using asprintf instead of sprintf or snprintf

james@stev.org — Fri, 10 Feb 2012 08:00:00 +0000

One of the problem that has always existed in c programming is to create a big enough buffer in order to store a string. This can create a chicken / egg problem when trying to use sprintf or snprintf where you don't know how big the buffer actually needs to be to store the string.

There is another function in linux which will allocate a buffer which is called asprintf (allocate string printf). this will make sure that the buffer is always going to be big enough to store the data and does not suffer from the same problems when using sprintf or snprintf.

The cost of using this is of course is performance releated since it needs to allocate memory and you also need to free it later. This type of allocation can be expensive if called a large number of times. There is also of course the problem of remembering to free the allocated memory later or it will create a memory leak.

To use this function you also need to define _GNU_SOURCE on the gcc command line (of in the program with a define before the includes)

Here is an example of using asprintf.

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv) {
    char *tmp = 0;

    if (asprintf(&tmp, "%s %d", "Magic Number", 1234) < 0) {
        perror("asprintf");
        exit(EXIT_FAILURE);
    }

    printf("%s\n", tmp);

    free(tmp);

    return 0;
}

Using perror in C to convert error numbers to a string

james@stev.org — Wed, 08 Feb 2012 08:00:00 +0000

A short example on how to convert error number to a human readable string in C.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char **argv) {
	int err = 0;

	if (argc < 2) {
		printf("Usage: %s <error number>\n", argv[0]);
		exit(-1);
	}

	err = atoi(argv[1]);

	printf("Error: %s\n", strerror(err));	


	return 0;
}

Postgres - Convert unix timestamp

james@stev.org — Thu, 02 Feb 2012 08:00:00 +0000

I am actually surprised that postgres does not have a builtin function to deal with unix timestamps. After all it does run on linux which does use unix timestamps. So it must be a common problem. Here is a quick solution.

CREATE OR REPLACE FUNCTION parseunixtimestamp(int)
	RETURNS timestamp AS 
		'SELECT TIMESTAMP ''epoch'' + $1 * INTERVAL ''1 second'''
	LANGUAGE SQL;

Now you can use it in select or insert statements. Like this

INSERT INTO TABLE (name, createdon) VALUES("a value", parseunixtimestamp(55436437))

If you need to you can remove it again later by running

DROP FUNCTION parseunixtimestamp(integer);

Example of how to overwrite argv in C

james@stev.org — Tue, 31 Jan 2012 08:00:00 +0000

A short code example to show how to overwrite the contents of argv. It is useful for when a password is passed on the command line and it should not be visible in the process list to other users on the same machine.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char **argv) {
	int i = 0;
	int len = 0;

	if (argc < 2) {
		printf("Usage: %s <some argument>\n", argv[0]);
		exit(EXIT_FAILURE);
	}

	len = strlen(argv[1]);
	for(i=0;i<len;i++) {
		argv[1][i] = 'x';
	}

	system("ps f");

	return 0;
}