Programmer's diary

Netlink sockets

2011-09-14T14:16:00.009+03:00

Recently I was requested to add IPv6 support to an existing IPv4 configuration interface written in C. The existing interface was written on top of a Linux OS, and it was used to bring up links, configure routes, attach alias addresses to interfaces, create VLAN interfaces and so on. Well, I rolled up my sleeves and begun the work. After a few hours of browsing the internet, I noticed that there really was not so much information about how to do all this. It seems to me that IPv6 is not yet so widely used. After a while I however was directed to netlink sockets.

First instructions about how to use netlink sockets, suggested using some wrapper library like libnl. However after bunch of unsuccessfull attempts I gave up with libnl (my mind was really not compatible with libnl documentation) and decided to use raw netlink sockets. Basic usage information for netlink sockets was easily avaliable. There is RFC discussing the netlink sockets, as well as man pages. However these do not seem to cover all pitfalls or usage information. In my journey to making this IPv6 interface to work I fell in more than one trap, and eventually ended up adding prints to kernel to see where the excution ended up, and where my requests were discarded...

But let's start the actual business.

Linux networking will be discussed in terms of links (interfaces), addresses and routes. For me these words mean something like:

link (interface), for example eth0 - a door to outer world. Or eth0.2, virtual interface (VLAN interface), nevertheless seen as a door to outer world by the system.

address: For example 192.168.1.77/32, or fe80::211:43ff:fe26:2b6c/64, a label telling what is behind the door - and address of one specific door.

route: for example 0.0.0.0 via dev eth0, or 10.34.143.0 255.255.255.0 gw 192.168.1.55 metric 2
which tells the system to kick a packet out through a specific door, if destination matches to certain something.

I guess I will shortly explain routes here. But before I do that, I'll mention that even though the netlink sockets are a way to configure and handle all these (well, if underlying drivers do support this - I guess they nowadays often do), they're also way more. Netlink is quite generic interface to exchange information between kernel and userspace. I know at least firewalls and neighbor cache can be managed via netlink interface on linux. However we concentrate on links, addresses and routes here.

Now, if we look at the previous route examples.

route 1 was format 0.0.0.0 via dev eth0
destination 0.0.0.0 (special, 'any' ip - means this is a default route, and all packacges for which we do not have better routing information will use this route.)
dev eth0 specifies interface in which this package should be directed.

route 2 was 10.34.143.0 255.255.255.0 gw 192.168.1.55 metric 2

destination 10.34.143.0
mask 255.255.255.0

This tells us that if package is destined to address 10.34.143.xx, then this route is selected. The mask information tells which bits of the destination address are meaningfull. Eg, all bits which are not zero in mask, will be meaningfull. destination/mask pair
192.168.254.0
255.255.254.0
would mean that packets targeted to 255.255.254.xx or 255.255.255.xx would go to this route. Another way to express this is using format < address > / < amout of meaningfull bits >

destination 10.34.143.0
mask 255.255.255.0

could be shown as
10.34.143.0/24 (mask 255.255.255.0 has 24 meaningfull bits since each 0-255 value occupies 8 bits. Thus 255.255.255 is 8+8+8 = 24)

gw means gateway. This is something like saying that the specified destinations are located behind a machine which address is xx.
Eg, saying that send these packets to gw machine, it knows where they should be redirected. Hence all packets going to this route will be first delivered to 192.168.1.55.

Metric 2... Metric is a way for us to define routes which are overlapping. Eg, if we have two routes, which' destinations would match, then we use
1. route with more accurate mask (Eg, if we specify route with mask 255.255.255.255 == host route, saying that destination address is exact, then this route will be preferred over routes with looser mask).
2. Route with smaller metric value.

With IPv6 the routing gets a bit more complicated, since some of the subnet information is built into addresses. I won't get into that now, hopefully you know what you're doing :)

So let's see. The netlink socket interface is typical socket interface. Eg, we send messages to socket, and receive replies from socket. We can also register for receiving reports from interesting events, and receive messages informing the changes. I however only show the typical send request = > receive responce sequences.

As mentioned, requests are divided to link, address and route requests. (families). And each of these have request types of

creating new < address/interface/route >
deleting old < address/interface/route >
getting information about existing < address/interface/route >

Exact defines (which also need to be filled in request) are:

RTM_NEWADDR
RTM_DELADDR,
RTM_GETADDR,

RTM_NEWROUTE
RTM_DELROUTE,
RTM_GETROUTE,

RTM_NEWLINK
RTM_DELLINK
RTM_GETLINK,

for links there also

RTM_SETLINK,

which need to be used to change existing link's attributes. NEWLINK requests with these attributes will be discarded (for me understanding this required adding prints to kernel...)

Actual request consist of:

1. message header type struct msghdr. I assume you're familiar with this standard message header.
2. netlink message header type struct nlmsghdr
3. request family specific header (struct ifaddrmsg / struct ifinfomsg / struct rtmsg)
4. set of family specific attributes

Netlink message header

struct nlmsghdr
{
    __u32     nlmsg_len; /* Length of message including header */
    __u16     nlmsg_type; /* Message content */
    __u16     nlmsg_flags;    /* Additional flags */
    __u32     nlmsg_seq; /* Sequence number */
    __u32     nlmsg_pid; /* Sending process port ID */
};

carries information necessary for
1. knowning the lenght of the message.
2. telling the type of the message (family)
3. flags telling how message should be handled - directly from /usr/include/linux/netlink.

/* Flags values */

NLM_F_REQUEST /* It is request message. */
NLM_F_MULTI /* Multipart message, terminated by NLMSG_DONE */
NLM_F_ACK /* Reply with ack, with zero or error code */
NLM_F_ECHO /* Echo this request */

/* Modifiers to GET request */
NLM_F_ROOT /* specify tree root */
NLM_F_MATCH /* return all matching */
NLM_F_ATOMIC /* atomic GET */
NLM_F_DUMP

/* Modifiers to NEW request */
NLM_F_REPLACE /* Override existing */
NLM_F_EXCL /* Do not touch, if it exists */
NLM_F_CREATE /* Create, if it does not exist */
NLM_F_APPEND /* Add to end of list */

for example every request should contain flag
NLM_F_REQUEST. Note that NLM_F_MATCH is propably not implemented, and you will get all routes/links/addresses that are specified - regardless the attributes / values in family specific struct. Flags are more accurately introduced for example in man pages.

3&4. identifying the request/responce pair. Typicaly you should set pid to something derived from process/thread id. I use
(pthread_self() < < 16|getpid());
Kernel's responces have pid set to 0.

You should propably use different sequence number for each request - that way you can associate correct responces with correct requests. I usually have global variable, and issue atomic incrementation operation to it when getting new sequence ID.

I also like to know if my request succeeded. Thus I always use NLM_F_ACK in other but GET requests. This will make kernel to send us reply with zero errorcode if requested operation succeeded. With GET requests I can expect getting reply anyways.

(The kernel's ACK message for successfull operation with NLM_F_ACK flag will be like:

struct nlmsghdr followed by struct nlmsgerr
where nlmsg_type == NLMSG_ERROR and error member of struct nlmsgerr is set to 0.
)

Family specific structs follow after the nlmsghdr. They're form:

Addresses:
struct ifaddrmsg

struct ifaddrmsg
{
    __u8        ifa_family;
    __u8        ifa_prefixlen; /* The prefix length        */
    __u8        ifa_flags; /* Flags            */
    __u8        ifa_scope; /* Address scope        */
    __u32     ifa_index; /* Link index         */
};

Family being address family (typically AF_INET / AF_INET6)
prefixlen telling the mask bits. (see explanation of mask in routes above. This is used when we for example add new ip-address to existing interface - so called alias IP address. Then the mask tells what network lies behind the door for outgoing packets!)
flags

/* ifa_flags */
IFA_F_SECONDARY
IFA_F_TEMPORARY

IFA_F_NODAD
IFA_F_OPTIMISTIC
IFA_F_DADFAILED
IFA_F_HOMEADDRESS
IFA_F_DEPRECATED
IFA_F_TENTATIVE
IFA_F_PERMANENT

whose meanings are quite unknown for me - I haven't really needed any of these.

ifa_scope
I've successfully used 0 as scope for all requests - I have no idea what the scope really is with addresses.

ifa_index (index number of the interface this address is bound, see man pages for
unsigned if_nametoindex(const char *ifname);

links(interfaces)
struct ifinfomsg

struct ifinfomsg
{
    unsigned char ifi_family;
    unsigned char __ifi_pad;
    unsigned short ifi_type;     /* ARPHRD_* */
    int     ifi_index;     /* Link index */
    unsigned    ifi_flags;     /* IFF_* flags */
    unsigned    ifi_change;     /* IFF_* change mask */
};

When I created new link (in order to do a VLAN interface - that was the only thing I used NEWLINK request for), I at first banged my head to wall by creating RTM_NEWLINK request, where ifinfomsg struct had fields filled... I always received error responces from kernel. When I finally compiled my own kernel, with lots of info prints included, I learned that the ifinfomsg struct should not contain much of values with NEWLINK request - or request would be discarded. Then I only filled ifi_change field with 0xffffffff and left rest of the fields to zero. Then I added attributes which specified the link to be VLAN interface. I'll show the attributes for requests later... This allowed me to create a new link.

After link was created, I used RTM_SETLINK request to set the link state to IFF_UP. For this step I filled ifi_index with index of newly created interface, and set flags to contain IFF_UP bit. ifi_change should still be 0xffffffff (at least according to man pages), since it is reserved for future use. I was lazy. I simply set the ifi_flags to be IFF_UP. I guess this approach may be hazardous (not sure though). It may be the interface has some other flags set up, and simply setting flags to be IFF_UP without knowing the original state may make us to lose some information. I do not know since I have not studied these flags so thoroughly. I just guess that better way could be first reading the flags, and then or (|) the IFF_UP with existing bits. However I guess making this so, that possible state changes between reading and setting the flags would be noticed - is hard. I just decided that there is no flags which I could accidentally zero... It is easier to not know ;)

routes
struct rtmsg

struct rtmsg
{
    unsigned char     rtm_family;
    unsigned char     rtm_dst_len;
    unsigned char     rtm_src_len;
    unsigned char     rtm_tos;

    unsigned char     rtm_table; /* Routing table id */
    unsigned char     rtm_protocol; /* Routing protocol; see below */
    unsigned char     rtm_scope; /* See below */
    unsigned char     rtm_type; /* See below    */

    unsigned        rtm_flags;
};

where types can be

RTN_UNSPEC,
RTN_UNICAST, /* Gateway or direct route */
RTN_LOCAL, /* Accept locally */
RTN_BROADCAST, /* Accept locally as broadcast,
send as broadcast */
RTN_ANYCAST, /* Accept locally as broadcast,
but send as unicast */
RTN_MULTICAST, /* Multicast route */
RTN_BLACKHOLE, /* Drop */
RTN_UNREACHABLE, /* Destination is unreachable */
RTN_PROHIBIT, /* Administratively prohibited */
RTN_THROW, /* Not in this table */
RTN_NAT, /* Translate this address */
RTN_XRESOLVE, /* Use external resolver */

For normal routes you propably want to use RTN_UNICAST. Although knowing the amount of spam flowing through the ethernet wires... Well, I feel the RTN_BLACKHOLE is tempting ;)

defined protocols:

#define RTPROT_UNSPEC 0
#define RTPROT_REDIRECT 1 /* Route installed by ICMP redirects;
not used by current IPv4 */
#define RTPROT_KERNEL 2 /* Route installed by kernel */
#define RTPROT_BOOT 3 /* Route installed during boot */
#define RTPROT_STATIC 4 /* Route installed by administrator */

/* Values of protocol > = RTPROT_STATIC are not interpreted by kernel;
they are just passed from user and back as is.
It will be used by hypothetical multiple routing daemons.
Note that protocol values should be standardized in order to
avoid conflicts.
*/

#define RTPROT_GATED 8 /* Apparently, GateD */
#define RTPROT_RA 9 /* RDISC/ND router advertisements */
#define RTPROT_MRT 10 /* Merit MRT */
#define RTPROT_ZEBRA 11 /* Zebra */
#define RTPROT_BIRD 12 /* BIRD */
#define RTPROT_DNROUTED 13 /* DECnet routing daemon */
#define RTPROT_XORP 14 /* XORP */
#define RTPROT_NTK 15 /* Netsukuku */
#define RTPROT_DHCP 16 /* DHCP client */

I used RTPROT_STATIC, which corresponds the situation where user adds a route using "ip route add" command.

possible scopes:

RT_SCOPE_UNIVERSE=0,
/* User defined values */
RT_SCOPE_SITE=200,
RT_SCOPE_LINK=253,
RT_SCOPE_HOST=254,
RT_SCOPE_NOWHERE=255

Well, for route which is not meant to stay inside some known system, it is feasible to use RT_SCOPE_UNIVERSE.

/* rtm_flags */

#define RTM_F_NOTIFY 0x100 /* Notify user of route change */
#define RTM_F_CLONED 0x200 /* This route is cloned */
#define RTM_F_EQUALIZE 0x400 /* Multipath equalizer: NI */
#define RTM_F_PREFIX 0x800 /* Prefix addresses */

I used no flags, eg set the flags to zero.

So at this spot, our request consists of netlink header telling the type of our request (address, link or route) and the type specific structure behind this netlink header. Now, so that it wouldn't be so simple, we'll introduce some more dynamic data :]

attributes

Each request supports variable amount of attributes which will further describe the address/route/interface being created (changed/deleted). Attributes are data prepended with struct rtattr structure. It looks like:

struct rtattr
{
    unsigned short rta_len;
    unsigned short rta_type;
};

After this structure we have actual attribute data, which of course depends on rta_type, and has lenght
rta_len - sizeof(struct rtattr). Aligned to 4 bytes :]

Well, as a C coder can imagine, this level of dynamic structures can be quite challenging to handle. And as can be guessed, there is macros written to ease the attribute handling. (I also wrote simpe functions to add attributes to the request.)

Macros to handle all this are:

RTA_ALIGN(len)
RTA_OK(rta,len)
RTA_NEXT(rta,attrlen)
RTA_LENGTH(len)
RTA_DATA(rta)
RTA_PAYLOAD(rta)

RTA_ALIGN(len) Rounds the given lenght to next alignment boundary. Eg. with typical 4 byte alignment

RTA_ALIGN(1) would return 4 as would RTA_ALIGN(4). RTA_ALIGN(5) would return 8 and so on.
RTA_OK(rta,len) can be used to see if the attribute is Ok. It is quite common to use RTA_NEXT with RTA_OK to parse incoming messages. Eg, with RTA_NEXT we get the next attribute, and with RTA_OK we check attribute is ok to be inspected. len passed to these macros is originally the lenght of attribute buffer. Each call to RTA_NEXT shall update the lenght.
RTA_LENGHT(len) returns the lenght which is required to store attribute which data is len bytes. Eg, RTA_LENGHT adds the lenght of rtattr header to the lenght of the data, and adds required padding bytes to get the data correctly aligned.
RTA_DATA(rta) returns pointer to the beginning of the data in attribute rta.
RTA_PAYLOAD(rta) returns the lenght of the data payload in attribute.

Sounds confusing? Well, don't be worried. I'll later show you a few functions to handle the attributes...

Now we can get to the confusing point...
With VLAN interface creation I hit my head to wall. I did some googling. Then I googled more. Finally I had googled my *** off. "VLAN interface netlink sockets", "RTM_NEWLINK VLAN". "Create VLAN via netlink". "VLAN netlink attribute". It basically took all my googling skills, as well as jump into kernel sources to finally find it. What kind of magical attribute allows one to specify VLAN interface. Google gave me some hints that such an attribute exists, and finally I found it...

nested attributes. Nested attributes are attributes, which have attributes inside. That's it. One of the attributes to create a VLAN interface is one of these. But I'll show that later.

Attributes supported by different families are:

Addresses:

IFA_UNSPEC,
IFA_ADDRESS,
IFA_LOCAL,
IFA_LABEL,
IFA_BROADCAST,
IFA_ANYCAST,
IFA_CACHEINFO,
IFA_MULTICAST,

Routes:

RTA_UNSPEC,
RTA_DST,
RTA_SRC,
RTA_IIF,
RTA_OIF,
RTA_GATEWAY,
RTA_PRIORITY,
RTA_PREFSRC,
RTA_METRICS,
RTA_MULTIPATH,
RTA_PROTOINFO, /* no longer used */
RTA_FLOW,
RTA_CACHEINFO,
RTA_SESSION, /* no longer used */
RTA_MP_ALGO, /* no longer used */
RTA_TABLE,

Interfaces (links):

IFLA_UNSPEC,
IFLA_ADDRESS,
IFLA_BROADCAST,
IFLA_IFNAME,
IFLA_MTU,
IFLA_LINK,
IFLA_QDISC,
IFLA_STATS,
IFLA_COST,
IFLA_COST
IFLA_PRIORITY,
IFLA_PRIORITY
IFLA_MASTER,
IFLA_MASTER
IFLA_WIRELESS, /* Wireless Extension event - see wireless.h */
IFLA_WIRELESS
IFLA_PROTINFO, /* Protocol specific information for a link */
IFLA_PROTINFO
IFLA_TXQLEN,
IFLA_TXQLEN
IFLA_MAP,
IFLA_MAP
IFLA_WEIGHT,
IFLA_WEIGHT
IFLA_OPERSTATE,
IFLA_LINKMODE,
IFLA_LINKINFO,
IFLA_LINKINFO
IFLA_NET_NS_PID,

Some of these attributes are documented in rtnetlink man pages at man section 7 - some aren't...

Now just to ease the pain for those who struggle with the VLAN interface setup... I managed to do it with following attributes:

IFLA_LINK, data size of int, contains the real interface which this VLAN interface uses beneath.
IFLA_IFNAME, name of the new VLAN interface, lenght depends on lenght of the name you give. I used naming convention < original_interface > . < vlan Id >

nested attribute IFLA_LINKINFO, containing:
attribute IFLA_INFO_KIND, data is string 'vlan', lenght being the lenght of the string + padding.
another nested attribute IFLA_INFO_DATA, containing:
attribute IFLA_VLAN_ID, which data is the vlan id.

Uh oh... Sounds confusing, right? Nested attribute containing an attribute and another nested attribute which contains an attribute... Oh joy, occasionally I tend to believe that programmers have been REALLY drunk when getting their ideas... (http://xkcd.com/323/)

Anyways, I'll show you the code which I used to handle this:

#define NLMSG_BOTTOM(nlmsg) ((struct rtattr *)(((void *)(nlmsg)) + NLMSG_ALIGN((nlmsg)- > nlmsg_len)))

static int addAttr(struct nlmsghdr *nl_req, int attrlabel, const void *data, int datalen)
{
    struct rtattr *attr=NLMSG_BOTTOM(nl_req));
    unsigned int attrlen=RTA_LENGTH(datalen); /* sizeof(struct rtattr) + datalen + align */
    if(NULL==nl_req || (datalen > 0 && NULL==data))
    {
        printf("NULL arg detected!");
        return -1;
    }
    attr- > rta_type=attrlabel;
    attr- > rta_len=attrlen;
    memcpy(RTA_DATA(attr),data,datalen);

    nl_req- > nlmsg_len=NLMSG_ALIGN(nl_req- > nlmsg_len)+RTA_ALIGN(attrlen);
    return 0;
}

static struct rtattr * addNestedAttr(struct nlmsghdr *nl_req, int attrlabel)
{
    struct rtattr *nested = NLMSG_BOTTOM(nl_req);

    if(!addAttr(nl_req, attrlabel, NULL, 0))
        return nested;
    return NULL;
}

static void endNestedAttr(struct nlmsghdr *nl_req, struct rtattr *nested)
{
    nested- > rta_len = (void *)NLMSG_BOTTOM(nl_req) - (void *)nested;
}

/* ...snip - Add attributes to the nlmsg msg */

        struct rtattr *attr1, *attr2;
        if(addAttr(msg,IFLA_LINK,&orig_ifindex,sizeof(int)))
        {
            printf("IFLA_LINK %d adding as rtattr to req failed!",orig_ifindex);
            retval=-1;
        }
        else if(addAttr(msg,IFLA_IFNAME,ifname,strlen(ifname)))
        {
            printf("IFLA_IFNAME %s adding as rtattr to req failed!",ifname);
            retval=-1;
        }
        else if(NULL==(attr1=addNestedAttr(msg,IFLA_LINKINFO)))
        {
            printf("addNestedAttr IFLA_LINKINFO FAILED!");
            retval=-1;
        }
        else if(addAttr(msg,IFLA_INFO_KIND,"vlan", strlen("vlan")))
        {
            printf("IFLA_INFO_KIND \"vlan\" adding FAILED!");
         retval=-1;
        }
        else if(NULL==(attr2=addNestedAttr(msg,IFLA_INFO_DATA)))
        {
            printf("addNestedAttr IFLA_INFO_DATA FAILED!");
            retval=-1;
        }
        else if(addAttr(msg,IFLA_VLAN_ID,&vlanid,sizeof(unsigned short)))
        {
            printf("IFLA_VLAN_ID %hu adding as rtattr to req failed!",vlanid);
            retval=-1;
        }
        else
        {
            endNestedAttr(msg,attr2);
            endNestedAttr(msg,attr1);
            printf("VLAN ID %hu, orig ifindex %d and new ifname %s added as attrs",vlanid,orig_ifindex,ifname);
        }

This code assumes that the lenght of the nlmsg (in struct nlmsghdr) is summed up during message creation. Eg, that when the attributes are added, the lenght in nlmsghdr is updated to be the lenght of message constructed this far. Attribute addition relies upon this lenght, when adding new attributes && updates this lenght when attributes are added. (see the macro NLMSG_BOTTOM() )

Now I guess I am approaching the end of this short introduction. I'll however show you something from where you can get the idea of how messages are sent and received, and how attributes can be parsed. In order to sum up the full horror of this interface (dynamic = > flexible and generic = > terribly hard to use) I have to mention something about receiving the messages...

Messages will arrive from socket the socket. They will be placed in buffer you gave. You need to be prepared to handle:

1. reply where you have specified to short buffer = > you'll get reply with MSG_TRUNC bit set.
2. Reply where you have multiple nlmsgs in one received buffer. In that case, there's NLM_F_MULTI flag set. In that case last message shall have NLM_F_DONE set.

Eg. You may end up having a buffer, where you have variable amount of nlmsgs, wach containing different sized/type struct and variable amount of possibly nested attributes after that... Some (pseudo)code as an example...

struct sockaddr_nl kernproc;
struct msghdr msg;
struct nlmsghdr *netlinkresp;
struct iovec iov;

memset(&msg,0,sizeof(msg));

memset(&kernproc,0,sizeof(kernproc));
memset(&iov,0,sizeof(iov));

kernproc.nl_family = AF_NETLINK;

msg.msg_name=(void *)&kernproc;
msg.msg_namelen=sizeof(kernproc);

netlinkresp= < buffer allocated for responce > ;

/* Add NLMSG_F_ACK if no reply is to be expected othervice */

iov.iov_base=(void *)netlinkresp;
iov.iov_len= < nlmsg_len > ;

msg.msg_iov=&iov;
msg.msg_iovlen = 1; /* only one iov struct abowe */
iov.iov_len= < size of the resp buffer >

retry:
    retval=recvmsg(sock, &msg, 0);
    if(0 > =retval)
    {
        if(errno==EINTR)
            goto retry;
        else
        {
            printf("Error when receiving from netlink sock!");
            /* handle error */
        }
    }
    else
    {
        /* ...reply received */
    }

So
1. Check lenght of received message from return value of recv. Never exeed it.
2. Check the msghdr (not nlmsghdr) to see the message was not truncated.
if(msg.msg_flags&MSG_TRUNC)
... allocate more space for resp and retry...
3. strore pointer to the nlmsghdr message header.
4. start a loop and use NLMSG_OK() to see message is ok. If msg is not OK, then you have nothing to handle.
5. Check the type of nlmsg, and lenght. If lenght is greater or equal to type specific header, then use
NLMSG_DATA to get the actual message. Cast and store a pointer to this.
5. Check the received message for information you longed for. If whole nlmsg lenght is still not handled, then there probably are attributes.
6. Obtain ptr to first attribute by adding size of message specific struct to the NLMSG_DATA().
7. start a loop and check the attribute with RTA_OK() If RTA_OK fails go to step 9
8. check attribute type, and data according to type & len.
obtain next attr with RTA_NEXT - > end loop and go back to step 7.
9. when last attribute is handled, check if NLMSG had NLM_F_MULTI set, and at least not NLMSG_DONE was specified, then get next NLMSG with NLMSG_NEXT() and loop again from step 4

Sending a message is done using same generic iovec mechanism. Eg:

    struct sockaddr_nl kernproc;
    struct msghdr msg;
    struct nlmsghdr *netlinkreq;
    struct iovec iov;

    memset(&msg,0,sizeof(msg));this- > mypid

    memset(&kernproc,0,sizeof(kernproc));
    memset(&iov,0,sizeof(iov));

    kernproc.nl_family = AF_NETLINK;

    msg.msg_name=(void *)&kernproc;
    msg.msg_namelen=sizeof(kernproc);

    netlinkreq= < pointer to allocated and filled message request > ;

    netlinkreq- > nlmsg_pid= (pthread_self() < < 16|getpid());
    netlinkreq- > nlmsg_seq= atomicallyIncrementSeqId(seqid);

    #ifdef debug
    debugprint_msg(netlinkreq);
    #endif

    iov.iov_base=(void *)netlinkreq;
    iov.iov_len=netlinkreq- > nlmsg_len;

    msg.msg_iov=&iov;
    msg.msg_iovlen = 1; /* only one iov struct abowe */

    retval =sendmsg(sock,&msg,0);
    if(retval < =0)
    {
        printf("sendmsg() FAILED!");

    }
    return retval;

the debugprint function I have used contains following code:

    if(netlinkreq- > nlmsg_flags & NLM_F_ACK)
    {
        printf("Msg contains f_ack!");
    }
    if(!NLMSG_OK(netlinkreq,netlinkreq- > nlmsg_len))
    {
        printf("Looks like we're sending invalid nlmsg!! NLMSG_OK() == false at send!");
    }
    else
    {
        printf
        (
            "sending NLMSG: len %u, type %hu, flags %hu, seq %u pid %u",
            netlinkreq- > nlmsg_len,
            netlinkreq- > nlmsg_type,
            netlinkreq- > nlmsg_flags,
            netlinkreq- > nlmsg_seq,
            netlinkreq- > nlmsg_pid
        );
        switch(netlinkreq- > nlmsg_type)
        {
            case RTM_NEWROUTE:
            case RTM_DELROUTE:
            case RTM_GETROUTE:
            {
                printf
                (
                    "Req is route req (new %u, del %u, get %u)",
                    RTM_NEWROUTE,
                    RTM_DELROUTE,
                    RTM_GETROUTE
                );
                printf
                (
                    "family %u, dstlen %u, srclen %u, tos %u, table %u, proto %u, scope %u, type %u, flags %u",
                    (unsigned int)((struct rtmsg *) NLMSG_DATA(netlinkreq) )- > rtm_family,
                    (unsigned int)((struct rtmsg *) NLMSG_DATA(netlinkreq) )- > rtm_dst_len,
                    (unsigned int)((struct rtmsg *) NLMSG_DATA(netlinkreq) )- > rtm_src_len,
                    (unsigned int)((struct rtmsg *) NLMSG_DATA(netlinkreq) )- > rtm_tos,
                    (unsigned int)((struct rtmsg *) NLMSG_DATA(netlinkreq) )- > rtm_table,
                    (unsigned int)((struct rtmsg *) NLMSG_DATA(netlinkreq) )- > rtm_protocol,
                    (unsigned int)((struct rtmsg *) NLMSG_DATA(netlinkreq) )- > rtm_scope,
                    (unsigned int)((struct rtmsg *) NLMSG_DATA(netlinkreq) )- > rtm_type,
                    (unsigned int)((struct rtmsg *) NLMSG_DATA(netlinkreq) )- > rtm_flags
                );
                {
                    int len=netlinkreq- > nlmsg_len;
                    struct rtattr *at=(struct rtattr *)((char *)NLMSG_DATA(netlinkreq)+sizeof(struct rtmsg));
                    while(NULL!=at && RTA_OK(at,len))
                    {
                        char tmp[100];
                        switch(at- > rta_type)
                        {
                            case RTA_DST:
                                printf
                                (
                                    "dst is set to %s",
                                    inet_ntop
                                    (
                                        (at- > rta_len > 8)?AF_INET6:AF_INET,
                                        RTA_DATA(at),
                                        tmp,
                                        100
                                    )
                                );
                                break;
                            case RTA_SRC:
                                 printf
                                (
                                    "src is set to %s",
                                    inet_ntop
                                    (
                                        (at- > rta_len > 8)?AF_INET6:AF_INET,
                                        RTA_DATA(at),
                                        tmp,
                                        100
                                    )
                                );
                                break;
                            case RTA_GATEWAY:
                                printf
                                (
                                    "gw is set to %s",
                                    inet_ntop
                                    (
                                        (at- > rta_len > 8)?AF_INET6:AF_INET,
                                        RTA_DATA(at),
                                        tmp,
                                        100
                                    )
                                );
                                break;
                            case RTA_OIF:
                                printf
                                (
                                    "OIF is set to %u",
                                    *(unsigned int *)RTA_DATA(at)
                                );
                                break;
                            case RTA_PRIORITY:
                                printf
                                (
                                    "Priority is set to %u",
                                    *(unsigned int *)RTA_DATA(at)
                                );
                                break;
                            default:
                                printf("rta_type %u, len %u",at- > rta_type,at- > rta_len);
                                break;
                        }
                        at=RTA_NEXT(at,len);
                    }
                }
                break;
            }
            case RTM_NEWADDR:
            case RTM_GETADDR:
            case RTM_DELADDR:
                printf
                (
                    "Req is ADDR req (new %u, del %u, get %u)",
                    RTM_NEWADDR,
                    RTM_DELADDR,
                    RTM_GETADDR
                );
                printf
                (
                    "ifa_family %u, ifa_prefixlen %u, ifa_flags %u, ifa_scope %u, ifa_index %d",
                    (unsigned int)((struct ifaddrmsg *) NLMSG_DATA(netlinkreq) )- > ifa_family,
                    (unsigned int)((struct ifaddrmsg *) NLMSG_DATA(netlinkreq) )- > ifa_prefixlen,
                    (unsigned int)((struct ifaddrmsg *) NLMSG_DATA(netlinkreq) )- > ifa_flags,
                    (unsigned int)((struct ifaddrmsg *) NLMSG_DATA(netlinkreq) )- > ifa_scope,
                    (int)((struct ifaddrmsg *) NLMSG_DATA(netlinkreq) )- > ifa_index
                );
                {
                    int len=netlinkreq- > nlmsg_len;
                    struct rtattr *at=(struct rtattr *)((char *)NLMSG_DATA(netlinkreq)+sizeof(struct ifaddrmsg));
                    while(NULL!=at && RTA_OK(at,len))
                    {
                        char tmp[100];
                        switch(at- > rta_type)
                        {
                            case IFA_ADDRESS:
                                printf
                                (
                                    "IFA_ADDRESS is set to %s",
                                    inet_ntop
                                    (
                                        (at- > rta_len > 8)?AF_INET6:AF_INET,
                                        RTA_DATA(at),
                                        tmp,
                                        100
                                    )
                                );
                                break;
                            case IFA_LOCAL:
                                printf
                                (
                                    "IFA_LOCAL is set to %s",
                                    inet_ntop
                                    (
                                        (at- > rta_len > 8)?AF_INET6:AF_INET,
                                        RTA_DATA(at),
                                        tmp,
                                        100
                                    )
                                );
                                break;
                            case IFA_BROADCAST:
                                printf
                                (
                                    "IFA_BROADCAST is set to %s",
                                    inet_ntop
                                    (
                                        (at- > rta_len > 8)?AF_INET6:AF_INET,
                                        RTA_DATA(at),
                                        tmp,
                                        100
                                    )
                                );
                                break;
                            case IFA_LABEL:
                                printf
                                (
                                    "IFA_LABEL is set to '%s'",
                                    (char *)RTA_DATA(at)
                                );
                                break;
                            case IFA_ANYCAST:
                                printf
                                (
                                    "IFA_ANYCAST is set to %s",
                                    inet_ntop
                                    (
                                        (at- > rta_len > 8)?AF_INET6:AF_INET,
                                        RTA_DATA(at),
                                        tmp,
                                        100
                                    )
                                );
                                break;
                            default:
                                printf("rta_type %u, len %u",at- > rta_type,at- > rta_len);
                                break;
                        }
                        at=RTA_NEXT(at,len);
                    }
                }
                break;
.
./* You could add other type of requests here too */
        }
.
.
.

There you see how browsing through one message contents can be done. One loop more with some extra checks, and NLMSG_NEXT and NLMSG_OK macros would allow you to go through received nlmsgs.

Anyways, I guess I can end my brief introduction to netlink sockets here. Remeber to check the
man 7 rtnetlink and man 3 netlink for more information. Here is just some pieces like nested attributes and VLAN interface setup explained... Maybe you do not need to bang your head to wall just as much as I had to do. :]

Oh, and if you find this post or the code examples usefull, please do leave me a note :) And as allways, code presented here can be used/modified to suit your purposes, as long as you either drop me a note in Mazziesaccount@gmail.com or comment in here, and mention original author (me, Maz) && this site in your codes - especially if you publish them to be public somewhere.

Have fun!

Download links in this blog.

2010-07-07T10:52:00.004+03:00

Hi dee Ho again!

Times change. I iced my bot project (lack of time etc. etc...). The svn repository which is often referred here got shot down. The latest sources should still be available at

http://xp-dev.com/svn/MazBotV4/trunk/

Hi dee Ho peeps!

I have often given you a link to svn repository which is locadted on blackdiam.net/svn. Many of the sources for my C stuff are located there.

I did some refactoring in the repository, and most of the links got dated. If you encounter non working link to blackdiam.net svn repository in this blog, following corrective actions can be done:

Old link to dev repo base was:
http://blackdiam.net/svn/MazBot/

New link to dev repo base is
http://blackdiam.net/svn/MazBot/trunc/
And more stable environment is in
http://blackdiam.net/svn/MazBot/releases/0.2.1/

Hopefully you find what you're looking for :)

-Maz

More tools to write better C

2010-06-29T16:49:00.005+03:00

EDIT: 15.09.2011 Another SVN working now. Sources can be found from

http://xp-dev.com/svn/MazBotV4/trunk/generic/src/

files:

MbotBitset.c
MbotBitset.h
MbotPackedArray.c
MbotPackedArray.h

are covered here.

Oh my oh my. It has been ages since I last annoyed you my dear readers. Well, I figured I could now do that again and share some bits (pun intended) which may come handy.

Bitset:

With bitset I mean a simple way of decreasing memory usage while storing information. There's plenty of situations where we need to store some state for multiple things. A simple (and very much schooly) example could be student's course proceedings in a school. Eg, let's imagine a school where we have 10000 students. Each student needs to pass set of courses, and these courses are either passed or not passed. Now, we could assign an ID-number for each student, starting from 0, and ending up to 9999. (Gosh how fast these classes grow nowadays. What does this tell about us parents...).

Now first idea how to simply store this status is to create an array for each course, index it with student IDs, and set the element of array to 1 if student has passed the course, or 0 if he/she has not. Eg. we might end up in a solution like:

int biology1[10000];

int has_student_passed_biology1(int student_id)
{
if(student_id<0 || student_id>10000)
return -1;
else
return biology1[student_id];
}

Lets ponder a lil while. An integer on x86 machine takes up 32 bits of space. How many bits do we need? Naturally, we only need 1 bit, (student can have passed the course (1) or not passed (0). (Of course there is cases when we do not need even this many bits, for example the introduction to quantum mechanics was such a course for 1.st years physics students... No one passed, but let's ignore that now :D) So for each course we end up wasting 31 bits of memory/student. This is 310000 bits (we had 10000 students), Eg. roughly 37.8 kilobytes. Well, when we think of modern PCs, this might not be significant. But on the other hand, when we think of all the possible places where we can do this wasting... Well, have you ever wondered M$ product's memory usage? :D And if we think of embedded systems with limited amount of memory.........

To tackle this problem we could change ints to chars. That would divide the loss by 4. But it would still be a waste. Especially in already bloated world of SW.

So my (and many other's) solution is a bitset. In a simplest form it is just a way to read/write state of single bit. And basically this is what my bitset does. Eg, at the beginning, you initialize my bitset by telling how many bits you need, and later you set/unset/get the state of N.th bit of bitset.

My bitset implementation can be obtained from my bot project's repository:
http://xp-dev.com/svn/MazBotV4/trunk/generic/src/
The files are MbotBitset.h and MbotBitset.c

NOTE! I've not tested this bit set on 64 bit architectures, and it may fail there. (It may also work though). If you test it, please report the results in this blog for example (as a comment), or by email (Mazziesaccount@gmail.com).

Packed Array:

Well, if we continue playing with thoughts of students, we can go back in good ol' times where grades from courses at university had only 4 possibilities. Failed, 1,2 or 3. Now if we think of storing these numbers, we note following:
1. bitset cannot really be easily used, since it only allows 2 states / Id. (since it only uses 1 bit to represent the value, but representing 4 states requires 2 bits, 00, 01, 10, 11)
2. If we use char array, only 2 of 8 bits is used / student.

To tackle this one I wrote a packed array.

Basically my packed array just calculates amount of bits needed to represent the largest value (told at array initialization), and then allocates the amount of memory that is needed for array with X members (also told at initialization). Then it splits the memory to slots and allows writing/reading values stored in N bits wide slots.

My packed array implementation can be found from same place as the bitset, files MbotPackedArray.h and MbotPackedArray.c

NOTE! Packed array has not been tested on 64-bit machine (and I assume it will not work there). Also 32 bit environment has not been so throughoutly tested, but at least I've not noticed that there is bugs left...
Anyways, I'd be gratefull if you reported results of all experiments with the packed array / bitset.

(To my mail (Mazziesaccount@gmail.com) Bugs, success stories, improvements...
or in this blog as a comment.Bugs, success stories, improvements...)

New C project started...

2009-07-22T23:50:00.009+03:00

I abandoned the foreverlasting MazPong project (for a while?). Reason you may ask. Because it grew into mess. Try writing project which gets bloated since you just write things you notice are handy, and forget to make small, compact TESTED things - and document them as you go on. Well, it may work if you can work continuously, so that you can remember the big picture throughout the project. That's not the case with MazPong. I started writing it at 2005 (or so), and it has been completely redesigned three times while going on... And I have never worked 5 days in a row, more like two days of work - two - six months break and so on... Yet I never started from scratch...

So it became a jungle of wery different pieces of code, some of which are dublicated (because I forgot I did something last year)...

So it is on hold. I do not say it is totally forgotten - it is foreverlasting project for god's sake!

Now.. I started another project, where I have a small motivator - a friend of mine might have use for it :D At this time it's not a game, it will be yet another IRC bot called MazBotV4 (working as an IRC client). I have always had obsession with IRC bots. First I wrote with mirc scripts, and it terrorized #oldgames channel related to old computer games. Then I wrote a bot using Eggdrop and TCL - with little success. TCL was not for me. V3 was written in C++, but I grew tired with it since I had no use for it. Now V4 will be pure plain C.

This bot is meant to give easy interface for doing some tasks based on text file configurations - while it also offers almost unlimited possibilities to someone familiar with C - I plan writing a nice callback interface, where people with limited skills can add their own functions to be executed when an event occurs on IRC channel. And finally, I plan to write an interface, where bot user's can specify network addresses where bot connects on certain events. Idea is that this way one can send/retrieve information from web servers using the bot.

- no. I am not going to write HTML parser in bot. period. It will be user's responcebility to write some scripts to web server - which can provide the data in simple format to bot. But of course, if someone wants to add html parser in the bot... ;) And other help will propably be appreciated (at least at some point - if I manage to keep this going. Oh yes, if anyone familiar with GNU Makefiles wants to improve the make scripts I have written [No autotools - I dislike them!!] and still keep them simple - welcome to take a peek and send suggestions!)

Oh, the title of this post should link to source repository.
And in case you want to see stable version, here:
http://xp-dev.com/svn/Mazzie-mazbot/

Oh, another bug found from Cexplode- put up a "correction release" v0.1-1

And finally, I have planned to use my Cexplode function set in IRC parser - and I have added some mysterious functions into it... So there's enchanced (and at the moment quite untested) version available. I just changed the file name to helpers.h and helpers.c (in generic/src folder). Also test/usage example is present in test/src folder. (This info is provided since according to my tracker, ~60% of people visiting this blog are looking for explode function for C)

License terms are same as before - use/modify as pleases you - but let me know it by email or via a comment in this blog. But if you redistribute it, mention the original author (me) and this website (or the finnish equivalent: http://c-ohjelmoijanajatuksia.blogspot.com/ )

All in all, happy summer, remember your caps - don't fry your brains :)

-Matti

Example of object oriented C

2009-05-06T17:58:00.003+03:00

I have previously written an entry stating that c can be used as object oriented language. Now I decided to write an example demostrating this. So here it is, a program drawing lottery numbers, written in ansi c, using object oriented approach.

I know the code looks horrible. It is. Object oriented code often is. And this example could have been written so much easier with plain procedural c. But this shows how basic requirements of object orientation are fullfilled. Lottery machine is (virtual) base class, and basic & viking lottery are derived classes. If i had more enthusiasm, i would have written joker class too.--

And as an explanation, in finland we have two different types of lottery, viking lotto and regular lotto. Theres different amount of numbers drawn, but i cant remember the accurate amounts. Theres also so called joker, but i cant remember the amount of numbers in it either.

And as a final note, i know rand() is not perfect function to use, but since this was not meant to be real lottery machine, i could not care less :)

So here's the lottery source. Compile with

gcc -Wall -o lotteryExe lotto.c lottoTest.c regularlotto.c vikinglotto.c

What I learned today.

2009-04-02T22:08:00.003+03:00

I've noticed that SW designer's job is constant learning. If you stop learning, tou stop advancing. For a newish guy on the field like me, most of the learning is related to existing systems - but even the old chaps need to keep making progress, new technologies keep coming, and falling back means giving advantage to others.

So, I guess I'll start writing down this "What I learned today" post, in which I try to add new comments ehen I have learned something new. Just short sentences, with no long background, but with some information that was new to me.

But since I started this now, not at my first day in the job, I'll write down a couple of things in this post too :)

Last December.

Untill the really latest glibc versions, there's a bug in glibc's timer_create() function implementation, which may cause a crash under certain conditions. (When timer expires, attempt to read the internal list where timers are kept in glibc, may access to invalid memory location)

A few months ago.

Helgrind is yet another great tool built on valgrind.
(It can be used to detect synchronization issues in multithreaded applications - although it has it's limitations)

A few weeks ago.

In some glibc versions for powerpc have a bug, which makes valgrind yelling a lot of false alarms!

A few days ago.

pthread's stack size cannot be easily (or portably) changed after the thread is created.

(It can be set during the thread initialization. It is not really common to have real need for changing the default stack size [actually this sounds like terribly failed design], but there may be cases where one needs to create many [read some hundreds of] threads with limited virtual address space [*shrugs*]. On linux at least, the real memory is not used more than really required, but virtual address space will be reserved for whole threads stack (no matter how much the thread really needs the stack).

Today: (02.04.2009)

In most linuxes, there's a program called "yes" included, which will repeatedly print the text given as argument. My co-worker told that to me today, and I used it to generate some CPU load when I needed to do some performance analysis. I used:

yes Matti > /dev/null
renice -19

Linux virtuall address space.

2009-03-25T22:08:00.003+02:00

Today I spotted a question about ability to read process data after process freeds the memory it has used. (It was on a one of the best C programming related forums I know, at CBoard )

So I started writing an answer, which accidentaly became lenghty. After I first read it I felt that there's no hope it could clarify a thing to anyone. But after a few beers, I started to feel pretty damn proud about that babbling. So I decided to copy it to here too. So enjoy.

About the reading memory after freeing it etc. I'll try to shed some light on this topic, and I hope if the more experienced guys spot a problem in my explanation, they will further teach both me and you

I will be talking in linux point of view, but I assume that most modern systems do it in somewhat similar manner. Of course there's some realtime systems (like OSE), in which all memory is available to all brogram blocks (Eg. processes), but I assume you're talking about some desktop system.

So when we simplify enough we can say following. . Most modern OSes do map the real physical memory into virtual memory addresses, and give to each process their own virtual address space. Actually, in most modern computer systems, we have a hardware memory management unit which handles conversions from virtual addresses to physical memory and vice versa. So basically, at user space (which is basically all applications that aren't either part of the kernel, or a kernel module) it is not possible to access straight to hardware (memory or other hardware). Hardware is invisible for userspace processes, it can only be accessed via drivers written in kernel space. (drivers offer interface for user space applications). [[There is a workaround, but let's not mess with it now]].

So process cannot directly access to RAM.

When we launch a process, certain virtual address space is given to it (and certain physical ran corresponding to it, but the process has no means to get direct HW addresses). The process has no access outside of this sandbox. When we launch another process, it will have it's own virtual address space, and it's own corresponding physical ram. If process is running out of ram, kernel has means to increase it.

If another process tries to access outside of it's sandbox, Eg. requests reading from / writing to location which is not in it's address space, MMU will wake up the kernel, which will check out what is happening. If request was illegal, kernel will generate signal SIGSEGV, and send it to your program. => segmentation fault.

However physical ram is not cleaned, and the amount assigned to each process may change. There's also possibility of swapping, where some memory pages are written temporarily to hard disk, to free up some ram for other purposes.

So basically, it indeed is possible that there is leftowers of data written by your program in the memory. But there should be no traces where in the physical ram your data was, so knowing which bytes belonged to what program - not to mention what were these bytes used for - is quite close to impossible. I say quite close because I've never considered that, and I do not have any accurate information how possible it could be.

If I go a little further off topic to see if I have understood it correctly (I believe there is far more experienced fellows amongst us - who will correct me if I go wrong), there is at least following benefits of having physical addresses hidden:

1. Security. No user space process has access to go and mess the HW as they want - they need to use drivers, which should be written in such a manner that hugest hazards are not possible...

2. Per process memory mapping. If memory would be accessible between processes... Oh joy. If you have ever attempted to build huge and complicated multithreaded software you know what I mean. It really is a mess. Use of one uninitialized pointer may crash anything anywhere, or just make generally bizarre things to happen.

I guess steps 1 and 2 could be implemented even if the applications could access ram directly - but I am not sure how easy / difficult it would be.

3. Shared libraries. Shared libraries are loaded to memory during runtime. When each process has their own virtual address space, shared libraries can be downloaded to physical memory only once. The same physical memory block containing the library can then be mapped for each process' address space.

Finally, the inter process communication often uses "creature" called shared memory. processes can request a shared memory block from kernel, which will then map some physical memory chunk to more than one process' virtual address space. Note however that virtual addresses in different processes need not to be the same, even though they point at same physical ram. Hence shared memory is usually accessed according to the offsets from the start of the memory pool.

However, note that when you quit your program, the shared memory is not automatically freed. (thanks to brewbuck for confirmation ). It will stay accessible (if it was created to be accessible).

And finally the [[]] which I wrote at the beginning. It is possible to write a driver, which can map some real hardware addresses to shared memory, which can then furthermore be mapped by a user space process. That way the HW can be accessed directly by a user space application.

C can be used as object oriented language!

2009-03-24T10:27:00.007+02:00

I am sick of C++ coders, who tend to yell their OOP terms to the world, and claim that OOP is the only way, and C++ is the best... Actually they're wrong when they say that C is a bad language, because it is not object oriented. They're wrong in two things.

1. C is far from bad language. Some of the most significant, most efficient, and most important things are still written in C. Linux, loads of embedded SW, (did you know that there is probably almost same amount of code in a modern car that there is in Windows XP? Scary thought.) and so on.

2. C can be seen as object oriented language, and sometimes it actually benefits to use it as such.

Actually, there is only a few things in C++, that you cannot do with C (not with same syntax of course, but with similar idea behind). The most important of those are probably destructors, templates, and overloading.

Did I hear lil C++ coder screaming about classes, virtual classes and inheritance? Allright. Maybe I'll explain a little further.

The most critical characters of object are:
1. It binds the data, and functionality (note, I avoided the "method" word... ehh.. Obviously I didnt :D )
2. It can be instantiated.

Yeah, how do you do that with C - you rat. I heard you, and don't worry, I'll explain.

I guess that even though you're so full of your fancy classes, you must've seen reserved word struct somewhere. At least when you did try to use it as a variable name last week. (Yes, the structs are in Cpp too). Then a bit more difficult thing, function pointer. If that's not a familiar concept to you (that exists in C++ too - although it is not as easy to use them there. Especially not for someone like me, who has not written C++ in... ... long time.) you can chek out my blog post here (in Finnish though).

So how to do a C representative for


const int AGE_STOMACH_RATIO 5
class dog
{
   private:
   EColour furColour;
   int age;
   unsigned int stomach_state;
   bool check_condition();
   int calculateStomachCapacity();
   void die();
   public:
   dog(EColour colour);
   void feed(int food_amount);
};

dog::dog(EColour colour)
{
   furColour=colour;
}

bool dog::check_condition()
{
    if(calculateStomachCapacity() < 0 ...

Yeah, do that with C you say...


struct Sdog;
int dog_check_condition(Sdog *_this);

typedef int (*check_conditionF)(Sdog *_this);
typedef int (*calculateStomachCapacityF)(Sdog *_this);
typedef void (*dieF)(void);
typedef void (*dogF)(Sdog *_this, EColour colour);
typedef void (*feedF)(Sdog *_this);
typedef struct Sdog
{
   EColour furColour;
   int age;
   unsigned int stomach_state;
   check_conditionF check_condition;
   calculateStomachCapacityF calculateStomachCapacity;
   dieF die;
   dogF dog;
   feedF feed;
}Sdog;

int dog_check_condition(Sdog *_this)
{
    //starvation                    //overeating
    if(_this->stomach_state == 0 || _this->calculateStomachCapacity(_this) <>stomach_state )
        return 0;
    return 1;
}
int dog_
void initDog(Sdog *_this, EColour colour)
{
   _this->furColour=colour;
   _this->age=1;
   _this->stomach_state=3;
   _this->check_condition=&dog_check_condition;
   .
   .
   .
   and so on..
}

I hope you can see the analogy.

Okay okay. How about the second important thing in allmighty OOP:

The instantiation.

There can be several instances of dog ... blaa blaa blaa...



int main()
{
   Sdog *lassie;

   lassie=malloc(sizeof(Sdog));
   lassie->dog=&initDog;
   *(lassie->dog)(lassie,EColor_brown);
   *(lassie->feed)(lasie,2);
   .
   .
// or as a local variable:
/*
   Sdog lassie;
   *lassie.dog=&initDog;
   *lassie.dog(&lassie,EColor_brown);
   *lassie.feed(&lassie,2);
   .
   .
*/

What's next? Data encapsulation? Allright.

This requires some "cheating", but what was the point of this blog is - it is doable.

Now we divide our Sdog into two different structs. Public Sdog, and private S_dog for example


typedef struct Sdog
{
   dogF dog;
   feedF feed;
   char internalData[1];
}Sdog;

typedef struct S_dog
{
//As abowe.
}S_dog;

Now we place this struct Sdog and it's functions into a public header, but keep S_dog and private functions in private headers. The .c file (implementation of functions) then includes both headers, but only public one is given to dog users. When we initialize Sdog, we'll allocate space for S_dog too (in the placeholder)(by using realloc, or then we can make the struct allocation in dogInit() function, and change it to return the pointer to created object). And when we use internal functions, we'll just cast the placeholder (Eg. char internalData[1]) to S_dog * type.

Inheritance you say

Okay. This is rather simple. We can for example define structure Sdoberman, which has structure Sdog as it's first part, and append doberman specific data and functions at the end. Then when we wish to only use generic data, we'll handle the pointer to struct as Sdog, and when we want to use doberman features, we'll cast struct to Sdoberman.

Virtual classes someone whispers...

Yeah. Now what? If we have virtual class representative Sdog, and "classes" Sdoberman, Schihuahua and Smixed derived from virtual class. Now we can use aforementioned idea, and place Sdog struct as first item. We can also add a "dogType" variable in Sdog struct, which is initialized correctly when doberman, chihuahua or mixed is created. Then in virtual functions we can simply check the dogType variable, perform correct cast to _this pointer, and call correct frunction from Sdobermann, Schihuahua or Smixed.

Another approach (which also sounds more natural to C), is to have the "dogType" as first member in Sdog struct, use void * pointers, to the struct, dereference first sizeof(dogType) bytes and then perform the cast to correct dog struct.

I know. All C++ coders are terrified. But when a C coder looks at this, he/she notices the idea (I hope), and when we look the interface this creates, we'll really see the basic OOP structures.

A note about destructors and C

Furthermore, it is also possible to create a destructor type thing for your objects - when they're dynamically allocated with malloc() type call. It involves quite simple steps:

1. build your own malloc() wrapper (MyAlloc() for example), which allocates the space user requested + some extra for some internal structures. Then return pointer to this "user data area". Allow registration of callback function, which will be executed when memory is freed (the destructor), and store pointer to that in the extra space (which is invisible for user) you allocated. You can also create some additional debug things, like add some space after the user requested block, and write somekind of "end mark" to it. End mark which will be destroyed, if user overwrites the reserved memory... (This naturally requires storing the allocated blocks size in internal "header" too)

2. build your own free() wrapper, which executes the callback (destructor) and checks the endmark validity (if such was implemented), and frees the (internal + user) memory.

Final Words...

Okay. I think I have trolled enough now, and most of the C++ coders are so upset by now, that they have propably stopped reading ;) So I guess it is good time to tell what I really think. I think that C++ is great language, providing many things that are better than plain C. However C++ is not really usable in all environments, or at least C++'s best parts aren't. For example, in an embedded project where I was working I did one test using C++ and STL. Required stack size was tripled compared to C implementation. You may tell that it is because I used buggy compiler/buggy STL implementation. You can say this, you can say that. However fact is, that with C++, you'll easily get bloated code, larger executables, increased resource usage. You may be able to avoid these (partially), by limiting the C++ toolset you use - but then you cannot keep yapping how great the C++'s all cool features are, since you cannot really use all cool features. C++ is great for many purposes, but there is things where C is far better option.

I'll see if I can later show you some simple implementation of C++ style C - I have really sometimes used it, and it may be a good approach to some problems. Actually, I described one such situation in my Finnish blog here (In finnish blog => it is written in Finnish.)

Answer to pitfall III (I forgot to post this earlier)

2009-03-20T21:51:00.005+02:00

The answer is simple. I bet the bells start ringing when I say this. Usually with 32 bit systems, memory address is 32 bit wide. With 64 bit systems it often is 64 bits wide...

...

...Now, look at the pitfall III again, and you'll see that we do pinter arithmetics by handling pointers as integers. This means that half of the address will be cropped, and remaining address is most propably something pointing at invalid locations. After you use it, MMU will detect your process accessing out of the virtual address space mapped for the process. It'll kick up kernel, which most propably decides to send SIGSEGV signal to your program. And if youre writing anything more critical than snake game, you haven't registered handler which forgives SIGSEGV. Eg. result is segmentation fault. Adios. Bye. Game over. Sommoro ja Soronoo.

My first C quiz at 2009

2009-03-20T21:37:00.005+02:00

Hi dee Ho peeps!

It has been looong since I wrote last pitfall/quiz in here, so I guess it is about the time to continue :)

So at this time I'll just remind you of one nastyish C specific bug - which is easy to avoid by reading the compilers warnings. I've tested it on gcc.

So following code will work on 32 bit machine, but on 64 bit machine you are allowed to expect sudden death... What's the cause?

And as always, you can use comment section to give answers :)

file main.c

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main()
{
int data[32];
int i;
int *copydata;

for(i=0;i<32;i++)
{
data[i]=i;
printf("data[%d]=%d\n",i,data[i],i);
}

copydata=allocateMyMate(sizeof(data));
if(copydata==NULL)
{
printf("Allocation Failed!!\n");
return -1;
}
else
printf("Allocation succeeded!\n");

memcpy(copydata,data,32*sizeof(int));
printf("Looks like you had 32bit machine :) \n");
for(i=0;i<32;i++)
{
printf("data %d == copydata %d",data[i],copydata[i]);
}
return 0;
}

myalloc.c

#include <stdio.h>
#include <stdlib.h>

void * allocateMyMate(size_t size)
{
return malloc(size);
}

file main.c

So where's the bug?

And even though the free() corresponding to malloc() is missing, do not consider it as a bug at this time. (Naturally it is a bug, but it was not the bug I wrote this question for).

HINTS BELoW! Do Not Look YET ;)

1. With c++ youre safe from this bug.
2. Try compiling and check the compiler warnings!

C - Explode.

2008-09-16T10:45:00.021+03:00

15.09.2011 Sources can be found from svn repository at
http://xp-dev.com/svn/MazBotV4/trunk/generic/src/ CExplode functions are nowadays part of the helpers.c and helpers.h files.

Some kind of an API doc can be found from
http://xp-dev.com/svn/MazBotV4/trunk/docs/html/helpers_8h.html
http://xp-dev.com/svn/MazBotV4/trunk/docs/

06.08.2009 NOTE: Another bug found, put up "correction release" v0.1-1
(svn repository at here , full checkout can be done if svn client is installed with: "svn checkout http://xp-dev.com/svn/Mazzie-mazbot/" ), but if you only wish to get the Cexplode, you can simply go there with web browswe, and save files generic/src/helpers.h and generic/src/helpers.c There propably is also some other helper functions, which may be dependant to other bot project sources, If that is the case, just erase everything except the Cexplode related stuff, and files should be compilable. If you need help, leave me a comment, and I may do a separate zip with Cexplode and makefile only :)

And please, if anyone uses the code and finds a bug, please file me a bugreport at Mazziesaccount@gmail.com

27.07.2009 NOTE: The code presented here is faulty. I found a bug which can cause the last character of last exploded piece to be random. (And possibly other errors). I have fixed this in code I have written for my MazBot project. (Look my comment below to find the updated version) I will not update code on this website because formatting C code to be displayed correctly on this blog is somewhat troublesome && errorprone and when done manually. I am terribly sorry for inconvenience!

I decided to write also ANSI C version of php's explode() function. Why? Well, because it seems to me that most of the visitors here have been searching for C/C++ implementation for php's explode() function. So here it is my loyal readers, cxplode() written in C. a.k.a Cexplode() by Maz :)

(I'll also place it in downloadable Cexplode.tar.gz when I'll find the time.)

Cexplode.h

/* ******************************************************** */
/*                                                          *
*      Implementation of php's explode written in C        *
*      Written by  Maz (2008)                              *
*      http://maz-programmersdiary.blogspot.com/           *
*                                                          *
*      You're free to use this piece of code.              *
*      You can also modify it freely, but if you           *
*      improve this, you must write the improved code      *
*      in comments at:                                     *
*      http://maz-programmersdiary.blogspot.com/           *
*      or at:                                              *
*      http://c-ohjelmoijanajatuksia.blogspot.com/         *
*      or mail the corrected version to me at              *
*      Mazziesaccount@gmail.com                            *
*                                                          *
*      Revision History:                                   *
*                                                          *
*      -v0.0.1 16.09.2008/Maz                              *
*                                                          */
/* ******************************************************** */

#ifndef CEXPLODE_H
#define CEXPLODE_H

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

typedef struct CexplodeStrings
{
int amnt;
char **strings;
}CexplodeStrings;

typedef enum ECexplodeRet
{
ECexplodeRet_InternalFailure    = -666,
ECexplodeRet_InvalidParams         = -667
}ECexplodeRet;

int Cexplode
(
const char *string,
const char *delim,
CexplodeStrings *exp_obj
);
char *Cexplode_getNth(int index,CexplodeStrings exp_obj);
char *Cexplode_getfirst(CexplodeStrings exp_obj);
void Cexplode_free(CexplodeStrings exp_obj);


#endif

Cexplode.c

/* ******************************************************** */
/*                                                          *
*      Implementation of php's explode written in C        *
*      Written by  Maz (2008)                              *
*      http://maz-programmersdiary.blogspot.com/           *
*                                                          *
*      You're free to use this piece of code.              *
*      You can also modify it freely, but if you           *
*      improve this, you must write the improved code      *
*      in comments at:                                     *
*      http://maz-programmersdiary.blogspot.com/           *
*      or at:                                              *
*      http://c-ohjelmoijanajatuksia.blogspot.com/         *
*      or mail the corrected version to me at              *
*      Mazziesaccount@gmail.com                            *
*                                                          *
*      Revision History:                                   *
*                                                          *
*      -v0.0.1 16.09.2008/Maz                              *
*                                                          */
/* ******************************************************** */


#include "Cexplode.h"


int Cexplode
(
const char *string,
const char *delim,
CexplodeStrings *exp_obj
)
{
int stringL = 0;
int delimL  = 0;
int index;
int pieces=0;
int string_start=0;

char **tmp=NULL;

//Sanity Checks:
if(NULL==string || NULL==delim || NULL == exp_obj)
{
    printf("Invalid params given to Cexplode!\n");
    return ECexplodeRet_InvalidParams;
}
stringL = strlen(string);
delimL  = strlen(delim);
if(delimL>=stringL)
{
    printf("Invalid params given to Cexplode!\n");
    return 0;
}
for(index=0;index<stringL-delimL;index++)
{
    if(string[index]==delim[0])
    {
        //Check if delim was actually found
        if( !memcmp(&(string[index]),delim,delimL) )
        {
            //token found
            //let's check if token was at the beginning:
            if(index==string_start)
            {
                string_start+=delimL;
                index+=delimL-1;
                continue;
            }
            /*
               if token was not at start, then we
               should add it in CexplodeStrings
            */
            pieces++;
            if(NULL==tmp)
                tmp=malloc(sizeof(char *));
            else
                tmp=realloc(tmp,sizeof(char *)*pieces);
            if(NULL==tmp)
            {
                printf("Cexplode: Malloc failed!\n");
                return ECexplodeRet_InternalFailure;
            }
            //alloc also for \0
            tmp[pieces-1]=malloc
            (
              sizeof(char *)*(index-string_start+1)
            );
            if(NULL==tmp[pieces-1])
            {
                printf("Cexplode: Malloc failed!\n");
                return ECexplodeRet_InternalFailure;
            }
            memcpy(
              tmp[pieces-1],
              &(string[string_start]),
              index-string_start
            );

            tmp[pieces-1][index-string_start]='\0';
            string_start=index+delimL;
            index+=(delimL-1);
        }//delim found
    }//first letter in delim found from string
}//for loop

if(memcmp(&(string[index]),delim,delimL))
    index+=delimL;
if(index!=string_start)
{
    pieces++;
    if(NULL==tmp)
        tmp=malloc(sizeof(char *));
    else
        tmp=realloc(tmp,sizeof(char *)*pieces);
    if(NULL==tmp)
    {
        printf("Cexplode: Malloc failed!\n");
        return ECexplodeRet_InternalFailure;
    }
        tmp[pieces-1]=malloc
       (
          sizeof(char *)*(index-string_start+1)
       );
    if(NULL==tmp[pieces-1])
    {
        printf("Cexplode: Malloc failed!\n");
        return ECexplodeRet_InternalFailure;
    }
    memcpy
    (
      tmp[pieces-1],
      &(string[string_start]),
      index-string_start
    );
    tmp[pieces-1][index-string_start+1]='\0';
}
exp_obj->amnt=pieces;
exp_obj->strings=tmp;
return pieces;
}


char *Cexplode_getNth(int index,CexplodeStrings exp_obj)
{
if(exp_obj.amnt<index)
{
    return NULL;
}
return exp_obj.strings[index-1];
}

char *Cexplode_getfirst(CexplodeStrings exp_obj)
{
return Cexplode_getNth(1,exp_obj);
}
void Cexplode_free(CexplodeStrings exp_obj)
{
int i=0;
for(;i<exp_obj.amnt;i++)
    free(exp_obj.strings[i]);
free(exp_obj.strings);
}

Cexplode_example.c

/* ******************************************************** */
/*                                                          *
*      Implementation of php's explode written in C        *
*      Written by  Maz (2008)                              *
*      http://maz-programmersdiary.blogspot.com/           *
*                                                          *
*      You're free to use this piece of code.              *
*      You can also modify it freely, but if you           *
*      improve this, you must write the improved code      *
*      in comments at:                                     *
*      http://maz-programmersdiary.blogspot.com/           *
*      or at:                                              *
*      http://c-ohjelmoijanajatuksia.blogspot.com/         *
*      or mail the corrected version to me at              *
*      Mazziesaccount@gmail.com                            *
*                                                          *
*      Revision History:                                   *
*                                                          *
*      -v0.0.1 16.09.2008/Maz                              *
*                                                          */
/* ******************************************************** */

#include "stdio.h"
#include "Cexplode.h"

int main(int argc,char *argv[])
{
char *string;
char *delim;
int retval;
int index=0;
char *token;
CexplodeStrings expString;
if(argc!=3)
{
    printf("Test Command should be:\n");
    printf
    (
      "<testExe> \"original string\" \"delimiter string\""
    );
    return -1;
}
string=argv[1];
delim=argv[2];

printf("TestString is \"%s\"\n",string);
printf("Test Delimiter is \"%s\"\n",delim);
if(0>(retval=Cexplode(string,delim,&expString)))
{
    printf("CexplodeFailed!\n");
    return -1;
}
else
{
    //Way 1, use expString straight away:
    printf("Way 1, use expString straight away:\n");
    for(index=0;index<expString.amnt;index++)
    {
        printf
        (
          "token %d = %s\n",
          index+1,
          expString.strings[index]
        );
    }
    /*
       Way 2, you can use Cexplode_getNth,
       or Cexplode_getfirst()
    */
    printf(
     "Way 2 use Cexplode_getNth, or Cexplode_getfirst():\n"
    );
    token=Cexplode_getfirst(expString);
    printf("first token %s\n",token);
    index=1;
    while(NULL!=(token=Cexplode_getNth(++index,expString)))
    {
        printf("token %d = %s\n",index,token);
    }
}
Cexplode_free(expString);
return 0;
}

Answer to pitfall IV

2008-08-30T22:39:00.003+03:00

I"m in hurry, hence just a short answer.
This is where the problem lies:


fbs=(foosNbars*)malloc(amount*sizeof(foosNbars));
    if(NULL==fbs)

When user inputs big enough amount, the result of amount*sizeof(...) does not fit in 32 bit variable. Variable overflows. What does the overflow result? It results small positive number (since malloc takes arg as unsigned int). Hence malloc really allocates some space - but not the amount user originally requested. Thus the NULL check at next line does not detect the error, and later when user attempts to fill all structs he thinks is allocated... Yep, crash boom bang Segmentation Fault.

rtrim() in C

2008-08-23T23:06:00.006+03:00

Another nifty php funktion that C and C++ miss is rtrim(). trim is a function, which removes whitespaces from the end of a string.
So here's my implementation of rtrim() for char arrays:


#include <string.h>

/**
\brief removes trailing whitespaces and tabs from a string.
\returns number of characters removed
*/
int trim(char *text)
{
    int i;
    int len;
    int retval=0;
    len=strlen(text);
    for
    (
      i=len-1;
      i>=0&&( text[i]==' ' || text[i]=='\t' );
      i--
    )
    {
        text[i]=(char)0;
        retval++;
    }
    return retval;
}

It would not be too difficult to implement the possibility to trim other characters. Actually just an edition to 2 of the lines ;)

Thoughts.

2008-08-15T10:30:00.002+03:00

Life.

Dawn. Surrounded by loving smiles.
By caring, strong, safe people.
You learn to trust them to be always there.
You learn to love, love so much it hurts.

But dawn never lasts forever.
You carry them away, one by one.
Saying goodbye.
Never again - cruel, ruthless words.

Summer/Autumn 2008
-Maz

Answer to C - Pitfalls part II

2008-08-01T23:26:00.004+03:00

Porting/portability issues regarding C/C++

Allright. one of the big things in the programming today is portability. Windows is still the most common operating system in PCs, but finally one can see some ineterst in big companies to also build some linux support in their applications. Mac is still alive too, and it is far cheaper to write software so, that you can use at least some common parts of it for all platforms. Another portability issue is different hardware architectures. 64 bit systems are increasing, and soon they will propably be more popular than traditional 32 bit architectures.

So how does one create portable code? What things to consider?

It all begins from planning. First of all you need to carefullly estimate what kind of general requirements there will be for your program, supprted platforms amongst others. After these are somehow clear, you need to decide which language you will use for your program.

Let's assume you ended up needing support for many platforms, and both 32 and 64 bit architectures. Lets further assume, that for some odd reason, you deducted that best language to use is C. After you have reshecked your calculations, and still failed to find the error which led you to using C, you need to decide what should you do all by yourself, and where you should/can use 3.rd party libraries...

So first step is to choose portable libraries. Fortunately there's many good libraries which can be used on multiple platforms, via similar interface. There's openGL & co. for graphics, there's pthreads for windows and unix threads, there's PoSIX sockets and winsock (which have some differencies in basic usage, but those can be quite easily hidden)... Plenty of good stuff.

And what do you need to take in consideration while writing actual implementation? First you need to decide what compilers to use for which platforms. In linux world I can only see one reasonable possibility - gcc. But on windows... Well, this is a bit thougher. There is mingw version of gcc for windows, but I am really not that familiar with it, nor am I sure about the libraries available for it. There's also M$ Visual XXXXX family of compilers, and I've heard some of them are quite ok. But with M$ products you will propably get the world's most standards breaking product... And naturally that's bad for portability.

What ever do you decide, you must be carefull with the functions you use. ANSI C functions should all be quite similar on both M$ compiler and GCC, but the excellent features of PoSIX standard interface offers are often quite cribbled on M$ side. As a good example, I need to ask what was wrong with snprintf()? Honestly? Why did M$ have to do _snprintf? Just to be a pain??

And naturally there's those fundamental differences in the princibles of operating systems. For example in mapping the virtual memory for processes. Linux kernel may hide same physical addresses behind different virtual addresses in each process. Eg. Your perfectly valid pointer in one process pointing at your cleverly built data storage will be pointing at invalid location in other process. Thus making it meaningless to pass raw address data from one process to other. Also shared memory, signals, scheduling... all of these are different. And your application needs to tackle each of these problems. Hopeless mess? No, there is projects which have succeeded in quest of portability. Errorprone? Sure as hell. You need to plan things c*a*r*e*f*u*l*l*y beforehand. Hard but doable.

Now in the pitfall II I said I'll show you one portability problem. But it was not problem related to porting software from one system to other, instead it was a problem related to making software to work both on 32 and 64 bit systems. Remember following:

In 64 bit systems, forcing pointer to transform itself into a integer leads thy to death and destruction. Well, at least to segmentation faults.

At 64 bit systems, pointer is 64 bits wide, not 32 as it is in 32 bit systems. So when you place such a pointer in integer (by casting), you'll lose half of the address...

other issues to consider are:

Structures which contain pointers, and which are populated by believing in knowing their size / accessing members by using hardcoded offsets.
Functions which might return pointer or integer depending on situation.
hardcoded bit shifts.

And so on.

But once again the night is falling in Finland, and I'd better get my head on pillow...

C++ explode() function.

2008-08-01T20:44:00.004+03:00

Please see efnx's modified version in comments, it is way more efficient than the original. (it does not use temporary vector, and copy it to store - instead it modifies the store vector straight away - which is way more efficient && elegant)

As I have somewhere sometimes stated, I have been writing some php as a hobby. The evidence is CWF Freeware. There's some really handy php functions out there for string manipulation. And since I once needed this, and because I've heard many other's have needed this too, I'll share it with you my readers. (Yes, with both of you :D)

Just a compulsory disclaimer before representing it:

This is one of the functions which I did when I was still a beginner with STL. So there's propably hundred's of more efficient and elegant solutions, and those may be posted in comments here. I'll be glad to see them. But since I happened to find this function from my archives now, I'll give it now. It may be buggy, it may be flawed, but it sure is better than having no explode() in C++ at all. (And yes, standard libraries do not offer explode function in C or C++).

/* This one 'explodes' a string (cuts it into pieces based on some matchstring) and puts pieces into a vector. arguments are the string to be cutted, matchstring and the vector where to store pieces. It returns amount of pieces, and if no matchstring is found it returns -1. */


#include <iostream>
#include <vector>
#include <string>

using std::string;
using std::vector;


int explode(string tear_me,string cut,vector &store)
{
        unsigned int start=0;
        int len=0;
        int i=0;
        vector<string> temp;

        len=cut.length();
        if(tear_me.find(cut)==string::npos)
        {
                return -1;
        }
        while( (start=tear_me.find(cut))!=string::npos)
        {
                temp.push_back(tear_me.substr(0,start));
                tear_me.erase(0,start+len);
                i++;
        }
        temp.push_back(tear_me);
        store=temp;
        return i;
}

C/C++ - Pitfalls - part IV

2008-08-01T18:59:00.003+03:00

overflow. Almost as terrible as overrun.

Another C writer's nightmare is overflow. We must remember that C does have fixed lenghts assigned to variable types. For example, short integer is 16 bits wide, and integer is 32. When we consider what this means in terms of 10-base numbers... Well, 32 bits, each of which can be 1 or 0. So largest number one can have in such a field, is 2^32. And when we think of signed integers, then its half of that, because the other half of possibilities are reserved for negative numbers. It is quite a big number, but it can be exceeded. Especially this is a problem in mathematical calculations, which often involve calculating large series of powers, which may get really large values. There's also other issues with mathematical programming, like accuracy of real numbers (as accurcy what comes to results, but also as accuracy what comes to comparing numbers), but I won't get into that.

Just remember that when you overflow an signed integer, it results really negative value, and when you overflow unsigned integer, it results really tiny positive value. Also casting really large unsigned value to signed, will cause negative value... So beware - C is full of pitfalls :)

Well, I'll once again give you something to ponder for a while:

Example of a common - yet dirty and dangerous habit.
(No, I am not talking about smoking!)

often one sees things like following:


typedef struct foosNbars
{
    int foo[20];
    char bar[200];
}foosNbars;

foosNbars * obtain_foosNbars(unsigned int amount)
{
    foosNbars *fbs;
    fbs=(foosNbars*)malloc(amount*sizeof(foosNbars));
    if(NULL==fbs)
    {
        HANDLE_ERR("could neither allocate foos nor bars", ErrType_noMem);
        return (foosNbars *) NULL;
    }
    return fbs;
}

HANDLE_ERR() is here just some custom error handler, which gets the error type, and handles it. It is not really relevant in here. So question is, what happens? What is the issue? Where is the problem?

Answer to C - Pitfalls part II

2008-08-01T06:39:00.003+03:00

Okay. It's about a time to continue answering to my own questions. So let's see.

Problem with the kind of vector class usage I showed is that when a new item is inserted in a vector, vector may need to relocate all items into some other address space. If the space reserved for vector's contents is full, and no more space can be allocated from the back of the vector, vector will transfer all of it's contents in some new memory position where it can allocate a continuous block.

So, now when we hand the address of an object to new thread, and add items to the vector in other thread, the vector may suddenly need to move the object into new memory position. The other thread naturally knows nothing about this, and the address we use in new thread to access the object will no longer point to the object. Hence the new thread may suddenly crash in odd place.

Some ways to overcome this problem...

1. If you know the maximum amount of objects that will be placed in vector, you can reserve space in the vector to that lenght, so that it does not need to reallocate new space when items are added. Problem is, that often we use STL containers like vectors, exactly because we do not know the amount of items..

2. You can give the address of the vector itself as an argument to the thread. Vector's address should not change (even if the location of vector's contents does), so when you use the same vector in both threads, the changes in vector's contents locations will be known in both threads.

Answer to C - Pitfalls part I

2008-07-28T22:16:00.005+03:00

The first bug is in here:

/*get the bits which are not changed
(bit's value will be inverted when
XORed with bit 1)*/
tmp=(changedbits^(unsigned int)0x11111111);

Now, as explained in comment, the purpose is to set tmp to contain a value, where all bits which should remain same as in original value are 1, and all bits that should be changed to 0. Reason is that when we at next line use this value and AND operation with original value, we'll manage to set all bits which will be changed to 0. However the value 0x11111111 is not a value, where all bits are set to 1. A number starting with 0x is interpreted as hexadecimal value, and when we want to have a 32 bit wide value where all bits are set to 1, it is 0xFFFFFFFF.

Second bug is more of principal. The implementation shown in example will change all bits which are non zero in 'changeto' variable - regardless the changedbits variable. Correct implementation would be:

unsigned int changebytes(unsigned int const &real, unsigned int const &changedbits, unsigned int &changeto)

{
unsigned int tmp,tmp2;
/*get the bits which are not changed

(bit's value will be inverted when

XORed with bit 1)*/

tmp=(changedbits^(unsigned int)0xFFFFFFFF);
/*null the bits which will be changed.
(AND will set result to zero, if one of the bits
'anded' is zero).*/
tmp2=(real & tmp);
/*add the bits which should be changed from changeto in the tmp

(OR will 'sum' the bytes).*/
return (tmp2 | (changeto & changedbits));
}

C/C++ - Pitfalls - part III

2008-07-27T22:08:00.008+03:00

Make your code to work on multiple machines, for as long as possible.

Even though the C-language is well standardized, there's still many pitfalls regarding code portability. First of all, with pure ANSI C it is really hard to create any graphical applications. Real program writing usually relies upon some extensions, Eg. libraries not belonging in standard C. Luckily there's many libraries which are available for multiple platforms.

But even if we put that problem aside, we will find issues which may limit our code's portability. I will now represent one such thing because this must be noticed when one wishes his code to be working on different machines.

Back when things were simpler, I amongst many others used this kind of functions for calculating memory addresses.

void * get next_one(void **ptr, size_t sizeofobj)
{
    int address=(int)(*ptr); //store the address in integer
    address+=(int)sizeofobj;
    return (void *)address;
}

As we see, abowe function simply calculates a specific memory address. However, since computers are not same as they used to be, this nowadays introduces possible hazards... Can you spot the root of the problem?

C/C++ - Pitfalls - part II

2008-07-27T11:46:00.010+03:00

Memory violations

okay, let's now check a way to really mess things with bad design... Following bug actually demonstrates nastyness of C/C++, it is terribly easy to create numerous annoying bugs, which strike only occasionally. Memory violations. That's a word every C/C++ programmer should be afraid of. No, actually terrified.

Nastynes of memory violations is in the way they emerge. A code can be successfully executed numerous of times, and then suddenly it once crashes. Reason for such behaviour is the way operating system protects memory areas.

When a program is first launched, certain amount of memory is allocated for the process. It is perfectly ok to write/read anywhere inside that allocated memory chunck. Let's now assume we have somehow messed our code so that it writes in random memory location (There's really many ways to actually do this... And I bet every programmer has sometimes done such a bug.) It is fairly possible that our random location happens to be in memory chunck which was allocated for the process' use. So operating system think there was nothing wrong with the write. How ever it may be that we just overwritten some critical information, and that may cause a crash in arbitrary position in the code. Thus locating the real cause is not easy. It may as well be that we managed to write in allocated but unused space. Then it may be that our program completes flawlessly. It may also be that we just wrote in space which was protected by operating system. That will cause instant crash. Unfortunately even if we once can run the program flawlessly, it may well be that it missbehaves, or crashes next time we run it. When you create commercial program, it is not really appreciated amongst customers if your code crashes... And it may do that regardless of carefull testing.

Hence, beware the memory violations.

So let's now see one hideous way to create nasty memory violation...
I once almost did this mistake, but then... A thought stroke my head. How does vector class internally operate? And there's a bug related to this issue in the principle presented in my example below.

vector objvector;
.
.
.
objvector.push_back(obj A);
.
.
objvector.push_back(obj B);
.
.
.
//Create thread with entrypoint void * newthreadfunc(obj *);
// give &objvector[N] as an argument.
.
.
.
objvector.push_back(obj C)
.
.
.

void * newthreadfunc(obj *)
{
//do things
//do more things
//Do something with objvector
return something;
}

So what is the principal fault in this kind of approach?
(Idea was of course to use the same object in both threads, which would have been perfectly ok when correct semaphore protection would have been used. This time it has something to do with vector class' internal operation...).

C-pitfalls in which I have fallen.. Spot the Bug part I

2008-07-27T09:07:00.022+03:00

I have been writing C/C++ code for my work for over 4 years now. Yet I tend to do simple mistakes. How is that possible? Well, because C is full of pitfalls...

I will post some buggy codes here, as warning examples. I will not tell what the bugs are - yet. I will reveal that later in this blog, so you can try finding the problems yourself :) And naturally solutions can be posted in comments.

So mistake 1. It was a simple bit masking in C.

I had an unsigned int coming in a function, and I wanted to replace certain bytes from it. So I decided that I'll write a function, which gets the original unsigned int, indication about which bytes should be changed, and what those bytes should be changed to.

For example:
0xXXXXXXXX original value
0x00FF0000 bits to change
0x00AB0000 What to change the bits.
Result: 0xXXABXXXX

unsigned int changebytes(unsigned int const &real, unsigned int const &changedbits, unsigned int &changeto)

{
unsigned int tmp,tmp2;
/*get the bits which are not changed
(bit's value will be inverted when
XORed with bit 1)*/
tmp=(changedbits^(unsigned int)0x11111111);
/*null the bits which will be changed.
(AND will set result to zero, if one of the bits
'anded' is zero).*/
tmp2=(real & tmp);
/*add changeto bits in the tmp
(OR will 'sum' the bytes).*/
return (tmp2 | changeto);
}

Well, this actually introduces (at least) 2 bugs.

Having nothing to do? Get a car...

2008-07-26T22:49:00.000+03:00

24-25.07.2008

Thursday... It is wet. I am sitting in my car, driving back to home from work. Suddenly I notice the warning light telling there's something wrong with my ABS brakes. Damn. Occasionally I hate my life.

I have cranky old Nissan Primera SLX, with 2 litres engine. Well oldness is of course subjective matter, as is crankyness.. A few years ago I was driving Mazda 323 from 1985.. This one is done at 1999.

The ABS light itself did not really scare me. It had been acting up since last autumn, turning on when ever weather was humid. But the light had been silent almost a month, and I knew I had to drive my car for the compulsory annual inspection (is it MOT in english?) tomorrow...

Actually, I was terrified. I had started fixing my Nissan 2 months ago, when I changed the exhaust system since there were a leak. It's strange how each simple sounding task with car always turns out to be complicated. The exhaust system change involved totally jammed bolt, drilling it out under the car, replacing it with bigger bolt, and generally spending a whole summer day with my uncle cursing the car. (Thank's god I have my uncle, not only is he friendly and caring and funny person, but he also has the knowledge, tools and nerves for fixing cars).

Well, quite soon after changing the exhaust I tried to pass MOT (is it really called that??)... And naturally there were problems with everything else. Brake plates were too worn, handbrake did not work on other side, and the thing which is at the far end of the stick which turns the front wheels was loose... And naturally I had forgot to change pulbs so half of the lights were dark...

So a week ago I found myself being under my car with my uncle once again.. (Did I remember to thank him already? He's really my saver). We changed the brake plates, and that steering related thing. And of course there were unexpected problems, like jammed part in brakes... (The one which allows the brake to squeeze the plate, I mean that ~2 inch long stick which is kinda floating on vaseline). Replacing it was not that straightforward wither, it involved standing at the yard of closed spare part shop (in which I had called a few days before and ensured it was open on saturdays...), calling to the shop's owner's wife and asking her to call someone to sell us the missing part .... pleeeaseeee...

Well, I stared the ABS warning light shocked. Can't be I thought. I drove at home, did my daily things with kids, and get them to bed at 10.00 PM. Then I went out to take a look at the car. I started the engine, and after a little while the ABS light started burning... I shut down the engine and restarted.. No warning light. I decided to test the brakes just to be sure. Worked like charm. I decided to test the handbrake too, just to be sure... And right side did not work too well.

I was ready to take the biggest hammer I own (which actually is not too big, I only own one :D )..

Well, I took my jack, lifted the right rear and took the tire off. I checked the thing where the handbrake's wire is connected, and as I guessed, it was not moving too well... So oil and hammer, oil and hammer, oil and pulling and bushing the handbrake on and off like a maniac. Putting tyre back and lowering the car. Small test, and voila, handbrake worked...

At the friday morning I drove back at work once again. The ABS light was dead as it should. I had decided to drive my car for inspection after the workday was off. I was a bit excited my whole work day. I tried to think what kind of requirements supporting multiple processes in Linux would cause in SW, which was previously designer to be run on a system which does map the memory addresses similarly for each process. Somehow I however ended up just doing little this and that, amongst other things I swapped couple of my co-workers screensavers with the
"All work and no play makes Jack a dull boy" one... It's their own fault to have holidays when I am working. And moreover it's their own fault to have their .xscreensaver file unprotected...

Finally it was time to leave from work. I walked down to the car, and started the engine... ABS light turned on instantly, and brakes locked when I tested them. I pumbed the brake, turned car on and off, turned all electrical devices off and on, and finally the ABS light turned off and did not turn on anymore.. I drowe to the inspection, and felt sick while waiting for the inspector to step in my car and drive it to brake tester... Finally a friendly man told me it was my car's turn, stepped in my car and drowe it to tester. I watched the meter when he tested the car, and thank god, the results looked good. The inspector drove out of the tester, and yelled me through the window that I had "excellent brakes". I was ready for bursting into hysterical laughter, but I managed to controll myself and just mumbled: "I have been working with them"...

Then quick check for rest of the car, and papers out of the printer with comforting PASSED printed on them...

A day with kiddos...

2008-07-26T22:10:00.001+03:00

26.07.2008

It was heck of a good weather today. Just a weather you would hope there was whne you're having a bachelor parties. No, today was not a day when I have my bachelor parties, but at the 09.30 AM a motorbike rushed to our yard and took my wife-to-be to have her bachelor parties...

It means I was having a day together with kids.

I decided I'll try to write some code for the CWF today. After all, I have a bugtracker full of bugs and feature requests there, and I had neglected CWF for far too long. I know I have all the excuses in the world to not spend time developing CWF (studies, work, kids, weddings approaching...), but even if those reasons were good enough for others to forgive me my absence, they are not good enough for me to accept unfinished task. So after spending some hours with kids at the yard, I put them to have their daily naps...

I took the laptop, and started the notepad++. I was a bit annoyed because I could not use my desktop running linux and having all my real developing tools installed ( vi ). It is located in my older son's room, and I had just put him to sleep.

Well Well Well. I checked the bug tracker, and decided I would next do the 'user comments' section for CWF's game info pages. Eg a section where users can leave comments about games.

It hadbeen a few months since I last touched CWF's code, and I was a bit rusty. I sweated and sweared. Have I ever mentioned CWF's code is a horrible mess? The whole project started all wrong.

CWF begun when group of members of an abandonware site, which was abandoned by the admin, decided to find a new place to hang out. Unfortunately the starter did a baddish choise by installling phpbb plus board... And later I did an horrendous error by starting to write the actual site on top of the heavily modified phpbb2 forums engine.

Hey phpbb2 guys (as if you would ever read this :rolleyes: ) do not take it wrong. I do respect the work you've done. The phpbb2 forum works, and it is easy to use even for totally computer illiterate persons, but you must admitt it is not designed to be a framework for developing wholly new website...

Ok. I should not blame the phpbb2 only, there's far more critical things which lead the CWF code to be a mess. Main reason is, that CWF was my first real php project. And it still is. I have never read a single book about php, or had a single course about it. So I was (and in some sense I still am) a novise what comes to php or web publishing. As I have stated, I work with WCDMA/WIMAX base station project, and write C && C++.

However, after a few cups of coffee, I was back on the track and started writing a class for handling comments. or actually 2 classes, one to act as a struct in C (having comment data in a srtuct), and the other to handle storing and fetching the comments from database to the 'struct'. I then also decided to add dumping the data to template in the class too. Then I added a few lines to the actual game info page, (include for class, class creation and comments fetching and displaying using the created object). Finally I added a php script which handles data sent when sending new comments/editing comments.

So what is still lacking is the template files for editing comments, and displaying/sending new comments. But since I sincerely hate creating templates, it may take a few days-weeks to complete them... If only CWF could attarct some capable, freeware ebthusiast php coder who knows phpbb2... I would gladly split the work with someone...

About Me

2008-07-26T22:05:00.000+03:00

Since this is going to be kind of a diary, I decided first tell who the heck am I...

It all begun at 12.03.1979 in smallish town called Iisalmi in Finland. Around 03.00 AM I was born, and ever since I have loved doing things at night, and sleeping at day.. However since my first kid was born at 2005, I have not really managed to sleep at daytime, resulting me to sleep hardly at all. Well, sleeping is waste of time anyways ;)

When I was 4 years, my parents decided to say farewell to Iisalmi, and moved to the most distant place in Finland - to Enontekiö. It is muncipality at the northern finland. A place with lot's of rough beauty and fresh air, and with lot's of nothing to do besides fishing and hunting. So inevitably I grew up doing both of them, and I still love those cold autumn mornings when I wake up at forest with snow on my nose...

After serving the compulsory military service at 1999, I moved to Oulu, which is almost middle of the Finland considering north-south direction, but at the west coast. I started my physics studies here, and those are still unfinished... Hopefully not forever.

When I add that I have always been facinated by computers and games, it propably explains my current interests - Nature, Science, Technology and Computers && Programming.
.
.
.
Unfortunately regardless my former cross-country skiing hobby never continued after army - all I got from it is a cabinet full of medals and spoons. I wish I could still run for a while, without feeling that if I take one step more, I'll die...

However, that's quite a lot information about me, I guess far more than you cared to know...