Real-Time Embedded

Using Linux Netfilter framework

Hai Shalom — Sat, 14 Jul 2012 00:28:20 +0000

The Linux Netfilter framework is a flexible mechanism that allows you to examine packets during several processing points (for example, when they are received, after processing, before transmission, etc.). The framework is flexible, because you can register (and unregister) your hooks during run-time, and even to select the exact processing point you are interested in. Once you have finished the packet examination, you are able to modify the packet contents, and also either tell the system to continue processing, drop the packet, queue the packet for user space or to handle it yourself (steal it) and abort further processing. The Netfilter framework is very useful for firewall rules, but it is also useful for other advanced uses. In this post, I will show how to enable it in the kernel, discuss about the existing Netfilter hooks in the kernel, and provide examples how to register and implement a Netfilter hook.

Netfilter hooks

The Linux networking sub-system is instrumented with Netfilter hooks in various locations. As per LXR, the hooks are located in the IPv4, IPv6, ARP, DECNET and Bridging code. There are 5 processing points which you can register your hooks:

NF_INET_PRE_ROUTING – Before routing decision
NF_INET_LOCAL_IN – When locally receiving a packet
NF_INET_FORWARD – When forwarding a packet
NF_INET_LOCAL_OUT – When locally transmitting a packet
NF_INET_POST_ROUTING – After routing decision

The hook itself, is a macro, which is defined as follows:

#define NF_HOOK(pf, hook, skb, indev, outdev, okfn)

Where:

pf – is the protocol. It defines who calls the hook, so that the correct listeners would be notified. It could be PF_BRIDGE, PF_IPV6, PF_IPV4, etc.
hook – is one of the processing points listed above.
sbk – is a pointer to the current packet.
indev – is the network device that the packet was received in.
outdev – is the network device that is going to transmit the packet. Note that it could be NULL in some cases, before this decision was made, or if the packet’s destination is the local machine.
okfn – is a function which is called when the packet inspection of all listeners is complete, and all of them returned NF_ACCEPT, i.e. this packet is OK, continue processing. The OK Function is the final processing function of the specific networking sub-system. In most cases, the name of this function is xxx_action_finish.

In case the Netfilter sub-system is disabled, the NF_HOOK macro is expanded to call the okfn only.

Enabling Netfilter

Run “make menuconfig”, enter the “Networking support —> ” menu, then enter the “Networking options —>” menu, and enable the “Network packet filtering framework (Netfilter) —>” option. Additional configuration is available in this sub-menu, but nothing is needed to be changed for the purpose of enabling basic Netfilter functionality and registering your hooks. You will have to recompile your kernel after this change.

Registering Netfilter hooks

The linux/netfilter.h file typedefs the nf_hookfn callback type as follows:

typedef unsigned int nf_hookfn(unsigned int hooknum,
                               struct sk_buff *skb,
                               const struct net_device *in,
                               const struct net_device *out,
                               int (*okfn)(struct sk_buff *));

This function type must be implemented by all the listeners of the Netfilter hooks, and it has a similar set of fields to the NF_HOOK macro. I will show a simple implementation later.

The next thing we need to know about is the nf_hook_ops structure, also defined in the linux/netfilter.h file. This structure allows us to register our hook with the framework.

struct nf_hook_ops
{
        struct list_head list;

        /* User fills in from here down. */
        nf_hookfn *hook;
        struct module *owner;
        u_int8_t pf;
        unsigned int hooknum;
        /* Hooks are ordered in ascending priority. */
        int priority;
};

When we want to register a hook, we need to instantiate a variable of this type and use it to register. Here is the information we need to provide:

A pointer to our hook implementation (of type nf_hookfn)
A pointer to the owner – Use “THIS_MODULE” as an argument.
The protocol that we are interested in (one of the items I listed above).
The processing point (hook number) we are interested in.
Hook’s priority in ascending order – can be used to coordinate hooks ordering.

Let’s register two hooks with the framework. One hook will be called for each incoming packet (local in), and the other will be called for each outgoing packet (local out). Both hooks will be attached to the bridge; this will allow us to examine all packets which are received and transmitted by all the interfaces under the bridge. First, let’s declare the hooks prototypes:

static unsigned int rte_netfilter_local_in_hook( unsigned int hooknum, 
        struct sk_buff *skb,
        const struct net_device *in, const struct net_device *out,
        int (*okfn)(struct sk_buff *) );

static unsigned int rte_netfilter_local_out_hook( unsigned int hooknum, 
        struct sk_buff *skb,
        const struct net_device *in, const struct net_device *out,
        int (*okfn)(struct sk_buff *) );

Now, let’s instantiate the nf_hook_ops structure with our two hooks functions:

static struct nf_hook_ops rte_hook_ops[] __read_mostly =
{
    {
            .pf = NFPROTO_BRIDGE,
            .priority = 1,
            .hooknum = NF_INET_LOCAL_IN,
            .hook = rte_netfilter_local_in_hook,
            .owner = THIS_MODULE,
    },
    {
            .pf = NFPROTO_BRIDGE,
            .priority = 1,
            .hooknum = NF_INET_LOCAL_OUT,
            .hook = rte_netfilter_local_out_hook,
            .owner = THIS_MODULE,
    }
};

Note the __read_mostly declaration that may optimize access to this data. The last step is to call the Netfilter register function. The following example show both the register and unregister calls:

int rte_netfilter_init( void )
{
    int ret;

    /* Register netfilter hooks */
    ret = nf_register_hook(&rte_hook_ops[0]);
    ret |= nf_register_hook(&rte_hook_ops[1]);

    return ret;
}

void rte_netfilter_fini( void )
{
    /* Unregister hooks */
    nf_unregister_hook(&rte_hook_ops[0]);
    nf_unregister_hook(&rte_hook_ops[1]);
}

As soon as nf_register_hook returns successfully, our hooks will be called upon each packet reaching the specified processing point. Don’t forget to call the rte_netfilter_init function.

Implementing a Netfilter hook

In this section, I will provide a basic example which demonstrate a few operations that can be done on a packet. It is really up to you to decide what you want to do in each hook…

#include 
#include 
#include 
#include 
#include 
#include 

unsigned int rte_netfilter_local_out_hook( unsigned int hooknum, struct sk_buff *skb,
        const struct net_device *in, const struct net_device *out,
        int(*okfn)( struct sk_buff * ) )
{
    return NF_ACCEPT;
}

unsigned int rte_netfilter_local_in_hook( unsigned int hooknum, struct sk_buff *skb,
        const struct net_device *in, const struct net_device *out,
        int(*okfn)( struct sk_buff * ) )
{
    struct ethhdr *eth = eth_hdr(skb); /* Get a pointer to the Ethernet header */
    u_int16_t etype;

    printk( "%s: Received a packet %p, device in = %s\n", __func__,
                skb, in ? in->name : "" );

    /* Is this a multicast packet ? */
    if( is_multicast_ether_addr(eth->h_dest) )
    {
        /* Do something */
        printk( "Packet is multicast\n" );
    }

    if( is_broadcast_ether_addr(eth->h_dest) )
    {
        /* Do something else */
        printk( "Packet is broadcast\n");
    }

    /* Get EtherType field */
    etype = ntohs( eth->h_proto );

    if( etype == ETH_P_IP )
    {
        struct iphdr *ip = NULL;
        struct udphdr *udp = NULL;
        struct tcphdr *tcp = NULL;

        /* This is an IP packet */
        ip = ip_hdr(skb);
        if (ip == NULL)
        {
            return NF_ACCEPT;
        }

        if (ip->protocol == IPPROTO_UDP)
        {
            /* UDP packet */
            udp = (struct udphdr *)(skb_network_header(skb) + ip_hdrlen(skb));

            /* Do something here */
            printk( "UDP packet\n");
        }
        else if (ip->protocol == IPPROTO_TCP)
        {
            /* TCP packet */
            tcp = (struct tcphdr *)(skb_network_header(skb) + ip_hdrlen(skb));

            /* Do something here */
            printk( "TCP packet\n");
        }
    }

   /* We might decide to drop this packet, so we need to:
      * return NF_DROP;
      *
      * If we want to handle this packet ourselves, we need to:
      * return NF_STOLEN;
      */

    return NF_ACCEPT;
}

As we can see, the local out function does not do anything, and accepts all packets (i.e. allowing all packets to be transmitted). The local in function also accepts all packets, however, I added some example code that allows us to examine the Ethernet, IP, TCP and UDP headers (if they exist). It is really up to you to do whatever you want to do with this hook. I also specified 3 types of return values.

Running our Netfilter hook

Here’s the example code running in a loadable kernel module context. We can see plenty of incoming packets on device eth0. One of the packets is a broadcast UDP packet.

rte_netfilter_local_in_hook: Received a packet 872ae2c0, device in = eth0
rte_netfilter_local_in_hook: Received a packet 87b75380, device in = eth0
rte_netfilter_local_in_hook: Received a packet 87b74520, device in = eth0
rte_netfilter_local_in_hook: Received a packet 872ade20, device in = eth0
rte_netfilter_local_in_hook: Received a packet 872ade20, device in = eth0
Packet is multicast
Packet is broadcast
UDP packet
rte_netfilter_local_in_hook: Received a packet 87b65ca0, device in = eth0
rte_netfilter_local_in_hook: Received a packet 872aea40, device in = eth0
rte_netfilter_local_in_hook: Received a packet 87b68540, device in = eth0
rte_netfilter_local_in_hook: Received a packet 8787c800, device in = eth0
rte_netfilter_local_in_hook: Received a packet 87b67260, device in = eth0

Additional resources

Writing netfilter modules
Netfilter.org: Netfilter howto

Macros in the C programming language

Hai Shalom — Fri, 06 Jan 2012 22:04:46 +0000

Macros are very common and often used in C programming to declare constants, such as strings, addresses, or any other values such as maximum number of elements in an array, and so on. For those who are not familiar with them, Macros are declared by the #define keyword. Macros are also used to define some basic functionality given a set of 1 or more typeless parameters, similarly to an inline function.

There are some benefits by using Macros:

Code maintainability: Using a macro to define a constant provides an easy and correct method to ensure that if this value needs to be updated; all instances of this value will be automatically updated. Otherwise, the program may behave incorrectly, and even crash.
Code readability: Specifying a macro name is much more readable and understandable to someone who needs to read the code, than a plain number or address.

There are also some disadvantages:

Hard to extend: If the macros become complex and a change is required, any errors you introduce may yield vague compilation errors by the compiler, and always, the error line number points to the start of the macro.
Hard to debug: Debuggers often do not provide clear access and step ability to a code inside macros.

In this post, I’ll review some intermediate stuff about macros, and also show how we can use macros to automate code generation (often referred as X-Macro magic).

Some facts about macros:

Macros are expanded from the top of the file, to the bottom.
Macros can be defined, undefined and then redefined.
Macros, unlike other code, are not being compiled. You can write junk stuff as a macro and the file will be compiled. Due to the same reason, macros can contain any text, even if it is not declared or defined yet.
Macros are expanded only when used in the code. If you’ll use a junk macro in the code, it will be converted to junk which will fails your compilation.
Macros can span over only a single line. The only way to extend a macro to another line is by adding a new line marker ‘\’ in the end of the line. No other character is allowed after the new line marker.
You can always inspect the expanded macros by using the same compilation command, with the addition of the -E parameter. The output in this case, will be the preprocessed source code, where all the macros are expanded.

Basic Macro usage

Macro definition

In the C programming language, Macros are defined using the #define keyword. A Macro could be just defined, defined as a value, or defined as a piece of code. When it is defined as a value, of instances of this Macro’s name in the code will be replaced with this value, and only then, the code will be compiled. A Macro can also accept typeless parameter(s) or place holders. These Macro parameters will be replaced by actual variables or statements during macro expansion and then will be compiled. Here are some examples or a simple macro, a macro that holds a value, and a macro with a parameter that implements some logic. The last line shows how to cancel a macro (un-define). Please note that it is very important to have parentheses around every value.

#define ENABLE_MY_FEAUTRE
#define MAX_ITERATIONS   (4)
#define IS_POSITIVE( _x ) ( _x > 0 )

#undef ENABLE_MY_FEATURE

Conditional checks

Macros allow you to apply logic and conditional checks very easily using the #if and #ifdef keywords. You can ask if a macro is defined and implement some specific code for it, apply logic like OR, AND and NOT, and also comparisons like < > = and !=. It is required to end the conditional by an #endif keyword. You can also use #else, and #elif. Here’s an example:

#ifdef ENABLE_MY_FEATURE
/* Implement my feature */
...
#endif

#if (MAX_ITERATIONS > 5) && defined(ENABLE_MY_FEATURE)
/* Implement the better implementation */
#else
/* Do something else */
#endif

Intermediate Macro usage

The do-while(0) mystery

I bet you’ve encountered these weird definitions of macros, especially if you saw a header file from the Linux include directory. This wrapper opens a new name space and performs the loop only once, because the control enters the loop without any condition, and terminates it in the end because the condition is always false. This wrapper is used when the macro performs more than one command, and it ensures that all of them are performed, by preventing accidental conditional mistakes.

Let’s look on this example, where we check for a condition, and if true, we will use a macro that we’ll define shortly. The else is not that important now.

if( condition )
    DO_SOMETHING_HERE(x);
else
    ...

Now let’s define the DO_SOMETHING_HERE macro, and expand it ourselves:

#define DO_SOMETHING_HERE(_x) foo(_x); bar(_x);

if( condition )
    foo(_x); bar(_x);;
else
    ...

This is a compilation error because if condition is true, we will call foo, but bar will be always called and the if statement is terminated. But then, the else is not attached to an if. A problem.

Now, if will add curly parentheses around the macro:

#define DO_SOMETHING_HERE(_x) { foo(_x); bar(_x); } 

if( condition )
    { foo(_x); bar(_x); };
else
    ...

We will still have a compilation error because there is an extra semi-colon in the if statement, where there shouldn’t be one. You shall have to remove it if you want this version to compile.

Now let’s try the do-while(0) version:

#define DO_SOMETHING_HERE(_x) do{ foo(_x); bar(_x); }while(0) 

if( condition )
    do{ foo(_x); bar(_x); } while(0);
else
    ...

Now, the program compiles and works correctly.

Where am I?

When using gcc C compiler (and probably other compilers), there are some built in variables that can tell you which code is currently executing. Although this is not completely macro stuff, I added this here because they are very useful with macros. The following variables are available:

__FUNCTION__: Contains a string of the current function.

__LINE__: Contains the current line number.

__FILE__: Contains the name of the current source file.

If you’ll use these along with printf, you’ll be able to understand the exact origin of any message. Macros for example, can help you find who calls a specific function, and print a message each time it is called. All you have to do is add a ‘__’ prefix to your original function, and create a macro with the original function name. Inside this macro, use the 3 variables I just mentioned in a printf, and then call the original function (now starts with a ‘__’).

Stringification

One cool feature that macros can do, is to convert any code text to strings. This is a nice and an automatic way to convert any code text to readable and printable text, and it’s very informative. Basically, all that is required to convert something to a string is to add a ‘#’ sign before. However, it is common to use the following form when converting code to strings:

#define STR(_x)   #_x
#define XSTR(_x)  STR(_x)

There are two stages to this macro expansion. The reason is that if the parameter to XSTR needs to be expanded first, it will be done in this stage and the final output will be the contents of the parameter and not its name.

For example:

#define RT_EMBEDDED cool

STR(RT_EMBEDDED) /* Results in "RT_EMBEDDED" */
XSTR(RT_EMBEDDED) /* Results in "cool" */

Concatenation

Another cool feature is the ability to concatenate various code pieces to create new code (i.e. code generation). In order to concatenate, you need to specify the ‘##’, and it is also common usage to wrap this inside a macro. Here’s an example:

#define MACRO_CONCAT( _x, _y )   _x##_y
#define FUNC_PROTO( _handler ) int handler_func_##_handler( int );

MACRO_CONCAT( hello, dolly ) /* Results in hellodolly */
FUNC_PROTO( integer ) /* Results in: "int handler_func_integer( int );" It's a function prototype declaration */

The MACRO_CONCAT macro will create a new code which is the concatenation of what you write in _x and what you write in _y. Note that the macro can add more text, so the result could be more complex than a simple concatenation, but rather composition of some text along with _x and _y. The FUNC_PROTO macro generates a function prototype from the given keyword.

Multiple arguments

Macros also support list of parameters of a dynamic size, similarly to the printf functions group, where the list of arguments to it can be of any size. It is done by specifying the ‘…’ sign. The following example, taken from the pcd source code, shows how we can wrap a call to printf to add some more information:

extern bool_t verboseOutput;

#define PCD_PRINT_PREFIX                            "pcd: "
#define PCD_PRINTF_STDOUT( _format, _args... )        \
    do { if( verboseOutput ) fprintf( stdout, "%s"_format "%s", PCD_PRINT_PREFIX, ##_args, ".\n" ); } while( 0 )

When the macro is used, the expanded version check if verboseOutput is true, and only if true, it formats a string to output on the console with the “pcd :” prefix. It also adds a point and a new line in the end of the string. In this way, all printouts from this program will have the same look and feel.

Return value

Macros can also be used to do some computation and to “return” a value. This is not actually a classic return value like in a function call, but rather a result of a flow. The following example shows a macro that checks if a number is even or odd. In the following example, the macro’s return value is a string. We use the string to print our conclusion.

#include 
#include 

#define IS_EVEN_STR( _x ) \
    ( _x & 1 ? "odd" : "even" )

int main( int argc, char *argv[] )
{   
    int val;

    if(argc<2)
        return;

    /* Convert to integer */
    val = atoi(argv[1]);

    /* Print our conclusion */
    printf( "The number %d is %s\n", val, IS_EVEN_STR(val));

    return 0;
}

The output from this program is:

$ ./even 45
The number 45 is odd
$ ./even 64
The number 64 is even

Advanced Macro usage

Assertion

The following set of macros are very useful for debugging user space applications. You should actually use only the MY_ASSERT macro. The rest are helper macros. I designed it to operate as follows; in case the assertion fails (the condition is false), the macro will print a very detailed error message, including the file name, function name and line number, and then, it will terminate the program. You can add calls to this macro in every location of your program where a sanity check is required. The macro takes a condition to check, and format with arguments to display a message in case the condition fails.

/* Crash the process */
#define __CRASH()    (*(char *)NULL)

/* Generate a textual message about the assertion */
#define __BUG_REPORT( _cond, _format, _args ... ) \
    fprintf( stderr, "%s:%d: Assertion error in function '%s' for condition '%s': " _format "\n", \
    __FILE__, __LINE__, __FUNCTION__, # _cond, ##_args ) && fflush( NULL ) != (EOF-1)

/* Check a condition, and report and crash in case the condition is false */
#define MY_ASSERT( _cond, _format, _args ... ) \
do { if(!(_cond)) { __CRASH() = __BUG_REPORT( _cond, _format, ##_args ); } } while( 0 )

Now let’s take a look on this short program. It asserts that the number of parameters must be at least 3, but not more than 4. The call to my assertion macro will guarantee this condition, or else, it will display an error message, and terminate the program.

#include 
#include 

#define MIN_PARAMS 3
#define MAX_PARAMS 4

int main( int argc, char *argv[] )
{
    int params = argc - 1;
    MY_ASSERT( params >= MIN_PARAMS && params <= MAX_PARAMS,
        "Invalid parameters! must specify at least %d parameters, where %d specified", MIN_PARAMS, params );
    return 0;
}

Let’s see the output for various parameters:

$ ./macro 1 2
macro.c:21: Assertion error in function 'main' for condition 'params >= 3 && params <= 5': Invalid parameters! must specify at least 3 parameters, where 2 specified
Segmentation fault
$ ./macro 1 2 3
$ ./macro 1 2 3 4
$ ./macro 1 2 3 4 5
macro.c:21: Assertion error in function 'main' for condition 'params >= 3 && params <= 4': Invalid parameters! must specify at least 3 parameters, where 5 specified
Segmentation fault

Code Generation

I bet you thought the assertion macro is cool. There’s even cooler usage of macros, called Code Generation. Code generation is real magic. It may not be readable and maintainable at all, but on the other hand, it may automate many things, prevent human mistakes, and it is very useful when things are added in patterns. If you detect that some things you do in the code are repeating, or if you have to do the same or similar action per each object, you may be able to simplify things by using macros.

Here’s an example. Suppose you have a list of keywords. You would like to create an enumeration for these keywords, and for each keyword you need to define a callback function, and a flag. You would also like to have every keyword as a string for other purposes. Without macros, you will have to do the same actions for each and every new keyword you’ll add. It is also error prone, because there could be a mismatch between the enumeration and other things in case you forgot to add it there.

Here are my keywords (taken from the pcd project as an example): RULE, START_COND, COMMAND, END_COND, END_COND_TIMEOUT, FAILURE_ACTION, ACTIVE, SCHED, DAEMON, USER, VERSION and INCLUDE. The last 5 keywords are not mandatory for the implementation – This is a property flag which is attached to each keyword.

The first step we need to do is to create a logical list that will contain all the keywords. Inside the list, we will put each keyword in an internal macro, along with any other information we would like to include with this keyword. In our example it’s the mandatory flag setting. The internal macro is our tool to selectively extract the information provided in each line. Remember, the internal macro is not defined yet. We will define it according to what we would like to generate. This is not a compilation error because the list macro is not expanded until it is used. Note that we are not obligated to use the entire information in each line.

/* Keyword,        Mandatory */
#define PCD_PARSER_KEYWORDS \
    PCD_PARSER_KEYWORD( RULE,               1 )\
    PCD_PARSER_KEYWORD( START_COND,         1 )\
    PCD_PARSER_KEYWORD( COMMAND,            1 )\
    PCD_PARSER_KEYWORD( END_COND,           1 )\
    PCD_PARSER_KEYWORD( END_COND_TIMEOUT,   1 )\
    PCD_PARSER_KEYWORD( FAILURE_ACTION,     1 )\
    PCD_PARSER_KEYWORD( ACTIVE,             1 )\
    PCD_PARSER_KEYWORD( SCHED,              0 )\
    PCD_PARSER_KEYWORD( DAEMON,             0 )\
    PCD_PARSER_KEYWORD( USER,               0 )\
    PCD_PARSER_KEYWORD( VERSION,            0 )\
    PCD_PARSER_KEYWORD( INCLUDE,            0 )

Now, we can start generating code out of this list. Let’s start by generating an enumeration in the header file. The following code automatically generates an enum from the list. In this example, we define PCD_PARSER_KEYWORD as a macro that takes two parameters (because that’s how we declared it on the list), but it actually uses only one, the keyword. Using concatenation with the SET_KEYWORD_ENUM macro, we create a list of names, all of them have the PCD_PARSER_KEYWORD_ prefix.

/***********************************************
 * Keyword enumeration
 ***********************************************/
#define SET_KEYWORD_ENUM(x) \
 PCD_PARSER_KEYWORD_##x

#define PCD_PARSER_KEYWORD( keyword, mandatory ) \
 SET_KEYWORD_ENUM( keyword ),

typedef enum parserKeywords_e
{
    PCD_PARSER_KEYWORDS

    PCD_PARSER_KEYWORD_LAST

} parserKeywords_e;

#undef PCD_PARSER_KEYWORD

That’s the preprocessed code output (actually, the real output does not have a new line after each keyword, which I added here for cosmetic purposes only. Compilers ignore white spaces anyway):

typedef enum parserKeywords_e
{
    PCD_PARSER_KEYWORD_RULE,
    PCD_PARSER_KEYWORD_START_COND,
    PCD_PARSER_KEYWORD_COMMAND,
    PCD_PARSER_KEYWORD_END_COND,
    PCD_PARSER_KEYWORD_END_COND_TIMEOUT,
    PCD_PARSER_KEYWORD_FAILURE_ACTION,
    PCD_PARSER_KEYWORD_ACTIVE, P
    CD_PARSER_KEYWORD_SCHED,
    PCD_PARSER_KEYWORD_DAEMON,
    PCD_PARSER_KEYWORD_USER,
    PCD_PARSER_KEYWORD_VERSION,
    PCD_PARSER_KEYWORD_INCLUDE,

    PCD_PARSER_KEYWORD_LAST

} parserKeywords_e;

Now, we can generate the callback functions’ prototypes. In order to take a keyword and generate a full function declaration, we will have to add for each keyword a prefix (like the return value, and a standard name prefix) and a suffix (function parameters and a semicolon). In this example, all function are static, and all return “int32_t”. The common prefix name is PCD_parser_handle_. All functions take a char* line as a single parameter. Here’s what we do:

/**************************************************
 * Declarations for the keyword handlers.
 **************************************************/
#define SET_HANDLER_FUNC(x)   PCD_parser_handle_##x
#define PCD_PARSER_KEYWORD( keyword, mandatory )\
    static int32_t SET_HANDLER_FUNC( keyword ) ( char *line );

PCD_PARSER_KEYWORDS

#undef PCD_PARSER_KEYWORD

That’s the preprocessed output (this time I did not add too much new lines):

static int32_t PCD_parser_handle_RULE ( char *line ); static int32_t PCD_parser_handle_START_COND ( char *line );
static int32_t PCD_parser_handle_COMMAND ( char *line ); static int32_t PCD_parser_handle_END_COND ( char *line );
static int32_t PCD_parser_handle_END_COND_TIMEOUT ( char *line ); static int32_t PCD_parser_handle_FAILURE_ACTION ( char *line );
static int32_t PCD_parser_handle_ACTIVE ( char *line ); static int32_t PCD_parser_handle_SCHED ( char *line );
static int32_t PCD_parser_handle_DAEMON ( char *line ); static int32_t PCD_parser_handle_USER ( char *line );
static int32_t PCD_parser_handle_VERSION ( char *line ); static int32_t PCD_parser_handle_INCLUDE ( char *line );

Note that you shall have to implement the functions; there is a limit to what a macro can generate. However, if the functions are also made out of patterns, you can generate them as well! Here’s a hint: for each PCD_PARSER_KEYWORD that we declared in the table, you can use the information in each line to construct an entire function. You can decide what to do in run time for each keyword, based on the information provided in the line, or you can decide in compile time in the following way; If you have for example 2 types of handler functions, you can declare keywords that need the first type of handler using PCD_PARSER_KEYWORD_HANDLER1 macro, and the rest with PCD_PARSER_KEYWORD_HANDLER2 macros. Then, use these macros to generate different functions for each keyword group.

Now, in the source file, suppose we have a structure that holds the keyword string name, it’s callback function pointer, and the flag value. It can also hold other information that we do not initialize now. We can construct this structure automatically using the following code. Our internal macro now fills each line of the structure:

typedef struct configKeywordHandler_t
{
    char      *name;
    int32_t     (*handler)(char *line);

    /* set at run time. */
    u_int32_t    parse_flag;

    /* indicate if this is a mandatory field. */
    u_int32_t    mandatory_flag;

} configKeywordHandler_t;

/**************************************************************************
 * Initialize keyword array
 **************************************************************************/
#define PCD_PARSER_KEYWORD( keyword, mandatory ) \
 { XSTR( keyword ), SET_HANDLER_FUNC( keyword ), 0, mandatory },

configKeywordHandler_t keywordHandlersList[] =
{
    PCD_PARSER_KEYWORDS
    { NULL,       NULL,          0, 0},
};

#undef PCD_PARSER_KEYWORD

Note that the XSTR call converts the keyword to a string, and the SET_HANDLER_FUNC generates a function name.

That’s the code preprocessed output (again, I added new lines after each line. You can see that NULL is expanded to ((void *)0) ):

configKeywordHandler_t keywordHandlersList[] =
{
    { "RULE", PCD_parser_handle_RULE, 0, 1 },
    { "START_COND", PCD_parser_handle_START_COND, 0, 1 },
    { "COMMAND", PCD_parser_handle_COMMAND, 0, 1 },
    { "END_COND", PCD_parser_handle_END_COND, 0, 1 },
    { "END_COND_TIMEOUT", PCD_parser_handle_END_COND_TIMEOUT, 0, 1 },
    { "FAILURE_ACTION", PCD_parser_handle_FAILURE_ACTION, 0, 1 },
    { "ACTIVE", PCD_parser_handle_ACTIVE, 0, 1 },
    { "SCHED", PCD_parser_handle_SCHED, 0, 0 },
    { "DAEMON", PCD_parser_handle_DAEMON, 0, 0 },
    { "USER", PCD_parser_handle_USER, 0, 0 },
    { "VERSION", PCD_parser_handle_VERSION, 0, 0 },
    { "INCLUDE", PCD_parser_handle_INCLUDE, 0, 0 },
    { ((void *)0), ((void *)0), 0, 0},
};

What did we gain here?

Big amount of generated code that we didn’t have to write ourselves.
No mistakes, no compilation errors.
Adding a new keyword requires only a single line addition in the main table!

Conclusions

Once you grasp the idea of macros, understand how to see the patterns, and the fact that the macros are not expanded until they are used, you could create wonderful things with these macros, such as this example.

You can download the pcd project source code here, and continue to review the usage of code generation.

Resolving/Debugging memory corruption

Hai Shalom — Mon, 16 May 2011 15:00:42 +0000

Memory corruption is a scenario where a given buffer (or memory area) is unintentionally modified by an unknown piece of code. This data change causes the rightful users of the buffer to receive bad information which could either modify their behavior or even crash the program. It is usually very hard to find the root cause of memory corruptions because the corruption itself may take place asynchronously to the actual use of the buffer, thus when it actually happens, the crash may occur seconds or minutes later.

In this post I will present one of the ways that may help you catch such corruptions using memory protections.

In most complicated memory corruption scenarios, a buffer is passed along multiple functions, some of them may reside in external libraries (it can get even more complicated if you don’t have all sources). Due to the asynchronous nature of the effects of memory corruption it won’t help if you’ll place a break point when the rightful users access the buffer, because it’s already too late.

In order to help us catch the code that writes to our buffer, we utilize memory protections. The idea is simple. First, we need to allocate memory for the buffer, then, we need to somehow make it “read-only”, and lastly, let the program run using our read-only version of the buffer. When some code tries to write something to it, we will be notified.

Linux allows us to assign different protection properties to different pages in memory. Therefore, we can create our own page, allocate the required amount of memory for our buffer, initialize it with the correct data and then lock it for reading only. Once it is locked, we can pass on this buffer and see what happens. In case some code tries to write to this region, the program will get a SIGSEGV exception and terminate. This will prevent the corruption on one hand, but will not let us know yet who did it. In order to figure it out, we must use a signal handler which will handle this exception and provide some more information. In case your project does not have a process controller, a recommended option is to use pcd; otherwise, you will have to write your own signal handler. With the information provided by the signal handler, we will get the address of the instruction that tried to write to our buffer, and using addr2line application, we could retrieve the source code file name and line. In case you don’t have the source code, this is a good proof to your library provider, that there is a bug in his library.

As an example, I wrote two helper functions that will help us facilitate this ability. The first one; rte_malloc, has a similar interface to standard malloc. However, instead of allocating the size of your buffer, it allocates a complete page and places the buffer inside of it. The reason is because memory protection can be applied only on pages and the buffer address must be page aligned. The second function; rte_protect, activates the required protection on the buffer, either read, write or both. You can call this function anytime and dynamically change the protection on your buffer during your investigation. Here’s the source code of these functions:

/* Tracking buffer abuse using read/write protection.
 * Written by Hai Shalom.
 *
 * http://www.rt-embedded.com/
 */

#include 
#include 
#include 
#include 
#include     /* for PAGESIZE */
#ifndef PAGESIZE
#define PAGESIZE 4096
#endif

void *rte_malloc( size_t size )
{
  char *buf;

  /* Allocate memory, reserve space for a full page */
  buf = malloc( size + PAGESIZE - 1 );

  if( !buf )
    return NULL; /* Failed to allocate memory */

  /* Align pointer to a multiple of PAGESIZE */
  buf = (char *)(((int) buf + PAGESIZE - 1 ) & ~( PAGESIZE - 1 ));

  return buf;
}

int rte_protect( void *buf, int protect_read, int protect_write )
{
  int prot = PROT_READ | PROT_WRITE; /* Default buffer protection */

  /* Configure protection */
  if( protect_read )
    prot &= ~PROT_READ;

  if( protect_write )
    prot &= ~PROT_WRITE;

  /* Now, buffer is aligned, apply desired protection */
  return mprotect( buf, PAGESIZE, prot );
}

Now let’s write a short program to demonstrate how to use it. It is assumed that the above functions are available to it. I also instrumented it with the pcd’s exception handlers which you can either use if you have pcd running in your target, or replace with your own implementation:

/* Optional, required only if PCD is used */
#include "pcdapi.h"

#define BUF_SIZE 100

int main( int argc, char *argv[] )
{
    char *buf;
    char c;

    /* Register to the PCD exception handler.
     * Note that this is optional, and added only for
     * getting a detailed signal information.
     */
    PCD_API_REGISTER_EXCEPTION_HANDLERS();

    /* Allocate aligned memory */
    buf = rte_malloc( BUF_SIZE );

    if( !buf )
    {
        perror("Failed to allocate memory");
        return errno;
    }
    else
    {
        printf( "Buffer address = %p\n", buf );
    }

    /* Write OK */
    buf[ 0 ] = 1;

    /* Read OK */
    c = buf[0];

    /* Now apply write protection */
    if( rte_protect( buf, 0, 1 ) )
    {
        perror( "Failed to apply protection" );
        return errno;
    }

    /* Read OK */
    c = buf[ 0 ];

    /* Write protected, expect program to terminate */
    buf[ 0 ] = 1;

    return 0;
}

As we can see from this example, we allocated a buffer in the size of 100, wrote a byte and read a byte. Then, we applied the write protection and tried to repeat these action. The program will receive a SIGSEGV signal when it will try to write the byte in the second time. Now let’s review the crash dump from pcd (taken in MIPS platform):

/tmp # ./protect
Buffer address = 0x412000
/tmp #
**************************************************************************
**************************** Exception Caught ****************************
**************************************************************************

Signal information:

Time: Thu Jan  1 10:55:12 1970
Process name: ./protect
PID: 2283
Fault Address: 0x412000
Signal: Segmentation fault
Signal Code: Invalid permissions for mapped object
Last error: Success (0)
Last error (by signal): 0

MIPS registers:

regmask=0x2ab8d510
status=0x0000003b
pc=0x00400940
zero=0x00000000
at=0x00000001
v0=0x00000000
v1=0x2ab1d000
a0=0x00412000
a1=0x00001000
a2=0x00000001
a3=0x00000000
t0=0x00000000
t1=0x81730470
t2=0x00000000
t3=0x00000000
t4=0x00000000
t5=0x81f145f0
t6=0x00100071
t7=0x81f145f0
s0=0x00412000
s1=0x00000001
s2=0x7fb930b8
s3=0x7fb93174
s4=0x00000001
s5=0x00400614
s6=0x004008c8
s7=0x00000000
t8=0x00000000
t9=0x2ab2a040
k0=0x00000000
k1=0x00000000
gp=0x00418b60
sp=0x7fb93078
fp=0x0045a3ac
ra=0x00400934

Maps file:

00400000-00401000 r-xp 00000000 00:09 170737     /tmp/protect
00410000-00411000 rw-p 00000000 00:09 170737     /tmp/protect
00411000-00412000 rwxp 00000000 00:00 0
00412000-00413000 r--p 00000000 00:00 0
2aaa8000-2aaad000 r-xp 00000000 1f:01 59         /lib/ld-uClibc-0.9.30.so
2aaad000-2aaae000 rw-p 00000000 00:00 0
2aabc000-2aabd000 r--p 00004000 1f:01 59         /lib/ld-uClibc-0.9.30.so
2aabd000-2aabe000 rw-p 00005000 1f:01 59         /lib/ld-uClibc-0.9.30.so
2aabe000-2aac0000 r-xp 00000000 1f:01 77         /lib/libpcd.so
2aac0000-2aacf000 ---p 00000000 00:00 0
2aacf000-2aad0000 rw-p 00001000 1f:01 77         /lib/libpcd.so
2aad0000-2aad2000 r-xp 00000000 1f:01 81         /lib/libipc.so
2aad2000-2aae1000 ---p 00000000 00:00 0
2aae1000-2aae2000 rw-p 00001000 1f:01 81         /lib/libipc.so
2aae2000-2ab0c000 r-xp 00000000 1f:01 83         /lib/libgcc_s.so.1
2ab0c000-2ab1c000 ---p 00000000 00:00 0
2ab1c000-2ab1d000 rw-p 0002a000 1f:01 83         /lib/libgcc_s.so.1
2ab1d000-2ab75000 r-xp 00000000 1f:01 68         /lib/libuClibc-0.9.30.so
2ab75000-2ab84000 ---p 00000000 00:00 0
2ab84000-2ab85000 r--p 00057000 1f:01 68         /lib/libuClibc-0.9.30.so
2ab85000-2ab86000 rw-p 00058000 1f:01 68         /lib/libuClibc-0.9.30.so
2ab86000-2ab8b000 rw-p 00000000 00:00 0
7fb7f000-7fb94000 rwxp 00000000 00:00 0          [stack]

**************************************************************************

As we can see from the program printout, the buffer address is 0×412000. Now let’s examine the map file (which is also accessible while the program is running by running cat /proc//maps). We can see that there is a region in the range of 0×412000 – 0×413000 which has the “r–p” protection bits. This information corresponds to our buffer address and applied protection, and in fact, this line describes the page our buffer was allocated from. Next, we can see that the reported fault address is also 0×412000. It means that the program crashed while accessing this address. This also corresponds with our program, because we indeed tried to access buf[ 0 ]. The last piece of information is the program counter register (pc) which holds the address of the instruction that caused the memory write. The pc shows that the instruction address is 0×00400940, which suggests that it’s inside our application and not in any of the linked libraries. We can now use addr2line application to match the address to a file name and line number:

mips-linux-uclibc-addr2line -e /home/hai/protect 0x00400940
/home/hai/protect.c:93

Line 93 indeed matches to the write memory access in the program, after the protection was enabled.

For conclusion, although this example was very simple, you can apply the same technique for large programs, by dynamically enabling and disabling the memory protection. For example, disable the protection for pieces of the code that you examined and found out that are OK, and enable for the rest of the program.

This memory protection technique can be also used to resolve other kinds of problems. For example, use after free. Use after free is a scenario where a piece of code tries to access portion of memory that was already freed and might read garbage (although sometimes that right data may still be there, that makes it even worse). In order to do that, replace the call for free by enabling read protection. This will catch the code that tries to access this memory area.

Another example is memory overrun detection. In this case, a piece of code write more bytes than the buffer was designed to hold, and might overwrite some other data. In this case, you can allocate two pages of memory. Enable write protection to the second page, and allocate the buffer from the end of the first page. In this case, when a memory overrun will occur, it will be written in the first byte of the second page, thus triggering the SIGSEGV signal.

You can download the source code for this example here.

How to write u-boot scripts

Hai Shalom — Tue, 10 May 2011 00:32:02 +0000

U-boot is one of the most popular boot-loaders that are used in the industry. It supports plenty of platforms and it’s open source. There are many other advantages of using u-boot, however, in this post I will present the concept of u-boot scripts (or macros) which could improve your productivity (and your life…) if you are a programmer or a user of u-boot and do many repeating operations.

We all know that when we work on a project, we need to perform various repeating operations, such as erasing a partition in the flash, loading an image, loading parameters block, or any other repeating operation. These operations usually are composed of several commands which also require us to type addresses in the memory. This operation is inconvenient and error prone. There are some cases where the memory map changes, and the commands which were used are invalid and need to be updated.

The u-boot supports scripts (or macros), which are similar to batch/shell scripts. Using these scripts, you can define new commands and actually concatenate many other commands to form one multi-step command. These scripts are saved as part of the environment variables and are visible when running the “printenv” command. Using the same method, we can also define memory addresses or any other values which will be computed by the code and set in the environment.

Let’s take an example of how this method could improve your user experience: Loading an image (either kernel image, filesystem or a unified image). Assuming our RAM address starts at 0×80000000 and our image address in the flash starts at 0x9f500000, we would have to run the following commands in order to load an image:

# tftpboot 0x81000000 image.img

TFTP from server 192.168.1.10; our IP address is 192.168.1.1
Filename 'image.img'.
Load address: 0x81000000
Loading: #################################################################
 #################################################################
 #################
done
Bytes transferred = 750731 (b748b hex)

# erase 0x9f500000 +${filesize}

Erasing flash... ............
Erased 12 sectors

# cp.b ${fileaddr} 0x9f500000 ${filesize}

Copy to Flash... ...........done

Note that u-boot initializes the filesize and fileaddr right after the TFTP session terminates. As mentioned, it requires 3 steps, and requires us to know the destination addresses. It might get complicated if we have multiple files to load each time, thus we are required to repeat these commands with different file names and addresses.

As first improvement, we can create a script (in the form of a new environment variable) which will be called update. This script can do this work in a single command:

# set update "tftpboot 0x81000000 image.img && erase 0x9f500000 +"\${filesize}" 
       && cp.b "\${fileaddr}" 0x9f500000 "\${filesize}""

# saveenv

Note that this command is made of concatenating 3 commands using the && operator, which can also be used in the u-boot prompt. We also saved this command in the environment using the “saveenv” command. This may not be applicable for platforms without environment sectors. In this case, you’ll have to use the other method which I will present later. Now, we can get the image updated in a single command:

# run update

TFTP from server 192.168.1.10; our IP address is 192.168.1.1
Filename 'image.img'.
Load address: 0x81000000
Loading: #################################################################
 #################################################################
 #################
done
Bytes transferred = 750731 (b748b hex)

Erasing flash... ............
Erased 12 sectors

Copy to Flash... ...........done

This is already better. However, this command has hard coded values for file name and address. The next step is to replace these hard coded values with variables, and using recoursive calls. The addresses and file names can be stored in other environment variables, and we can create another script that initializes these values as parameters to our update script. Here’s the updated version of update that uses only vairables (assuming the 0×81000000 address never changes). The updatek script updates the kernel image and the updatefs script updates the file system image. Both initialize the appropriate file name and address, and call update.

# set update "tftpboot 0x81000000 "\${filename}" && erase "\${loadaddr}" 
      +"\${filesize}" && cp.b "\${fileaddr}" "\${loadaddr}" "\${filesize}""

# set updatek "filename=zImage.uImage loadaddr=0x9f500000 && run update"

# set updatefs "filename=rootfs.img loadaddr=0x9f800000 && run update"

# saveenv

Now our update script is generic and can be further enhanced for other images or any other actions. The next step is to plant these scripts inside the u-boot binary image. Otherwise, we will have to manually initialize them in each new target we receive and this is not what we want. Furthermore, platforms without environment variables in the flash may not be able to save them at all.

In u-boot, each platform has a configuration header file which is stored in the include/configs directory. Locate your platform’s header file by reading the include/config.h file. This file configures your platform, and also configures the default environment variables settings. The u-boot has default variables that will be initialized during the first boot on a new board and we can enhance this list very easily using the CONFIG_EXTRA_ENV_SETTINGS macro. The u-boot code in environment.c file will use all the values in this macro and enhance the default variable settings. For the purpose of our example, we may want to add at least 3 lines to this macro that define update, updatek and updatefs. We can add further variables to define file names and addresses if they are dynamically auto-detected or changed by some logic (for example, if the code detects various flash devices with different sizes). For the example, we will add the following:

#define ENV_UPDATE \
 "update=tftpboot " MK_STR(CFG_LOAD_ADDR) " ${filename} && " \
 "erase ${loadaddr} +${filesize} && "    \
 "cp.b ${fileaddr} ${loadaddr} ${filesize}\0"

#define ENV_UPDATE_KERNEL \
 "updatek=loadaddr=0x9f500000 && filename=zImage.uImage && " \
 "run update\0"

#define ENV_UPDATE_FS \
 "updatefs=loadaddr=0x9f800000 && filename=rootfs.img && " \
 "run update\0"

#define CONFIG_EXTRA_ENV_SETTINGS \
 ENV_UPDATE \
 ENV_UPDATE_KERNEL \
 ENV_UPDATE_FS \
 ""

Note that each command must end with a NULL (\0) and the last entry in the CONFIG_EXTRA_ENV_SETTINGS macro must be empty. As I mentioned, the addresses in the macros can be environment variables as well.

Now, after we recompile u-boot, these new 3 macros will be added to the environment variables on new boards, or on all boards without a saved environments. For all other boards, you will have to erase the environment sector and reset to take effect.

References

http://www.denx.de/wiki/U-Boot

How to setup an NFS client on the target

Hai Shalom — Tue, 01 Feb 2011 23:03:24 +0000

The NFS (Network Filesystem) is a very common Linux feature. It enables any Linux machine to mount directories which are not physically available in the machine’s hardware, but located somewhere else, and reachable only via a network interface. Mounting an NFS allows us to extend the storage capabilities of the mounting machine beyond its physical storage limitations. The NFS mount can be used to store additional software or data. For embedded devices, NFS is usually used for debugging purposes, mostly because these products must operate without network availability and must contain all required software in their storage device. With limited storage capabilities, NFS is a great way to extend the machine’s storage capabilities without changing any hardware. Make sure that you grant access to the IP address of your target to the requested directory.

Step 1: Setup an NFS Server

The first step is to setup an NFS server. There are many good articles of how to do that in any standard Linux distribution, and there are also nice solutions for Windows hosts (although not very recommended). Checkout the reference section, and setup your NFS server. Note that if you don’t have a Linux machine or are not allowed to change the settings in your existing machine, you can always create a virtual machine and use it as your NFS server. I personally recommend VirtualBox, which is an open source software from Oracle.

Step 2: Enable NFS support in your Kernel

In order to enable NFS, the Kernel you are using on your target must support it. Usually, this option is disabled by default in order to save space of an unneeded feature. In order to enable it, run “make menuconfig” and enter the “File systems —>” sub menu first.

Select the “Enable POSIX file locking API“, which is required for NFS. Make sure there is a * inside the brackets. Now, select the “Network File Systems —>” option and get inside this sub menu. Enable both “NFS client support” and “NFS client support for NFS version 3“. Whilst the former enables NFS client support, the latter implements the version 3 of the protocol which might be mandatory for some recent servers. You can also enable the version 4 support if your server supports it, although the code maturity in older kernels is experimental. There is no need to enable the NFS server options here for client support. The NFS server configuration here is only if you want your target machine to be an NFS server, which is not what you want in most cases. Now exit and save your configuration and recompile your kernel.

Step 3: Compile port mapper utility

The port mapper utility is another essential piece of software you must have in order to enable NFS support. It implements RPC (Remote Procedure Call) layer that NFS uses. Currently, there is no support for it in BusyBox, so you must download, compile and install it manually in your target. You can download the latest version here. In order to compile it for my embedded target, I had to comment the line that includes tcpd.h in pmap_check.c file, and compile it while defining CC to be my cross compiler and NO_TCP_WRAPPER=y. The Makefile generates three executable files: portmap, pmap_dump, and pmap_set. Copy all three to your target’s file system image in /bin directory.

Step 4: Enable NFS mount in BusyBox

By default, the NFS mount is disabled in BusyBox in order to save space. However, we must enable it so it will include the required code to mount an NFS directory. In the BusyBox directory, run “make menuconfig“. Enter the “Linux System Utilities —>” sub menu and enable the “Support mounting NFS file systems” option under the “mount” option.

Now exit and save your BusyBox configuration and recompile BusyBox.

Step 5: Runtime configuration

Once your updated kernel and filesystem are loaded in your embedded device, and your networking interface is up, it is time to mount your NFS directory. Assuming you’ve done the configuration correctly in your server, all you have to do is the following:

Run the port mapper utility: Just type “portmap” in your shell.
Create a mounting point; a directory which will be the gate to your NFS directory. You can either create a mounting point in /var/ or /tmp/ directories using the mkdir command. For example: “mkdir /var/mnt“.
Mount an NFS directory using the mount command. This command accepts the filesystem type (NFS), the IP address of your NFS server, the source directory and the mount point (destination) directory. For example, assuming my NFS server is on 192.168.1.100, and I exported the /export/hais-stuff, my mount command would be: “mount -t nfs 192.168.1.100:/export/hais-stuff /var/mnt“.

Now I can change directory to /var/mnt and use it as if it was a standard storage on my embedded device. Note that any file changes you’ll perform in the server will be seen in the target as well.

References

VirtualBox - http://www.virtualbox.org/
Setup an NFS server in Ubuntu – https://help.ubuntu.com/community/SettingUpNFSHowTo
Setup an NFS server in Fedora – http://www.howtoforge.com/setting-up-an-nfs-server-and-client-on-fedora-13
Port mapper utility – http://neil.brown.name/portmap/
NFS FAQs – http://nfs.sourceforge.net/

A typical embedded system description

Hai Shalom — Tue, 14 Sep 2010 15:32:06 +0000

Today’s consumer electronics, networking and communication devices are small and highly sophisticated computers (or Systems-on-Chips; SoCs). Each chip contains numerous peripherals and special features that require the right software and drivers to drive them correctly and efficiently. In this article, I am going to describe in high level the system’s hardware components and the system’s software components. I will also describe what they do in high level. This article is meant for people who want to get a high level introduction to the Real-Time Embedded systems.

The System’s hardware in high-level

A final product is based on an optimal board design, which is usually small and contains only the minimum hardware that is required for it. However, development boards (EVMs) are usually larger and contain many debug ports and additional hardware to support the development. An example for it, is the JTAG port which allows to connect a hardware emulator (debugger) that connects a special software in the PC to the target board and allows the user to read and modify memory and registers, and debug the system (steps, breakpoints, etc.). Another useful debug port is the serial console port (RS232) which allows the running programs to display information and logs on the console. There are cases where vendor remove these ports in the final product in order to reduce manufacturing costs and prevent access to hackers.

A typical system, as seen in the picture is usually composed of the following items:

A target CPU: There are many target CPUs available today, each one with its own characteristics. The CPU is selected according to the requirements of the final product, such as processing power, power consumption, opcode compatibility and more.
Main RAM: The main memory of the system which stores all the code and data in run-time. There are many types of RAM, such as SDRAM, DDR, DDR2, DDR3, etc. where each device has its own characteristics. Similarly to the CPU, the RAM device is selected according to the system’s requirements and limitations.
Storage device: Many embedded systems store the code in a storage device such as a Flash device or a Disk. There are many types of devices in various speeds, capacities and data retention abilities.
Power connector: Some boards require DC voltage in order to operate, in this case, they will be provided with an external power supply unit and the power connector’s shape will be round. Other designs has a built-in power supply and need to be connected directly to the mains power. In such designs, the board will have large capacitors and coils to convert and regulate the power. Note that in this case, you need to make sure that the mains voltage in your location matches the board design (connecting 220v to a 110v design could cause electric shock!). Moreover, this design exposes you to electrical shock if you’ll touch the power supply area, so be careful.
Expansion sockets: Various board designs implement expansion sockets, such as eSATA, PCIe and other proprietary connections that allows the user to enhance the features of the board with minimum hardware changes (for example, connect an LCD display or another Chip).
Debug ports: Such as Serial port (console, RS-232), JTAG port or any other port.
Communication ports: The board in the picture features an Ethernet port, a 4-port Ethernet switch, and a USB host port. Other consumer electronics devices might have different types of ports, such as coaxial, optical, telephony, USB device or any other proprietary port (like in Apple’s devices). Wireless board devices also contain one or more antennas.
Reset button: Most boards contain such a button, which resets the system, without the need to physically power-cycle it.
LEDs: Most board design contain one or more LEDs (Light-Emitting-Diodes). These small lights are used as indication or action marks that signal a specific activity, according to the design of the system.

The System’s software in high-level

Now that we have reviewed the hardware in high-level, let’s review the software in high-level. A typical system is composed of the following components:

A boot-loader: A short binary code which initializes the basic hardware, and prepares the system for the Operating System startup. The boot-loader usually resides in the first sector(s) of the flash (or storage device) and it is usually very small in size (typically 64KB – 128KB).
An image: A binary image which contains all the code that runs the product. In some cases, the image is composed of a big linked-executable file, and in other cases (such as Linux OS), the image is made of a Kernel and one of more file-systems which hold all the programs and files. Some products require that there will be two images stored on the storage device, for backup and reliability reasons (to cover a case where the first image gets corrupted for any reason).
Miscellaneous: The storage device can contain other parts, such as parameter sectors which can be stored in small sectors on the top or bottom of the flash device (depends on the flash support), or environment variables that configure the boot-loader (used by u-boot). Other systems might have some other miscellaneous blocks for their own special requirements.

A typical boot process

When powering up the board, the boot ROM code inside the chip does the most basic hardware setup and jumps to the first address of the storage device. This address contains the first command of the system’s boot-loader. In its turn, the boot-loader initializes the hardware which is required for the board setup, like memory controller, Ethernet port, Serial port or any other system specific peripheral. Once the hardware is initialized, the boot-loader loads the image which implements the system’s functionality. There are many variations for this stage, depending on the system type and usage. In some systems, the boot-loader uncompresses the compressed image to the RAM and jumps to the first address of the uncompressed image. Systems which run Linux usually skip this stage because the image contains a strap code which is attached to the start of the image, that does the uncompression work. Other systems load the image through the network (usually with TFTP) or through another standard or proprietary bus.

Once the boot-loader jumps to the image’s first address, it is no longer active, and the control is passed to the Operating system. The boot process of an OS is usually very complex and there are many books about the subject. In general, it initializes the OS sub-systems (such as memory management, file systems, etc) and initializes the rest of uninitialized hardware by calling the drivers’ init functions.

Once the OS has finished its initialization, it starts all the system’s programs or tasks which implement the product’s functionality.

Bitwise Operations

Hai Shalom — Mon, 23 Aug 2010 13:01:50 +0000

Bitwise operations are widely used in embedded systems, both in assembly code and in C code. Bitwise operations are used for reading and writing hardware registers, enabling or disabling hardware features, setting or masking interrupts, writing values to GPIO pins and many other uses. Bitwise operations are different than the corresponding logical operations. In this article I will cover the basic bitwise operations and provide some macros for the most common bitwise operations and some common practices when using them.

The OR Operation

The bitwise OR operation takes two arguments (usually two register values). Its output is composed of set of bits which are either 1′s in the first argument OR 1′s in the second argument. For example, given the first argument value (in binary) of 01110000 and second argument value of 10011101, the result would be 11111101. The bitwise OR operation can be treated as a bit level carry-less addition operation, according to the following truth table:

Arg 1	Arg 2	Result
0	0	0
0	1	1
1	0	1
1	1	1

The bitwise OR operator is “|” (a single pipe), and it has a different meaning than the logical OR operation, which is marked as a double pipe “||”. The Logical OR operation is usually used in conditional clauses, and the Bitwise OR operation is used for changing data (usually, setting 1 bits in a register). For example, suppose that a is an 8-bit register and we want to set the 4 high bits regardless of the actual data it contains, we can use the following command:

a = a | 0xf0;

or in short writing:

a |= 0xf0;

The Bitwise OR operation precedes the Logical OR operation.

The AND Operation

The bitwise AND operation takes two arguments (usually two register values). Its output is composed of set of bits which are 1′s in the first argument AND 1′s in the second argument. For example, given the first argument value (in binary) of 01110000 and second argument value of 10011101, the result would be 00010000. The bitwise AND operation can be treated as a bit level multiplication operation, according to the following truth table:

Arg 1	Arg 2	Result
0	0	0
0	1	0
1	0	0
1	1	1

The bitwise AND operator is “&” (a single ampersand), and it has a different meaning than the logical AND operation, which is marked as a double ampersand “&&”. The Logical AND operation is usually used in conditional clauses, and the Bitwise AND operation is used for changing data (usually, masking or clearing bits in a register). For example, suppose that a is an 8-bit register and we want to clear (or mask) the 4 high bits regardless of the actual data it contains, we can use the following command:

a = a & 0xf0;

or in short writing:

a &= 0xf0;

The Bitwise AND operation precedes the Logical AND operation.

The NOT Operation

The bitwise NOT operation takes one argument (usually a register value). Its output is composed of set of bits which have the opposite value of its input argument. For example, given the first argument value (in binary) of 01110000, the result would be 10001111. The bitwise NOT operation can be treated as a bit level reverse (complement) operation, according to the following truth table:

Arg	Result
0	1
1	0

The bitwise NOT operator is “~” (a Tilda), and it has a different meaning than the logical NOT operation, which is marked as an exclamation mark “!”. The Logical NOT operation is usually used in conditional clauses, and the Bitwise NOT operation is used for reversing a register’s data. Note that in C, a clause is False when its evaluation is 0, and True if its evaluation is not 0 (it could be any other value which is not 0, not necessarily a 1). Therefore, the True value is evaluated as “!0″. On the contrary, if an 8-bit register a has the value 0 (or 00000000 in binary), and we want to negate all its bits, the !a operation will not negate its bits, but change a to a value that evaluates as True. In order to negate a‘s bits, we should use the ~a command. The result would be 255 (or 11111111 in binary). As an example, we can use the following command:

a = ~a;

In order to initialize a register of any size to all 1′s, use the following syntax:

a = ~0;

The Bitwise complement operation precedes the Logical negation operation.

The Exclusive OR (XOR) Operation

The bitwise XOR operation takes two arguments (usually two register values). Its output is composed of set of bits which are 1′s only in bit indexes that the two arguments differ. In other words, an output bit will be set when the two input bits are different. For example, given the first argument value (in binary) of 01110000 and second argument value of 10011101, the result would be 11101101. The bitwise XOR operation can be treated as a bit level addition modulo 2 operation, according to the following truth table:

Arg 1	Arg 2	Result
0	0	0
0	1	1
1	0	1
1	1	0

The bitwise XOR operator is “^” (an up arrow) register) and it mainly used for computations (such as CRC or parity) and cryptographic functions. Sometimes, it is used as a means to clear a register (think about using XOR-ing a register with itself). As an example, we can use the following command to clear a register:

a = a ^ a;

The Shift Left/Shift Right Operations

The Shift operations move a register bits either to the left or to the right. The Right-Shift operator is “>>” (greater-greater than sign), and the Left-Shift operator is “<<” (smaller-smaller than sign). A bit which is moved beyond the register bit capacity (for example, the first bit before doing a right-shift) is lost forever, and a new inserted bit (for example, the first bit right after doing a shift-left) is always assigned with the value 0. The shift operations are mostly used to convert bit indexes to numbers or masks, and have another interesting property; Left shifting is equal to multiplication by 2 and Right shift is equal to division by 2 (without remainder). Repeating these action will raise the power of 2 of the same operation (for example, 1024 >> 3 == 1024 / 2^3 == 128). The compilers are aware of this property, and convert any division or multiplication by the power of 2 to an optimized shift operation.

Typical uses and macros for bit setting

The following section contains various examples for working with bit fields efficiently.

Bit Mask – Convert a bit index to a number

The following macro converts a bit index to a number. It is a simple macro that helps the programmer to write readable code for accessing specific bits in a register.

/* Get the bit mask from a given bit index */
#define BIT_MASK( bit ) \
    ( 1 << bit )

In many cases, the hardware data sheet specifies which bits to use. Instead of calculating the value and using hard coded numbers, it is a better approach to use this macro. This way the code is written more easily and more readable.

Read a bit field value

The following macro reads a bit field value from a specific offset in a register:

/* Read a bit field value from a register.
   input:
     reg: register
     offset: offset of the bits in the register
     bits: amount of required bits

   output: the required value
*/
#define READ_VALUE( reg, offset, bits ) \
    ( ( reg >> offset ) & ( ( 1 << bits ) - 1 ) )

The macro shifts the register value to the required field offset and masks the data to show only the required bits.

Write a bit field value

The following macro writes a bit field value to a specific offset in a register:

/* Write a bit field value to a register.
   input:
     reg: register
     offset: offset of the bits in the register
     bits: amount of required bits
     value: the value to be written

   output: none
*/
#define WRITE_VALUE( reg, offset, bits, value ) \
do { \
    reg &= ~( ( ( 1 << bits ) - 1 ) <<  offset ); \
    reg |= ( value & ( ( 1 << bits ) - 1 ) ) <<  offset; \
} while ( 0 );

The macro works in two steps. First step clears the required bit field with the amount of the required bits, and the second step writes the value to the register. First it masks the value to fit to the bit field size and then it shifts the value to the correct offset.

Examples

Let’s see an example of usage (assuming the macros are defined in rte-bitwise.h header file):

#include 
#include 
#include "rte-bitwise.h"

int main( int argc, char *argv[] )
{
  int a = 0x48F03456;
  int b = 0;

  printf( "a = 0x%x\n", a );

  /* Write 0x2 in bit field that starts at offset 12, with length 3 */
  WRITE_VALUE( a, 12, 3, 0x2 );

  printf( "a = 0x%x\n", a );

  /* Read the value of bit field 12..13 (start at 12, length 2) */
  printf( "field 12..13 = 0x%x\n", READ_VALUE( a, 12, 2 ) );

  /* Generate a bitmask with 4 bits */
  b = BIT_MASK( 31 ) | BIT_MASK( 12 ) | BIT_MASK( 13 ) | BIT_MASK( 1 );
  printf( "b = 0x%x\n", b ); 

  return 0;
}

The number in register a is a random number, which was made for the purpose of the example. Let’s compile and run this program, and see what are the results:

# ./bitwise
a = 0x48f03456
a = 0x48f02456
field 12..13 = 0x2
b = 0x80003002

The results show that the macros work as expected. We can see that bits 12..13 were changed from 3 to 2, and that they are read correctly. We can also see that the mask that was generated matches the bits we requested.

For conclusion, we now understand the difference between logical and bitwise operations. The provided macros are correct and efficient, and it is recommended to use them in your project when accessing hardware registers. If you have other ideas of useful bitwise macro, please write a comment and I’ll be happy to add them to the post.

Enabling Core Dumps in Embedded Systems

Hai Shalom — Tue, 10 Aug 2010 16:02:08 +0000

Core dumps are the standard Linux way for post-mortem analysis of crashed applications. Given some preconditions, the core dumps can provide a detailed back trace and shed some light about the last whereabouts of the crashed application.

The core dump itself is an executable file format, and it is generated by the Kernel. The core dump contains a list of memory sections which were accessible by the process, and their memory image. This allows you to analyze the core dump offline, on a host machine, using gdb which was compiled for the appropriate target.

Core dump is not enabled by default in embedded systems mainly due to memory limitations. In this article I will explain what is required in order to enable core dumps support on Embedded Linux/uClibc/Busybox platforms for a specific debug task.

As I mentioned, core dumps are not enabled by default in embedded systems, which usually have a very limited storage space and more RAM than non-volatile storage. The following steps describe what is needed in order to enable core dumps in your platform.

Enable Kernel support for Core Dumps

Core dump support in the Kernel can be disabled in compilation time in order to reduce the kernel size. Without this support, no core dump will ever be generated. In order to enable the Kernel’s support for Core dumps, enable the following option in the Kernel configuration menu:

General setup  --->
      Configure standard kernel features (for small systems)  --->
            Enable ELF core dumps

Enabling this option adds about 4KB to the final kernel size.

Enable debug mode in uClibc

The uClibc library must be configured differently in order to support the back trace. In case your application is multi-threaded, then you need to enable to following configuration options:

General Library Settings  --->
     Build pthreads debugging support

Also, configure the uClibc to skip the library stripping, in order to retain its symbols:

uClibc development/debugging options  --->
      Strip libraries and executables

Enable Core Dump support in Busybox

The Busybox provides the init process of the system, which is the father of all processes. The Busybox can be configured to enable core dumps to all its children by inheritance. Actually, it sets the init process as “dumpable”, and sets the core dump size limitation to “unlimited”. All the children of init will inherit these properties and allow core dumps. There are two actions required in order to enable the Busybox support for core dumps. First, enable this option in the Busybox configuration menu:

Init Utilities  --->
     Support dumping core for child processes (debugging only)

Second, create an empty file in the target’s filesystem’s root directory, called “.init_enable_core“.

Setup the Core dump size limitation

In systems without Busybox, it is required to setup the Core dump size limitation. By default, the size is 0, and it means that the Core dump is disabled. In order to get a readable (non-truncated) Core dump, setup the limitation to “unlimited” using the ulimit application:

# ulimit -c unlimited

Setup the Core file name and path

By default, the Core dump file is named core, and it is created in the current working directory of the process. There are cases where a process’s current working directory points to a read only filesystem. In this case, the Core dump file could not be created. Furthermore, the default file name does not indicate what was the process name or id, and by listing the file, you can not which process created it. The /proc/sys/kernel/core_pattern file can be set to define a template that is used to name core dump files. The template can contain % specifiers which are substituted by the following values when a core file is created:

%%  A single % character
%p  PID of dumped process
%u  real UID of dumped process
%g  real GID of dumped process
%s  number of signal causing dump
%t  time of dump (seconds since 0:00h, 1 Jan 1970)
%h  hostname
%e  executable filename

This file allows you to setup a different output directory as well. This option allows you to configure the Kernel to generate the files in a writable filesystem, such as a RAM disk, which could have enough storage space to hold the core dump file. Use “echo “your_pattern” > /proc/sys/kernel/core_pattern” to change the Core dump file name pattern.

The /proc/sys/kernel/core_uses_pid file determines if the Core dump file name will end with .PID.

Setup the Compilation flags of your debugged Application(s)

In order to make an application debugable, it must be compiled a bit differently, using some compilation flags that create debug symbols and enable support for stack frame.

The applications needs to be compiled with the “-g -ggdb” flags in gcc, in order to enable debug symbols.
The optimization level: Optimization levels higher than -O1 trigger frame pointer omitting, which makes back tracing impossible. The reason is because the frame pointer can be reused for other purposes, and the fact that saving the frame pointer requires additional opcodes. In order to keep the frame pointer in higher optimization levels, use the “-fno-omit-frame-pointer” flag in gcc. If possible, reduce the optimization level to -O1 for the purpose of the debug. Higher optimizations may alter the final binary and omit or inline some functions. The -O0 optimization level might break your application, so I don’t recommend using it.
If the target platform is ARM and the binaries are in Thumb mode, add the following flags in gcc to keep the stack frame in the standard form: “-mtpcs-frame -mtpcs-leaf-frame“.

Enable Core Dumps manually in your Application(s)

It is possible to set your application as “Dumpable” and its limit as “Unlimited” using standard system calls.

The “Dumpable” state can be set or retrieved by the prctl system call, using the PR_SET_DUMPABLE and PR_GET_DUMPABLE flags accordingly. When using the set command, the second argument determines if the process is “Dumpable” (1-yes, 0-no). Here’s an example code snippet:

{
    #include 

    int ret;

    /* Get the Dumpable state */
    ret = prctl( PR_GET_DUMPABLE, 0, 0, 0, 0 );
    printf( "PR_GET_DUMPABLE returned %d", ret );

    /* Set the Dumpable state to be enabled */
    ret = prctl( PR_SET_DUMPABLE, 1, 0, 0, 0 );
    printf( "PR_SET_DUMPABLE returned %d", ret );

    /* Get the Dumpable state again, make sure it was set */
    ret = prctl( PR_GET_DUMPABLE, 0, 0, 0, 0 );
    printf( "PR_GET_DUMPABLE returned %d", ret );
}

The limitation of a core dump can be set manually by the application using the setrlimit function call, using the RLIMIT_CORE flag. It receives a structure with two fields. The first sets the current limitation, and the second sets the maximum allowed limitation. The fields accept any number, and RLIM_INFINITY to specify “Unlimited”. Here’s an example code snippet:

{
    #include 
    #include 

    int ret;
    struct rlimit rlim;

    /* Get the core dump limitation */
    ret = getrlimit(RLIMIT_CORE, &rlim);
    printf( "RLIMIT_CORE returned %d (%d, %d)", ret, rlim.rlim_cur, rlim.rlim_max );

    /* Set the core dump limitation to be unlimited */
    rlim.rlim_cur = RLIM_INFINITY;
    rlim.rlim_max = RLIM_INFINITY;
    ret = setrlimit(RLIMIT_CORE, &rlim);
    printf( "RLIMIT_CORE returned %d", ret );

    /* Get the core dump limitation again */
    ret = getrlimit(RLIMIT_CORE, &rlim);
    printf( "RLIMIT_CORE returned %d (%d, %d)", ret, rlim.rlim_cur, rlim.rlim_max );
}

Custom Signal Handlers

Unfortunately, if your application is equipped with a customized signal handler, no core dump will be generated, because it is generated only by the default signal handlers. In case your application has a custom signal handler, disable it before starting to debug, otherwise, no core dump will be generated. Some sources in the Internet mention that restoring the default signal handler inside the signal handler after the exception has occurred, and sending it again in a loopback can trigger a core dump. In the tests I did, it did generate a core dump, but the only thing I saw in the core dump was the code that my handler executed (i.e. the calls to signal and kill), so this did not help me. Perhaps on other platforms this trick works better.

Conclusions

There are many actions you need to do in order to enable core dumps in Embedded Linux system. Most of them can be automated in your Makefiles and configuration system when a Debug-Mode option is enabled. When a process crashes and a Core dump was generated successfully, the following message will appear:

Segmentation fault (core dumped)

In the next step, download the generated core dump to the host machine and run gdb in order to analyze it. Use the core command to load it, and file command to load the symbols of your application. Use the bt full command to get a complete back trace.

Resources

http://linux.die.net/man/2/prctl
http://linux.die.net/man/7/signal
http://linux.die.net/man/2/setrlimit
http://linux.die.net/man/5/core
http://www.gentoo.org/proj/en/qa/backtraces.xml
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.faqs/ka12961.html

Coretrace debugger tool: http://www.arbetsmyra.dyndns.org/coretrace/

Open-Source Licenses

Hai Shalom — Tue, 27 Jul 2010 04:15:07 +0000

The Open Source concept is not new, and has been studied and researched by professionals and by the academia for quite a while. Open Source software products are the outcome of collaboration and cooperation of individuals programmers from all over the world, who gather in communities. The communities are non-profit organizations that distribute and support their software without any charge. The term Open Source, also referred as Free Software, constitutes the main concept of the Open Source movement, which believes that the user must have the rights to use, modify or redistribute the software as he wishes, while granting the same rights to the users of his derivative work or redistribution. The term “Free” is referred as “Free Speech” and not “Free Beer”.

Nowadays, Open Source software is widely used by for-profit organizations. In this article, I will provide an overview of the most used Open Source Licenses, the rights that they grant, and the obligations they require.

There are two main Open Source organizations that support the Free/Open Source concept:

The Free Software Foundation – Was founded in 1985 and promotes the idea of the freedom for the users, to use, modify, copy and redistribute the software without restrictions.
The Open Source Initiative – Was founded in 1998, and agrees with the general concept of FSF, but believes that the cause of free software is about collaboration and creation of better software for everyone.

Open Source Licenses

GNU General Public License: The most used and common open source license available. The GNU GPL is the most restrictive license available today, and it has introduced the “Copyleft” concept, where rights that were usually reserved by commercial firms are now granted to the users. The GNU GPL license grants the users the right to; retrieve the source code, modify it, copy it, build upon it a new product and redistribute it, freely. On the other hand, the GNU GPL does not allow the user to change the license terms, and requires that any derivative work or redistribution must retain the original license terms. Moreover, the GNU GPL does not allow mixing proprietary code (private software that is not free) with a GPL licensed software, meaning that any private changes to a GPL software, or usage by a other proprietary code is prohibited. In this case, the GPL license has a “Viral” property that “infects” any other piece of software which uses a GPL licensed software, with the GPL license. In other words, any software that links with, or adds functionality to a GPL licensed software must be licensed with GPL. The GPL also discourages software patents in this way. The GPL allows the distribution of proprietary software and GPL software in the same media. The most common open source project that is licensed with GPL is the Linux operating system.

GNU Lesser General Public License: This is a weaker version of GPL, that is usually used by libraries. The license is similar to the GPL, with the only exception that it allows to link against it without forcing the linked code to be released under the GPL or compatible license. It also requires that a downstream user be able to link it against a newer version of the library (say, one with bug fixes). This means that you either have to dynamically link it, or provide object files for your proprietary code if you statically link it. In other words, proprietary code can use and call functions that are provided by an LGPL licensed software without the “viral” effect. A common open source project that is licensed with LGPL is the uClibc library.

Barkley Software Distribution License: Another widely used type of license is the BSD license, which was first used by the Barkley University. This license, with some minor modifications from the original license, is the most permissive license available. Similarly to GPL, it allows the user to use, modify and redistribute the software. But unlike the GPL, any derivative work could remain private, and it is possible to mix proprietary code with the BSD licensed code. The only requirement this license has is to give the credit to the original authors of the software. The main concept of this license was born from the fact the in the 90′s, many software projects were funded by the US government, so you can actually consider the software as if it is owned by the public. Some open source software that use BSD-style license add some more requirements from the user, such as the prohibition to use the software name as a means to promote a derivative software product.

Massachusetts Institute of Technology License: Another permissive license, similar license to the BSD license.

Artistic License: The Artistic license was originally developed and used by the perl project, and is commercially oriented. The license allows to keep modifications as private, but prohibits to sell the software, unless it is bundled with a bigger software package.

Netscape PL/Mozilla PL: The NPL/MPL licenses were developed in 1998 by the Netscape corporation, and present a trade-off approach between the restrictive GPL and the permissive BSD. Similarly to GPL, they grant the users the freedom to use , modify and redistribute the software while retaining the original license, and similarly to BSD, they permit mixing and usage of proprietary software. The NPL has another special property that gave Netscape extra privileges over derivative work. It allowed Netscape to reuse an open source derivative work, modify it privately and sell it. Of course, this license was not very popular due to the fact the communities felt that Netscape was going to harvest their efforts for its business. The MPL license, which does not contain this property is more popular nowadays.

CreativeCommons License: A rather new type of license that was developed in order to protect any type of work (text, pictures, music, code) while granting some rights to the users and reserving some rights to the owner. The license has a few flavors where the owner of the work can decide which right to grant, and which right to reserve, such as allowing or disallowing redistribution, modification and commercial use.

Misconceptions about other Licenses

There are other types of licenses that are used by software authors, which are not considered to be Open Source licenses, due to the fact that they lack one or more properties of the Open Source requirements:

Public Domain: Public domain license is not an open source license. Indeed, every user can obtain, modify and redistribute the code, but since the owner has waived his rights on the software, any other user can keep the changes private and even sell the software. Therefore, public domain software is not considered as open source.

Freeware: Freewares are usually provided in the binary form only, without access to the source code. Therefore, freeware software is not considered as open source.

Shareware: Shareware are usually provided in the binary form only. Furthermore, users of Shareware are required to pay some amount of money to the author, otherwise, its function is limited. Therefore, freeware software is not considered as open source.

Warranty and Liability

When a commercial firm or a private user wishes to use an open source software, they must be aware that the software comes AS-IS, with no warranty or liability, unlike proprietary, of-the-shelf software, which is provided with some kind of warranty. The authors and contributors are not liable for any of kind of damage that may rise from the usage of the software.

Resources:

The Free Software Organization: http://www.fsf.org/
The Open Source Initiative: http://www.opensource.org/
GNU Licenses: http://www.gnu.org/licenses/

Getting a process call-stack using pstack

Hai Shalom — Thu, 22 Jul 2010 15:03:37 +0000

Ever wanted to see (in run-time) which function a program is currently executing, or what is the current calling stack of a program? I’m sure you do. In many cases, it is very helpful to see a back trace of all the calling functions in a program, at any given time. Normally, this is a basic feature when you are using a debugger (either a JTAG based hardware debugger or a software based debugger like gdb). However, in this post I would like to show you that this is also possible, with some preconditions, to be done without any attached debugger. It could be useful to remotely debug units in the field or in the lab, where any extra debug equipment may be unavailable.

In this post, I would like to present the pstack utility, which displays the calling stack of a running program. First, let’s see what it does, and later we’ll review how it works in high level. In the following program, there is a nested call from main( ) to a( ) to b( ) to c( ), which remains there forever. Let’s assume that the following program is a very complicated program which appears to be running an infinite loop (the latter is indeed correct). Normally, without a debugger, we could not have any means to find out which function is causing this loop, or in other words, we can’t know what the program is doing (or doing wrong).

/* a very simple program */
int c( int cc )
{
    while(cc);
    return 0;
}

int b( int bb )
{
    return c(bb);
}

int a( int aa )
{
    return b(aa);
}

int main( int argc, char *argv[] )
{
    return a(1);
}

Now we compile and run it on the target. If we start this program in the foreground, it will hang our console because it never ends. So let’s move it to the background, and activate the pstack utility in order to understand what is going on. The pstack utility accepts the process id of the required process as an input parameter. Therefore, we’ll need to run ps first in order to get it. Here’s the output of pstack on our program:

# ./pstack 281922

8192: ./test

(No symbols found in /lib/libc.so.0)

(No symbols found in /lib/ld-uClibc.so.0)

0x000083ec: c() [at 0x000083d4]

0x00008420: b() [at 0x00008404]

0x0000844c: a() [at 0x00008430]

0x0000847c: main() [at 0x0000845c]

The pstack output displays the following information (from bottom to top):

Return address to the main( ) function and its start address.
Return address to a( ) function and its start address.
Return address to b( ) function and its start address.
Current execution address in c( ) function and its start address.
A warning that the C library was stripped (this isn’t critical, unless you want to see the function calls inside them as well).

We can see that our program is running the c( ) function, and if we’ll run pstack again, we’ll probably see a similar output. In the next step we can use either objdump or addr2line applications to find the exact source code. Here’s an example:

# addr2line -e ~/test 0x83d4

/home/hai/test.c:3

As addr2line suggests, we are stuck at line 3, which is the endless while loop.

Limitations of pstack utility

The original pstack utility supports only the x86 32-bit platform. You can find pstack utility in every Linux distribution for PCs, but not in any embedded platform. However, there are patches in the web that enable support for other platforms (a link for a patch for ARM platform will be given later).

Prerequisites of pstack utility

The pstack utility requires the following items enabled on your program in order to fully function and provide all the information:

The program must be compiled with gcc (GNU compiler collection).
The program must not be stripped. Otherwise, no symbols will be available and no functions name will be displayed. Instead you’ll get only question marks.
The program must retain the frame pointer. Otherwise, the pstack will not be able to crawl in the stack. When enabling the compiler optimizations (-O[s123]), the compiler throws away the frame pointer in order to reduce code size and reuse the register for other purposes. In order to retain it while enabling the optimizations, use the -fno-omit-frame-pointer flag.
When debugging a multi-threaded application, the pthread library must be compiled with debug support (Enabled the PTHREADS_DEBUG_SUPPORT define in uClibc).
For ARM platforms, when the program is compiled in Thumb mode, the compiler must be configured to generate a stack frame that is compliant with the Thumb Procedure Call Standard for all the functions. This option is required by pstack utility in order to crawl the stack. The following flags need to be enabled: -mtpcs-frame -mtpcs-leaf-frame. Note that when you enable these flags, the compiler will add about 20 additional opcodes in order to create the standard stack structure.

How does it work?

In order to understand how it works, we first need to understand the structure of the stack frame of each function. However, each platform has its own stack frame structure and therefore, I will only describe the ARM stack frame structure (there is plenty of information in the web). When working with frame pointer, the compiler generates code that stores it, among other registers in the frame in the following structure:

When entering a function, the code pushes the following registers to the stack: frame pointer (fp), stack pointer image (ip), link register (lr) and the program counter (pc). In the end of the function, the code restores the registers, where the frame pointer gets the last frame pointer, the stack pointer gets the saved stack pointer image, and the program counter gets the link register value, which is actually the return address. The ptrace utility, connects to the required process and first reads the frame pointer and program counter registers. The first provides an address to the stack frame and the latter provides the current executing address. It then shows what is the current address that the program executes. Using the frame pointer address, it reads the next frame in a loop until the next frame pointer equals to 0. In each frame in the loop, it extracts the link register in order to resolve of the return address. Per each return address, it browsed the symbol table in order to match it with a corresponding function, and then prints the function name.

Resources

The original pstack source code can be downloaded from here.
A patch for Little and Big endians ARM/Thumb code can be downloaded from here. Download and extract the original source code, download and save the patch file in the same directory, and run “patch -i pstack-1.2-arm.patch” to enable the ARM support.
GCC ARM options: http://gcc.gnu.org/onlinedocs/gcc/ARM-Options.html
pstack at debian: http://packages.debian.org/source/stable/pstack
pstack at SourceForge: http://sourceforge.net/projects/bsd-pstack/

Writing Efficient C Code for Embedded Systems

Hai Shalom — Thu, 15 Jul 2010 19:57:51 +0000

The traditional definition of efficiency has two aspects: speed and size. In most cases, optimizing the first aspect causes a degradation in the other, and it’s a matter of balancing between the two, according to the specific needs. Per each embedded system, or even a software module, the appropriate strategy must be balanced between the two. Nowadays, there is a new dimension to this definition; power. In this article, I am going to discuss about the traditional aspects. In the years I’ve been working as a software engineer, I gained a lot of experience about C code efficiency. I saw how changing a few lines of code makes the difference, either in performance, final size or memory consumption. The examples I’ll show here were written in C and tested on an ARM platform. You should expect similar behavior by other processors. I might update this article from time to time with some more tips, so it is recommended to bookmark it and catch up.

Global variables

The use of global variables is not recommended in most cases, but there are cases where we must use them. In some cases, we declare large tables/arrays which are used later by the software. When using global variables, follow the following guidelines:

Tables/Arrays which never change, should be defined as const. When defining them as constant, they will be moved to the code section. If this table/array was defined in a shared library, without the const directive, the table would be copied per each instance of a linked process. However, with the const directive, it will have only a single instance in memory.
Reading and writing global variables require additional opcodes (load the global address, load the value from the address, and then store the value back). If possible, try to use local variables, or use a local copy.
Declare all globals as static, unless other C files need to see them (not recommended at all. You can use a local read and write functions instead, and keep them static).

Variable types, signess and localization

In is very important to select the appropriate variable types you are using. There are a few rules of thumb in this case:

Try to use variables in the native size of the processor (like int for 32-bit processors). For a typical 32-bit processor, the use of short int or char types is less recommended. There are processors which need to read a 32-bit word anyway and do some shifting and masking, just to get the right value. Other pipeline processors which have 16-bit or 8-bit access opcodes, may read 32-bit anyway.
If you don’t require negative values, use unsigned variables. Unsigned variables yield better performance, especially when using division and multiplications.
Declare variables only when you really need them and not in the begining of the function. This will allow the compiler to better utilize the internal registers and avoiding assignment of a register to a variable which is used only later.

Division and Modulus

The division and modulus arithmetic operations require extensive CPU cycles. In ARM platform, they are done by software, and it is also known that in other processors, these operations are much slower than any other arithmetic operations.

The modulus operation can be implemented much more efficiently, in some cases, using a simply counter:

Bad example:

int tick_after_100_cycles_mod( void )
{
    static unsigned int i = 0;

    if( i % 100 == 0) {
        return 1;
    } else {
        i++;
    }

    return 0;
}

Good example:

int tick_after_100_cycles_cnt(
    void )
{
    static unsigned int i = 0;

    if( i >= 100) {
        return 1;
    } else {
        i++;
    }

    return 0;
}

The division operation is extremely efficient if the divider is a natural number in power of two (2,4,8,16,32…). The reason for this is the binary representation of the numbers inside the processor, where each division by two equals to a single right shift. In this case, if you are dividing a variable with a hard coded value in the power of two, the compiler will automatically convert it to a shift operation. In case you are dividing by a variable, the compiler can not know what values could be used in it and will not optimize this division, so you’ll have to help it by doing the shift operation manually. Here are a few examples and the corresponding output for ARM:

/* Results in a call to division opcode or function */
int test_div( unsigned num, unsigned div )
{
    return num / div;
}

/* Results in an optimized automatic shift right */
int test_div_hardcoded_power_2( unsigned num )
{
    return num / 8;
}

/* Results in an optimized automatic shift */
int test_div_shift_right( unsigned num, unsigned powerof_two )
{
    return num >> powerof_two;
}

00000000 :
   0:   e52de004        push    {lr}            ; (str lr, [sp, #-4]!)
   4:   e24dd004        sub     sp, sp, #4      ; 0x4
   8:   ebfffffe        bl      0 <__aeabi_uidiv>
   c:   e28dd004        add     sp, sp, #4      ; 0x4
  10:   e8bd8000        pop     {pc}

00000014 :
  14:   e1a001a0        lsr     r0, r0, #3
  18:   e12fff1e        bx      lr

0000001c :
  1c:   e1a00130        lsr     r0, r0, r1
  20:   e12fff1e        bx      lr

We can see that the functions that use shift operation are very efficient, while the function with the real division calls the external __aeabi_uidiv( ) function to calculate the result.

For loops

For loops are widely used in the code. The natural human thinking makes us write our for loops from 0 to the maximum value. However, in case the order of the index is not important, it is more efficient to do the for loop from top to 0. The reason is due to an extra comparison opcode required when not comparing with 0. Per each incrementing for iteration, the CPU needs to check that we have reached the maximum value, and then break. However, if the for loop is decrementing, the comparison is not required because the result of the decrement action will trigger the Z (zero) flag, and the CPU will know when to break just by looking at it.

extern void foo(
    int );

void test_incrementing_for_loop( void )
{
    int i;

    for(i=0; i<100;i++) {
        foo(i);
    }
}

void test_decrementing_for_loop( void )
{
    int i;

    for(i=100; i; i--) {
        foo(i);
    }
}

00000000 :
   0:   e92d4010        push    {r4, lr}
   4:   e3a00064        mov     r0, #100        ; 0x64
   8:   ebfffffe        bl      0 
   c:   e3a04063        mov     r4, #99 ; 0x63
  10:   e1a00004        mov     r0, r4
  14:   ebfffffe        bl      0 
  18:   e2544001        subs    r4, r4, #1      ; 0x1
  1c:   1afffffb        bne     10 
  20:   e8bd8010        pop     {r4, pc}

00000024 :
  24:   e92d4010        push    {r4, lr}
  28:   e3a00000        mov     r0, #0  ; 0x0
  2c:   ebfffffe        bl      0 
  30:   e3a04001        mov     r4, #1  ; 0x1
  34:   e1a00004        mov     r0, r4
  38:   e2844001        add     r4, r4, #1      ; 0x1
  3c:   ebfffffe        bl      0 
  40:   e3540064        cmp     r4, #100        ; 0x64
  44:   1afffffa        bne     34 
  48:   e8bd8010        pop     {r4, pc}

We can see that the decrementing for loop has one less opcode.

In some cases, when short loops are required, it could be more efficient (in terms of speed) to unroll the loop and save all the loop overhead. The compiler can be configured to do that automatically when using the -O3 optimization flag.

extern void foo( int );

void for_loop( void )
{
    int i;

    for(i=4; i; i--) {
        foo(i);
    }
}

void unrolled_loop( void )
{
    foo(4);
    foo(3);
    foo(2);
    foo(1);
    foo(0);
}

00000000 :
   0:   e52de004        push    {lr}            ; (str lr, [sp, #-4]!)
   4:   e3a00004        mov     r0, #4  ; 0x4
   8:   e24dd004        sub     sp, sp, #4      ; 0x4
   c:   ebfffffe        bl      0 
  10:   e3a00003        mov     r0, #3  ; 0x3
  14:   ebfffffe        bl      0 
  18:   e3a00002        mov     r0, #2  ; 0x2
  1c:   ebfffffe        bl      0 
  20:   e3a00001        mov     r0, #1  ; 0x1
  24:   ebfffffe        bl      0 
  28:   e3a00000        mov     r0, #0  ; 0x0
  2c:   e28dd004        add     sp, sp, #4      ; 0x4
  30:   e49de004        pop     {lr}            ; (ldr lr, [sp], #4)
  34:   eafffffe        b       0 

00000038 :
  38:   e92d4010        push    {r4, lr}
  3c:   e3a00004        mov     r0, #4  ; 0x4
  40:   ebfffffe        bl      0 
  44:   e3a04003        mov     r4, #3  ; 0x3
  48:   e1a00004        mov     r0, r4
  4c:   ebfffffe        bl      0 
  50:   e2544001        subs    r4, r4, #1      ; 0x1
  54:   1afffffb        bne     48 
  58:   e8bd8010        pop     {r4, pc}

In the result we can see that the unrolled loop is longer. However, if we take a closer look, we can see that in the unrolled loop runs faster, because each iteration is done with 2 opcodes, compared to 4. In the unrolled for loop, we have 2 opcodes; loading a value and calling foo( ). In the regular for loop, we have 4 opcodes; loading a value, calling foo( ), reducing the register by 1 and comparing the result to 0 (with an incrementing for loop there could be even another opcode).

If-Else and Switch

In many cases, we use branching in our code. We all know the cases where we need to add some more conditions when the code grows or new features are added. We might end up with a multiple if-else-if-else clauses. Long if-else clauses are not efficient because for the worst case scenario, where the last comparison is positive, the CPU must check all other possibilities. In this case it is more efficient to use a switch clause which is implemented as a lookup table. The compiler generates a list of addresses and the CPU jumps directly to the correct one using the index in the switch.

If it is not possible to use switch, it may be possible to apply the binary search strategy. If the range is large, we can divide it to sub areas, and increase the performance by reducing the amount of tests in the worst case scenario.

/* Bad in worst case scenario */
if(i==1) {
    do_something1();
} else if(i==2) {
    do_something2();
} else if(i==3) {
    do_something3();
} else if(i==4) {
    do_something4();
} else if(i==5) {
    do_something5();
} else if(i==6) {
    do_something6();
} else if(i==7) {
    do_something7();
} else if(i==8) {
    do_something8();
}

/* Improved version */
if(i<=4) {
    if(i==1) {
        do_something1();
    } else if(i==2) {
        do_something2();
    } else if(i==3) {
        do_something3();
    } else if(i==4) {
        do_something4();
} else {
    if(i==5) {
        do_something5();
    } else if(i==6) {
        do_something6();
    } else if(i==7) {
        do_something7();
    } else if(i==8) {
        do_something8();
    }
}

We can see that for case i == 5, it takes 5 comparisons in version 1 and 2 comparisons in version 2, and in case i == 8, it takes 8 comparisons in version 1 and 5 comparisons in version 2. The binary break can be even finer if required.

Lazy Evaluation

Another principle in the if-else calculation of C is referred as “Lazy Evaluation“. In multiple conditional clauses, where there is a need to check various conditions, the generated code will evaluate only the minimum conditions which are required to satisfy the clause, from left to right. For example, in a multiple OR clause, it is sufficient that only one of the conditions becomes true to satisfy the clause, and in a multiple AND clause, it is sufficient that only on of the conditions become false to negate the clause. We can utilize this property to put the “easy” or “trivial” checks first, and thus avoiding some work.

int check_something_1( int i )
{
    if( i > 99 && i % 100 == 0 ) {
        return 1;
    }

    return 0;
}

int check_something_2( int i )
{
    if( i == 0 || i % 100 == 0 ) {
        return 1;
    }

    return 0;
}

In the first function, we first check if i is bigger than 99, and only then perform the modulus calculation. So actually, the modulus will be skipped every time we call this function with 0-99. In the second example, we will skip the modulus when i equals 0. In both cases, it would be wrong to switch between the conditions because the “easy” condition is first.

Return Value checking

Another thing that could slightly optimize the code is to check return values of functions by comparing to 0. Since comparison to zero is native to the CPU, it will usually save one or two opcodes. If a function returns 0 on success and -1 for failure, don’t check if the return value is -1. Instead, check if the return value is less than 0. Another example; If a function returns OK or NOK, compare the return value against the enumeration which is equal to 0.

/* returns 0 on success, -1 on failure */
extern int foo( int i );

/* Good example 1: Good path */
if( foo( 7 ) == 0 ) {
    /* Good path: do something */
}

/* Good example 2: Bad path */
if( foo( 7 ) != 0 ) {
    /* Bad path: do something */
}

/* Good example 3: Bad path */
if( foo( 7 ) < 0 ) {
    /* Bad path: do something */
}

/* Bad example */
if( foo( 7 ) == -1 ) {
    /* Bad path */
}

Lookup tables

Lookup tables can speed up the processing but increase the size by actually having all the answers in advance without the need to calculate them. Here’s an example that beats both size and speed:

char get_char( unsigned int i )
{
    switch(i) {
    case 0:
        return 'r';
    case 1:
        return 't';
    case 2:
        return '-';
    case 3:
        return 'e';
    case 4:
        return 'm';
    case 5:
        return 'b';
    default:
        return '\0';
    }
}

static const char lookup_table[] = { 'r', 't', '-', 'e','m','b' };

char get_char_lookup( unsigned int i )
{
    if(i>=sizeof(lookup_table)) {
        return '\0';
    }
    return lookup_table[i];
}

00000000 :
   0:   e3500005        cmp     r0, #5  ; 0x5
   4:   979ff100        ldrls   pc, [pc, r0, lsl #2]
   8:   ea000005        b       24 
   c:   0000002c        .word   0x0000002c
  10:   00000034        .word   0x00000034
  14:   0000003c        .word   0x0000003c
  18:   00000044        .word   0x00000044
  1c:   0000004c        .word   0x0000004c
  20:   00000054        .word   0x00000054
  24:   e3a00000        mov     r0, #0  ; 0x0
  28:   e12fff1e        bx      lr
  2c:   e3a00072        mov     r0, #114        ; 0x72
  30:   e12fff1e        bx      lr
  34:   e3a00074        mov     r0, #116        ; 0x74
  38:   e12fff1e        bx      lr
  3c:   e3a0002d        mov     r0, #45 ; 0x2d
  40:   e12fff1e        bx      lr
  44:   e3a00065        mov     r0, #101        ; 0x65
  48:   e12fff1e        bx      lr
  4c:   e3a0006d        mov     r0, #109        ; 0x6d
  50:   e12fff1e        bx      lr
  54:   e3a00062        mov     r0, #98 ; 0x62
  58:   e12fff1e        bx      lr

0000005c :
  5c:   e3500005        cmp     r0, #5  ; 0x5
  60:   959f3008        ldrls   r3, [pc, #8]    ; 70 
  64:   83a00000        movhi   r0, #0  ; 0x0
  68:   97d30000        ldrbls  r0, [r3, r0]
  6c:   e12fff1e        bx      lr
  70:   00000000        .word   0x00000000

We can see that the lookup table implementation is fast and small.

Data caching and avoiding calling other functions/system calls

My meaning here is not for the actual CPU cache. Sometimes, we call some functions or system calls in order to perform some calculation or get some data. If possible, try to reduce and eliminate the use of these, especially system calls which cause a context switch and data copying (relatively slow actions).

In this example, we can skip calling getpid( ) starting from the second time we call this function:

#include 
#include 

pid_t my_getpid( void )
{
    pid_t pid = getpid();
    return pid;
}

pid_t my_getpid_optimized( void )
{
    static pid_t pid = -1;

    if(pid<0) {
         pid = getpid();
    }
    return pid;
}

00000000 :
   0:   e92d4010        push    {r4, lr}
   4:   e59f4020        ldr     r4, [pc, #32]   ; 2c 
   8:   e5943000        ldr     r3, [r4]
   c:   e3530000        cmp     r3, #0  ; 0x0
  10:   ba000001        blt     1c 
  14:   e5940000        ldr     r0, [r4]
  18:   e8bd8010        pop     {r4, pc}
  1c:   ebfffffe        bl      0 
  20:   e5840000        str     r0, [r4]
  24:   e5940000        ldr     r0, [r4]
  28:   e8bd8010        pop     {r4, pc}
  2c:   00000000        .word   0x00000000

00000030 :
  30:   eafffffe        b       0

While the my_getpid( ) function looks a lot shorter (both in the code and in the assembly), it is actually much slower than the other function. The reason is because the my_getpid( ) function will initiate a system call each time it is called, and on the other hand, the my_getpid_optimized( ) function, will call the system call only once. The rest of the times will return the stored process id. In this example, we can see the tradeoff between speed and size.

Efficient search and sort functions

The uClibc library (and probably other standard C libraries) provide very efficient search and sort functions, so if you need to perform searches or sorts on any type of variable, you don’t need to write an algorithm from scratch, but use the existing ones. The binary search function is called bsearch( ), and the quick sort function is called qsort( ). When using these functions, you need to specify the size of your element, and a comparison function which will be used by the algorithms to perform their actions. Here’s an example for quick sorting an array of elements. The elements are made of an index and a name. We define the sort criteria as the index number, and write the comparison function accordingly.

#include 

/* Entry element */
struct entry_t {
    int index;
    char name[ 10 ];
};

static int mycmp( const void *a, const void *b )
{
    /* Compare the elements by indexes */
    if(((struct entry_t *)a)->index == ((struct entry_t *)b)->index) {
        return 0;
    }
    return ((struct entry_t *)a)->index > ((struct entry_t *)b)->index;
}

/* Maximum elements */
#define MAX_ENTRIES 100

/* The array to be sorted */
struct entry_t entries_array[ MAX_ENTRIES ];

int do_sort( void )
{
    /* Sort the array */
    qsort( &entries_array, MAX_ENTRIES, sizeof( struct entry_t ), mycmp );
    return 0;
}

Inline functions

Inline functions are functions which are copied inside the calling function thus avoiding the function call overhead. It could be used to increase the performance of a specific critical area. Note that inlines create code duplication per each call to them, so use them wisely.

Bitmaps

A bitmap is usually a register (32-bit) variable that can represent data. The 32-bits can be divided to represent up to 32 pieces of data, wit 1 bit per piece which represent 2 states (on or off, enabled or disabled). The less data pieces represented, the more data can be saves (for 16 pieces, there are 2 bits which represent 4 different states). Sometimes bitmaps can be very efficient comparing to other data types, mainly because the single memory access, and the fact that data can be manipulated using bit-wise operations (&, |, ~, ^). For example, suppose we need to keep a list of objects (like interfaces, devices, etc.), and we want to keep track which of them has a specific property enabled (like, link, connectivity, etc.). Usually, we will use an array or any other data type to store this information. If the list is 32 entries long or less, we can store it in a register, and mark the property using a single bit. Each object will have its own offset in the register. Enabling a property is a single OR operation in the object’s offset, and disabling a property is a single AND operation in the object’s offset.

Compiler optimizations

Don’t forget to enable the appropriate compiler optimizations. You can read about it here.

Resources

Quick sort: http://linux.die.net/man/3/qsort
Binary search: http://linux.die.net/man/3/bsearch
Using gcc: http://www.rt-embedded.com/blog/archives/using-gcc-part-2/

Working with Kconfig

Hai Shalom — Tue, 06 Jul 2010 04:18:40 +0000

The Kconfig mechanism is today’s standard configuration mechanism and it is used by leading open source projects, such as the Linux kernel, Busybox and uClibc. The Kconfig has a basic configuration syntax that allows you to add configuration options of various types, create dependencies and write a few lines of description. The Kconfig utilities know how to read and parse the configuration files, and create project configuration files (usually by the name of .config for Makefiles and autoconf.h for source files). Other advantages this method has, are automatic menu generation (for both graphical and text based consoles), and ease of configuration management. With Kconfig, there is no need to specify any build flags to the project’s make. In this article I will show how to use the Kconfig to configure your Linux kernel, and how to easily modify the kernel’s Kconfig files in order to add a new driver somewhere in the tree. These methods of configuration can be also ported to any other projects.

The Linux kernel is instrumented with a Kconfig file almost per each directory. Each Kconfig file configures its own level. For example, you’ll find a Kconfig in each driver directory or arch directory. Each one of them configures a driver or architecture and includes other Kconfig files in the levels below it.

Configuring the Linux Kernel

There are 4 main ways to configure the kernel:

Running “make menuconfig”: Will show a text based menu configuration of the kernel. It also applicable for configuring a Linux kernel using telnet/ssh connections to a build server. This configuration option will read the existing configuration in case it exists. Otherwise, it will initialize all the menu items with their default values. When browsing in the menus, you could see that some configuration options are not marked, some are mark with star [*] and some are marked with [M]. These options mean: Disabled, Enabled and Modulized, accordingly. The difference between Enabled and Modulized, is the fact that when a driver or a feature is Enabled, it will be loaded automatically with the kernel, as an integral part of it without an option to remove. The Modulized option enables hot plug and unplug of the driver/feature, thus allowing you to use the memory for other tasks, in case it is not required.

: Screenshot: menuconfig (textual configuration menu)

Running “make xconfig”: Similar to option 1, but uses a graphical interace. It will not work with telnet/ssh connections, and it requires the “qt3-devel” package to be installed in the machine.

Screenshot: xconfig (graphical configuration menu)

Running “make defconfig”: Will read and parse all the configuration files and assign default values to the configuration variables. In case a default value is not available, the program will prompt for a selection.
Running “make oldconfig”: In case you were editing the configuration file (.config) manually, it is required to run this command in order for the setting to take effect.

The Kconfig syntax

The Kconfig syntax is very simple and it is documented in the Documentation/kbuild/kconfig-language.txt file. Let’s review some of the basics by a few examples:

An example from drivers/video/Kconfig:

config FB_CIRRUS

tristate "Cirrus Logic support"

depends on FB && (ZORRO || PCI)

select FB_CFB_FILLRECT

select FB_CFB_COPYAREA

select FB_CFB_IMAGEBLIT

---help---

This enables support for Cirrus Logic GD542x/543x based boards on

Amiga: SD64, Piccolo, Picasso II/II+, Picasso IV, or EGS Spectrum.

The example, defines a new variable in the name of CONFIG_FB_CIRRUS (note that the CONFIG_ prefix is added automatically and should be omitted from the Kconfig file itself). Let’s jump to the help section for a second. We can read that this configuration option enables support for various Cirrus Logic video cards. In the second line, we can see that this variable is in the type of “tristate”. It means that this configuration option has 3 possible states; Disabled, Enabled and Modulized. Each configuration option must have a type. The available types are: “bool“, “tristate“, “string“, “hex” or “int“. Each type has its own properties. The bool type can be either Enabled or Disabled. The string type will contain a list of characters. The hex type will contain a hexadecimal representation of a number (usually used to assign hardware address and ids) and the int type contains numbers. Right next to the type, you’ll find the prompt string that represents the configuration option in the menu (or in the prompt). Next, shows the dependency of this configuration option. It means that this option will not be available for selection if the CONFIG_FB option is not enabled and either CONFIG_ZORRO or CONFIG_PCI are enabled. In this way, the driver developer can be sure that this driver will not be available unless the Frame Buffer framework is enabled and the target has either Zorro or PCI interfaces enabled (otherwise, the driver will fail to compile or run). The next lines, with the select option, show options that the driver needs in order to operate correctly. In this example, we see that the driver requires a few Frame Buffer options to be enabled.

An example from net/Kconfig:

menu "Networking options"source "net/packet/Kconfig"

source "net/unix/Kconfig"

source "net/xfrm/Kconfig"

source "net/iucv/Kconfig"config INET

bool "TCP/IP networking"

---help---

These are the protocols used on the Internet and on most local

Ethernets. It is highly recommended to say Y here (this will enlarge

your kernel by about 400 KB), since some programs (e.g. the X window

system) use TCP/IP even if your machine is not connected to any

other computer. You will get the so-called loopback device which

allows you to ping yourself (great fun, that!).

…

…

endmenu

This example shows how to create a submenu. In this example, a sub menu in the name of “Networking options” will be created, and a bunch of related configuration options will be placed inside it. This menu also includes some other Kconfig files using the keyword “source”.

You can browse the Kconfig files in the kernel and learn from the many examples available. That’s the best way to learn.

Adding a new configuration option

In order to add a new driver or feature to the Kernel, first we’ll need to select the appropriate location inside the tree to store the code. Once selected, we’ll need to add a new configuration option in the corresponding Kconfig file, and update the directory’s Makefile to compile our new code, once our configuration option has been selected.

For our example, let’s add a new driver under the networking drivers. In the following example, our new driver contains two files; my_obj_file1.c and my_obj_file2.c. In order to add this driver in the kernel, we need to do the following:

Select a unique configuration option name, (or names, in case the driver/feature has additional configuration parameters). Otherwise, there will be a collision and a configuration mess. You can grep the .config file to make sure your selected name is not taken. For this example, the name is CONFIG_RT_EMBEDDED_TEST_DRIVER.
Define the configuration option type(s). The main feature/driver can be Boolean type (Enabled/Disabled), or Tristate type (Enabled/Disabled/Modulized). Other configuration types are driver specific (like hardware address, number of buffers, identification string, etc.).
Edit the appropriate Kconfig file. In this example, we will edit the drivers/net/Kconfig file:

Next, we add our new configuration option in the appropriate Makefile in order to build the code when it is enabled. In this example, we will edit the drivers/net/Makefile:

Now the new driver is a part of the Linux kernel. In the next step we must re-configure the kernel because its configuration has been upgraded with a new feature. We can use either configuration possibilities listed in the begining of this article, except for option 4 (its a new configuratin option which is not listed in the default configuration file yet). When using either menu configuration options, the new item will appear with [NEW] marking near it. When using oldconfig option, it will prompt for your selection, while capitaling the default option. In our example, since the default mode is Modulized, we will see n/y/M.

Here we need to select what to do with the new option, according to our requirements.

Resources

Kconfig description: http://lxr.linux.no/linux+v2.6.34.1/Documentation/kbuild/kconfig.txt
The Kconfig language: http://lxr.linux.no/linux+v2.6.34.1/Documentation/kbuild/kconfig-language.txt

Creating and using proc files

Hai Shalom — Wed, 30 Jun 2010 15:21:27 +0000

The proc files implement the simplest method of communication and data exchange between the Linux kernel or its drivers, and the user space applications or human users. The kernel provides many proc files for setting various settings, and getting plenty of information (see the article about process procs). Each driver or loadable kernel module can add more proc files in the /proc directory, in order to set its specific parameters or to get its specific information. The proc files are not really files, but referred as “pseudo-files”, where all the /proc directory is a “pseudo-filesystem”. The reason for this name convention is the fact that unlike other filesystems, these files do not really exist. Instead, the kernel (or each driver which needs a proc file) registers a file, defines its permissions and implements a read and/or write functions. The read function will be invoked whenever a user space application wishes to read information, so the information that is read is actually generated by the kernel upon request. The same applies for the write function.

Now that we know what a proc file is, let’s see how we can create and use it. First, in order to gain access to the proc functions and types, you will need to include the linux/proc_fs.h header file.

Creating a new sub directory in /proc

If you wish to place your proc files in a designated sub directory, it is possible to create one. This is usually done when your driver has more than one file, and therefore, it is easier to access them and a cleaner implementation. The following example function creates a newdirerctory under /proc directory in a given name, and returns a pointer. Note the S_IFDIR flags which marks this entry as a directory (defined at linux/stat.h file).

struct proc_dir_entry *make_proc_dir( char *name )

{

   struct proc_dir_entry *my_proc_dir = NULL;

/* Create a new directory */ my_proc_dir = create_proc_entry( name, S_IFDIR, NULL );

if (my_proc_dir) { /* No Read and Write functions here */ my_proc_dir->read_proc = NULL; my_proc_dir->write_proc = NULL; } return my_proc_dir; }

Registering a new proc file

The next stage, is to actually create a file entry. As I mentioned, the proc files are not real files, but yet we need to declare them by registering them with the proc filesystem. In this example, the file “rt-embedded” will be created under /proc. It is possible to create a file deeper in the tree simply by specifing the desired location (for example; net/rt-embedded will create the file under /proc/net). The file is created with 0644 permissions, means that the owner user can read and write, and the rest can read only (don’t forget the linux/stat.h file). The last parameter is optional and denotes the entry parent. Once the entry is created, we need to specify the read and write functions, according to the permissions we declared. We’ll see then next.

struct proc_dir_entry *my_proc_file = NULL;

/* Create a proc file */ my_proc_file = create_proc_entry("rt-embedded", S_IRUSR |S_IWUSR | S_IRGRP | S_IROTH, NULL);

if (my_proc_file)

{

  /* Setup the Read and Write functions */

  my_proc_file->read_proc  = my_proc_read;

  my_proc_file->write_proc = my_proc_write;

}

Implementing the read function

Once a file is registered, its contents is generated upon request by a “read” function, which fetches all the required information from the driver’s internal data structures and formats the data in a human readable form. The read functions must be implemented only if there is a read access to the file, or in other words, when this file provides information. The read function’s prototype is a bit complicated. Let’s see the example first:

int my_proc_read( char *buf, char **start, off_t offset, int count, int *eof )

{

    /* Initialize the total length */

    int len = 0;

    /* Format the data in the buffer */

    len += sprintf( buf + len, "my first line of proc data\n" );

    len += sprintf( buf + len, "my second line of proc data\n" );      

   

    /* If our data length is smaller than what

       the application requested, mark End-Of-File */

    if( len <= count + offset )

        *eof = 1;

    /* Initialize the start pointer */

    *start = buffer + offset;

    /* Reduce the offset from the total length */

    len -= offset;

   

    /* Normalize the len variable */

    if( len > count )

        len = count;

    if( len < 0 )

        len = 0;

    return len;

}

There are many bad implementations of the read function which ignore all the parameters. Here’s the description of each parameter:

buf: A pointer to a buffer that the kernel allocated for the proc file data.
start: A double pointer to indicate the start data.
offset: The offset of the data, from the start, that is currently requested.
count: The amount of bytes the reader wants.
eof: End-Of-File flag.

When calling the read function, the kernel usually allocates a page of memory (usually 4KB) for the data. In case the reading application allocates a small buffer, it might require that this function will be called multiple times in order to complete reading the entire data. Therefore, the kernel also provides the count, start and offset variables. The read function must honor the offset variable and set the start pointer to the appropriate offset, as requested. Only when the complete data has been transfered, then the read function turns the EOF flag on. The kernel will not call the read function again for this purpose, since the function marked that it has no more data to send.

Implementing the write function

The write function is used only in case the kernel or driver allows data to be set by the caller. For example, if you have 8 LEDs on your target, your LED driver can accept writing a number in a designated proc file which will light up a specific LED. Implementing this function is not mandatory in case the proc file is in read-only mode. When working with the write function, the data needs to be copied from the user’s address space to the kernel’s user space.

int my_proc_write( struct file *file, const char __user *buffer, unsigned long count, void *data )

{

    char *page;

    int size = sizeof(int); /* Example, we require to read a buffer in the size of an integer */

    int my_data = 0;

    /* Make sure that the data is in the right size */

    if (count < size)

       return -EINVAL;

    /* Allocate memory for the kernel buffer */

    page = (char *) vmalloc(size);

    if (!page)

       return -ENOMEM;

/* Copy the data from the user space */ if (copy_from_user(page, buffer, size)) { vfree(page); return -EFAULT; }

    /* Now do something with the data, here we just print it */

    sscanf( page,"%d",&my_data );

    printk( "User has sent the value of %d\n", my_data );

    /* Free the allocated memory */

    vfree(page);

return count; }

The file and data parameters are not required for general use. The kernel sends the user space buffer in the buffer variable and its size in the count variable. We need to verify that the data that was sent is in the right size, and in case we transfer structures of data, we can add a magic number member which will hold a constant value, just to make sure that the expected data type was actually sent. Then, we must copy the data to the kernel space because it is impossible to access it as is with the user space pointer. After we’ve copied the data, we can start parsing and working on the data. The return value tells the kernel if we failed (by returning a negative value) or processed all the data (by returning the count number).

Removal of a proc file

It is possible to remove a proc file from the proc filesystem. This function is mostly used by Loadable Kernel modules which are in the process for shutting down during removal (using rmmod). In order to remove a proc file, just use the following function with the file name:

remove_proc_entry( name, NULL );

Usage example from a user space application

Now let’s see how we can access these proc files. The simplest way to access them is manually, in the Linux shell. In order to read a contents of a proc file, we simply use the “cat” command and the file name, for example: cat /proc/my_proc_file. In order to write data to a proc file, we simply use the “echo” command with redirection of the output to the destination file, for example: echo “write data” > /proc/my_proc_file.

We can also access these files from our user space applications. The process is actually quite similar and very simple. Each program who wants to read (or write) data, is required to open the file in the standard way (either with stream access; fopen, or descriptor access; open). Then, it reads the contents of the file to a buffer and then it closes it. The data, is always in human readable text mode. Therefore, the program needs to convert it back to binary representation in case it needs to work or analyze the data.

Other proc helper functions

The kernel provides some more proc APIs that may short-cut or encapsulate some of the actions or parameters when using the standard create_proc_entry function. Some examples of such APIs:

proc_mkdir
proc_symlink
proc_net_create
create_proc_read_entry

Refer to the linux/proc_fs.h file for the complete list of available API.

PCD – Process Control Daemon

Hai Shalom — Mon, 21 Jun 2010 12:00:54 +0000

DOCUMENTATION

EXAMPLES

What is PCD?

PCD (Process Control Daemon), is an open source, light-weight, system level process manager for Embedded Linux based products (Consumer electronics, network devices, and more). With PCD in control, it is guaranteed that the system’s boot up time will be reduced, and its reliability, availability and robustness will be enhanced.

Why do you need PCD in your Embedded Linux based product?

With PCD driving your product, you will gain:

Enhanced control and monitoring on all the processes in the system.
Simple, yet powerfull means to configure each process and service in the system:
- When to start each process, configure its dependencies.
- What resource or condition to check, per each process, in order to verify its successful startup.
- What recovery action to take when a process fails to start or crashes during its work.
Reduced system startup time
- Rules are started as soon as their start condition is satisfied.
- No need for non-deterministic delays. Dependencies are well defined and enforced.
- Rules without inter-dependency are started in parallel.
Enhanced debug capabilities of faulty processes
- Exception handlers and logs provide useful and detailed information about process crashes, such as the process name and id, exception description, date and time, fault address, register dump, map file and more.
- Crash log storage in non-volatile memory for post-mortem/offline analysis.
Improvement in system stability and robustness.
Generation of a graphic representation of the system startup sequence for further analysis.

Some Background

The PCD project was started due to a need for a process controller which was missing in one of TI’s broadband solutions. In the early stages of the software development, the system was started using scripts, where the init.d/rcS script was doing some of the work, and it was calling other scripts which loaded all the processes, drivers and services. Needless to say, this method was far from fulfilling the required task, especially for a mass-production product. There were cases where we were not able to determine when a driver was initialized, so we added sleeps. In other cases, some process has crashed and we didn’t even notice it, but experienced unexpected system behavior. The debugging capabilities were also minimal. It is not trivial to work with gdbserver on an embedded target, it requires a stable communication channel and symbols.

Eventually, the PCD project has been initiated to provide an adequate solution to all these issues, and it has improved the product’s quality, robustness and stability. We were able to catch and fix all the crashes early using the extended debug information. Furthermore, it has reduced the boot up time significantly because the sleeps were not required anymore, and due to activation of many non-depended processes in parallel.

The PCD was originally designed and developed by Hai Shalom as a proprietary solution for Texas Instruments. During the last stages of the development, the PCD project was released by TI as an open source software and the project’s community is in process of growing. New programmers which find the PCD useful and interesting project are starting to contribute to the project with their ideas and innovations.

Supported Architectures

The PCD is a user space application, therefore its exposure to the actual architecture is limited. However, supported platforms have an enhanced and detailed crash dump which displays all registers, thus allowing an easier debug.

The PCD fully supports the following architectures:

ARM
MIPS
x86
x64

Non-supported architectures should also work, but without the enhanced exception details.
If PCD does not support your platform, it is possible to support it if you’ll send the hardware and BSP. Please use the contact page to contact me.

Already have a process controller in your product?

You can still use the PCD in “crash-daemon only” mode. In this mode, the PCD will not start your system nor monitor its processes. Instead, it will remain idle until one of your application crashes, and when that happens, the PCD will generate and log the detailed crash dump which could help you isolate and fix the bug. There is no need for any rules file, and in case you’ll like using PCD, you might consider upgrading your existing process controller.

PCD High Level Presentation (updated)

The following slides provide a high level overview on the PCD project.

Integrate PCD in 4 steps

Thinking about the integration effort? It’s not that difficult. Follow these 4 steps in order to integrate PCD in your Linux based system:

Step 1: Get introduced

Download the PCD source code from the PCD project page at SourceForge.
Download the PCD documentation and training presentation.
Unpack, compile and install the PCD in your target’s file system.

Step 2: Write rules

Determine the order and dependencies of each process, daemon or service.
Determine what is the condition for a successful init of each process.
Determine the recovery actions to take if they fail.
Write a rule per each process using the PCD’s simple syntax.
Run pcdparser to check your rules file for syntax errors.
Generate a dependency graph which describes your system’s startup.
Install the rules file(s) in the target’s file system.

Step 3: Enhance your processes

Install the PCD’s exception handlers in your processes (you’ll gain a detailed crash log).
Replace any fork-exec or system( ) calls with the PCD API with the corresponding rule.
Send a “process ready event” to the PCD when your process has completed its initialization.

Step 4: Activate PCD

Remove any script based startup. Remove explicit calls to daemons and processes.
Remove any delays and sleeps which were used for process synchronization.
Add one call to PCD along with your new rule file(s).

That’s it!

Your system will now be booted with the PCD synchronizing and monitoring everything. In case of an error, it will provide a detailed crash log and take the necessary steps in order to recover the system.

PCD Licensing

The PCD project is licensed under the GNU Lesser General Public License version 2.1 (LGPLv2.1), as published by the Free Software Foundation.
To view a copy of this license, visit http://www.gnu.org/licenses/lgpl-2.1.html#SEC1
There are no purchase costs or royalty fees.
This license allows commercial use without the need to expose proprietary source code. Any enhancements or features that are added to the PCD must be submitted back to the community.

PCD Resources

Browse the PCD website, join the community, request features, report bugs, discuss in the forum and get help:

PCD project page at SourceForge: http://sourceforge.net/projects/pcd/
PCD presentations: http://www.slideshare.net/haish
PCD Design document: http://sourceforge.net/projects/pcd/files/docs/PCD-High-Level-Design-Document.pdf/download
PCD Documentation http://www.rt-embedded.com/blog/pcd-process-control-daemon/pcd-documentation/
Examples for PCD debug capabilities: Resolving segmentation faults with PCD, Resolving alignment traps with PCD.

Need more information, training or support?

If you are managing a commercial project, it is possible to receive a dedicated, on demand PCD training and support. Please use the contact page for more details.

Size optimization – Part 3

Hai Shalom — Mon, 14 Jun 2010 16:16:54 +0000

Now that you’ve optimized your applications (programs) and archives (static libraries), we’ll discuss how to optimize your shared libraries. Unlike archives which are used only during link time on the host machine, shared libraries reside on the target’s file system, and cannot be reduced using the the same techniques. Furthermore, when you create a shared library, you can not know which functions will be used by the applications and which functions are not used. It is also not trivial to figure out the dependency between the library’s functions (which function requires another one inside the library). Therefore, shared libraries always contain the full set of functions. The question that we ask is; how much storage space is wasted for unused code of a shared library?

Assuming we always use small and light-weighted libraries, it may not be a serious issue, although the sum of many small libraries may add up eventually. However, there are cases where large libraries are used, for example OpenSSL. In terms of embedded systems, this library is huge and requires a large part of the system’s storage. In most cases, we don’t need the full functional set of this library. Some libraries (OpenSSL too) allow a level of configuration which allows the user to disable some of its features to reduce the final size. We can also work hard to manually remove some objects from the library according to our current needs. But what happens if we add a new program that requires some functionality that we already removed? We’ll end up with a broken system due to unresolved symbols. How can we automate shared library reduction? Here’s the answer: use mklibs.

What is mklibs?

mklibs is a script which was originally used for creating bootable floppy diskettes. Due to the fact that these historical storage medias were very limited in size, it was necessary to save only the needed code, and nothing else. The Internet is full of variants of this script, some variants use shell syntax and others use Python syntax. In the resources section I provided links for some variants. Nowadays, in modern embedded systems we have a similar problem. Embedded systems also have limited storage, not because of technological limitation, but due to the final cost. Smaller storage devices are simply cheaper.

How does it work?

The mklibs uses recursive iterations on all the program’s symbols and share library simbols as follows (quoting from the script):

Gather all unresolved symbols and libraries needed by the programs and reduced libraries.
Gather all symbols provided by the already reduced libraries (none on the first pass).
If all symbols are provided:
- we are done.
Else:
- Go through all libraries and remember what symbols they provide.
- Go through all unresolved/needed symbols and mark them as used.
- For each library:
  - Find pic file (if not present copy and strip the share library).
  - Compile in only used symbols.
  - Strip.
  - Back to the top.

How can it be used?

Please note that it may not be so trivial to use this script in your system, there are plenty of potential issues that will prevent the script from completing its task, or the outcome to actually work on the target. Here are a few pointers that will help you (I spent a lot of time figuring out everything):

Your shared libraries must be created with the soname flag enabled. This flag embeds the shared library name inside the shared library itself. Here’s an example: “-Wl,--soname,$(so_target)“, where the so_targetvariable contains the full library’s name (such as libmath.so).
Per each library you must create a pic file, which is actually a static library version of your shared library, compiled with function and data sections (see part 2 for more details). The archive name must be with the same name of the shared library, with the suffix _pic and the extension .a. In order to accomplish this, you need to use the linker to create a single object file from the list of library objects, and then call the archiver to create an archive from this linked object. For example, the following command will create a single object out of the library’s objects: “ld -r -o $(so_target)_pic.o $(objs) $(LDFLAGS)“, the following command to create an archive: “ar rc $(so_target)_pic.a $(so_target)_pic.o“. The last command is required after the archive is created: “ranlib $(so_target)_pic.a“.

Once the script encounter a shared library with the name embedded inside and a matching pic file, it will be able to maximize the reduction of this library. Otherwise, it will only strip it, which will not reduce it size further (stripping is usually done anyway).

The following parameters are supported by the script. Note that this list may change between the various variants:

-d, --dest-dir DIRECTORY create libraries in DIRECTORY -D, --no-default-lib omit default libpath ( /lib/ : /usr/lib/ : /usr/X11R6/lib/ ) -L DIRECTORY[:DIRECTORY]... add DIRECTORY(s) to the library search path --ldlib LDLIB use LDLIB for the dynamic linker --libc-extras-dir DIRECTORY look for libc extra files in DIRECTORY --target TARGET prepend TARGET to the gcc and binutils calls --root ROOT search in ROOT for library rpaths -v, --verbose explain what is being done -h, --help display this help and exit

For embedded systems, the following flags are recommended:

-D: You must configure the script not to use the host’s libraries.
-L : Specify where your libraries reside.
–ldlib : Specify where is the loader library. For uClibc systems, the loader is called ld-uClibc-0.9.XX.so, where XX stands for the uClibc version (28,29,30…).
–libc-extras-dir : Specify where the uClibc libraries reside. In uClibc systems, they reside in the uClibc directory, under lib/.
–target : Specify your cross compiler prefix.
-d : Specify the destination output directory where all the reduced libraries will be stored.

If you are using uClibc in your system, you should consider using the uClibc version of mklibs (see link in the resources section).

Estimated size reduction

If your system uses large libraries, especially ones that came from Open Source, and designed for general purposes, there is a good chance that you’ll see a lot of reduction in size. In the platform I am working with, I was able to reduce the final size of the OpenSSL library by 25%. It means that there were a few hundred bytes which were not used, just sitting on the precious and limited storage media. On the other hand, if most of the libraries are light weighted, and locally written, usually there are no excess functions which are not used, as most are used by design.

Other pointers

The script will give you hard time until it will produce the required results. However, if your system uses open source libraries and is limited with storage, this script will produce the most of size reduction. In case there is a shared library which is breaking due to this optimization process (it is possible), you can disable the optimization on this specific library by removing its pic file. As I mentioned, without a pic file, the script will only strip the library and keep it intact.

Resources:
mklibs from debian: http://ftp.debian.org/pool/main/m/mklibs/
mklibs with uClibc fixes: http://cristi.indefero.net/p/buildroot-cristi/source/tree/8a25f1c4e78f02b1e722ccc2f659c025ba662296/toolchain/mklibs/mklibs.py
http://tracker.coreboot.org/trac/buildrom/browser/buildrom-devel/bin/mklibs.py

Size optimization – part 2

Hai Shalom — Tue, 08 Jun 2010 16:38:31 +0000

The tips and information in part 1 are too general and common, and it is highly likely that you’ve already implemented them. In this article, I will show how we can reduce the size of static libraries (archives) and applications (programs) by specifying advanced compilation flags which utilize the special properties of archives and applications.

Optimizing the static library (archive)

Static libraries are actually archives of objects. An archive can contain one or more objects, where each object can contain one or more functions. When an application requires a specific function from an archive, it links with it, and the linker copies the entire object which contains this function from the archive to the application. This could cause an excess duplication of an unneeded code, in case this object contains some other functions and data, which is usually the case. In order to optimize the link process here, and copy the required function only, we can tell the compiler to put each function and global data variable in a separated section inside the object. Although this action slightly increases the size of the archive, it will produce better results in the target. Don’t forget that the archive is not a part of the final file system in the target. This option allows the linker to select only the relevant function’s code from the archive, without copying the entire object - thus reducing the final size of the application. The compiler flags for this options are given per each source code compilation:

--ffunction-sections --fdata-sections

Note that all the optimization techniques specified in part 1 can be also applied for static libraries.

Optimizing the application

The static library optimization which I described above can be also applied for applications, although it will yield less reduction because most of the functions inside the application are usually used (unused functions are usually removed by the authoer). When the compiler finishes to compile all the source files and you have a list of objects which need to be linked for the final application, you need to instruct the linker to clean the unused sections. If you remember, enabling the function and data sections allows the linker to remove any unused functions and data variables. The following command needs to be specified in the final link process of your application:

-Wl,--gc-sections

Another property of an application in Linux environment is that the application is a single entity without any connections to other applications. This property allows us to instruct the compiler to look at all the application’s code as a single entity (a whole program) and enable some more agressive optimizations which can be enabled only when the compiler sees the whole program (not possible when compiling single source files). This option is available from gcc version 4.1 and newer. In case you are using Makefiles which compile each source file separately, you will have to change it if you want to enable this option. Instead of compiling each source file, you need to compile the whole application in a single command line, while specifying all the source files and compilation flags, and the following flag to enable the optimization:

--combine

The result of this action is a large and optimized object file. Next, when you need to link the object, specify the following flag:

-fwhole-proram

While this method reduces the size of the application, it also has some drawbacks. One of them is lack of dependency, because it requires to recompile the entire source files (although in a single command) per each change in one of the sources. This method also requires more CPU power and memory in the host computer.

Here’s an example Makefile snippet for this method:

# Makefile snippet for enabling the whole-program optimization.

# Author: Hai Shalom, http://www.rt-embedded.com/

#

# Assuming the following variables:

# TARGET - contains the target application name

# CFLAGS - contains the project default C flags

# LDFLAGS - contains the project default linker flags

# SRC - contains the application source files

# LIBS - contains the libraries the application links with


$(TARGET)_combined.o: $(SRC)

 @echo "Compiling  $(SRC)"

 @$(CC) $(CFLAGS) -c --combine $(SRC) -o $@
$(TARGET): $(TARGET)_combined.o

 @echo "Linking  $@"

 @$(CC) -o $@ $(TARGET)_combined.o $(LDFLAGS) -Wl,--gc-sections -fwhole-program -Wl,--sort-common -Wl,--sort-section,alignment -Wl,--start-group $(LIBS) -Wl,--end-group

The last method to squeeze the last unused bytes in an application is to instruct the linker to sort the sections and remove unneeded padding’s. The following flags need to be added to the final link command:

-Wl,--sort-common -Wl,--sort-section,alignment

Note that all the optimization techniques specified in part 1 can be also applied for applications.

Size optimization – part 1

Hai Shalom — Sun, 30 May 2010 15:52:32 +0000

In embedded systems, size does matter. Embedded products are usually limited in resources of RAM and storage (usually Flash) and the cost pressure forces you to think about creative ways to reduce the overall size of the binary applications and libraries, without reducing the features and functionality. In the years I’ve been working in the embedded systems business, I often deal with requirements to reduce the overall size of the application due to system limitation. Therefore, I have a lot of experience in this field which I am going to share with you in this “Size optimization” series. The series will include information about size optimization and reduction, general tips, optimizing applications, static libraries, shared libraries, file systms and the Linux kernel.

Background

Embedded systems are usually equipped with limited amount of RAM (32MB – 128MB) and even more limited amount of flash memory (1MB – 16MB), where the sizes of the devices grow in the power of 2. So if your application image requires 4.1MB of storage, your product must be equipped with a 8MB flash device instead of a cheaper 4MB device, if the image was just a bit smaller. In the following series of articles, I’m going to show some techniques to reduce the binary size without actually changing the code itself, which could yield also better results. I will dedicate one or more article about writing a more efficient code as well. The input for this purpose is the source code which is refered as a “black box”.

General tips for size optimization

The following tips will yield smaller binary output.

Configure the compiler

The compiler can be configured to produce smaller binaris by applying optimizations to the assembly code that result in smaller output. As described in the “using the gcc” article, enable this option by adding the “-Os” flag in the command line. In case another optimization level is used (-Ox, where x could be 0,1,2,3), you’ll need to replace it. Note that replacing the optimization with size optimization may also result in performance impact, especially if levels 2 or 3 were previously used. These levels configure the compiler to modify the code to yield better performance, and level 3 even increases the output size in order to improve performance (like loop unfolding for example). The size optimization works on the other direction, where the performance is less important than the actual output size. This option is mostly used when compiling a boot loader which must be confined to the first 1 or few sectors in the flash, but can be also used for compiling the kernel and user space applications. The following table shows the output size (in bytes) of an application which was compiled with various optimization levels:

No optimization	-O2	-O3	-Os
503,751	466,469	522,369	449,817

As we can see from this table, the level 2 optimization has the best trade-off between size and performance, comparing to “no optimiztion”. The size in level 2 optimization has been reduced by 8%. Level 3 optimization has actually increased the size by 4% (with the supposed performance increase) and level s has reduced the file size by 11% (with some impact of performance).

Stripping the output

By default, and especially when the -g flag was used in the compilation command line, the output contains some debug symbols which are useful for debugging and analysis. However, in you final target, these symbols are not required and can be removed (stripped) in order to reduce the size of the image. In order to remove these symbols, use the stripapplication which comes in the binutils package with the –strip-unneeded flag. The following table shows the size differences (in bytes) before and after stripping:

Before stripping	After stripping
449,817	162,916

Stripping the output is safe and has no effect on the performance. It does not change the behavior of the application, just removes unused data.

Compiling in Thumb mode (ARM platforms only)

The ARM family of CPUs provides an option to work in a reduced instruction set, which is 16-bit in length instead of 32-bit. This instruction set is referred as “Thumb mode“. This mode actually uses a compressed instruction set and thus resulting with a smaller code. The Thumb mode may reduce the output size by up to 30%. The performance figures are not always clear here. In case the CPU is 16-bit native, it will yield better performance. However, on a 32-bit CPU, the thumb mode may either reduce the performance or not impact it at all (it’s application depended). In a mixed instruction system, you also need to configure the compiler to produce interwork code, which is required when switching from the ARM mode to Thumb mode and vice versa. Therefore, in order to produce a Thumb output, use the “-mthumb -mthumb-interwork” flags. Note that you can’t compile the Linux kernel in Thumb mode and most of boot loaders due to use of ARM assembly code and other reasons. However, it can be very useful for user space applications and libraries.

Resources:
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0135a/BABFCAHG.html
http://www.rt-embedded.com/blog/archives/using-gcc-part-2/

Compilation warnings management

Hai Shalom — Sun, 23 May 2010 12:19:42 +0000

One measurement of the quality of your project’s code is the amount of outstanding compilation warnings. In my opinion, these compilation warnings could potentially be responsible for a system crash and unexplained behaviour, and are one of the top 5 destructive bugs. If you are a project manager, it is your interest to eliminate them, especially if your deliverable is the actual source code. As a customer, a software deliverable with plenty of compilation warnings appears to be non-professional just by looking at the compilation process. If you are a developer, it is your aim to deliver warning-less code.

In this article, I will provide a few methods and tips for warning management, meant mostly for project managers. As I mentioned earlier, code that compiles cleanly looks better and more professional. It is even more important if you deliver source code to your customers. Compilation warnings could sometimes indicate a real bug or a potential issue. Furthermore, if your project is already full of warnings, you have no control of new warnings which are introduced in each code change. Since the screen is already so full of warnings, you’ll most probably miss new ones. This is the way to complete chaos in your project in terms of warning management and control. If you want to gain control over this issue and eliminate the warnings in your project, follow the following steps.

1st step: Assess the amount of warnings

In the projects I manage, I don’t allow any warnings. However, suppose your current project contains warnings, first you need to assess the number of warnings in high-level. You can later divide the numbers to groups, according to the corresponding component. With these numbers at hand, you can assign warning-fixing tasks to your staff. The following shell script will help you getting the warning numbers. It will clean and compile the component in question and provide the number of outstanding warning count. You should allocate some time before each delivery deadline, and run this script, to make sure your project is meeting the warning count goals. In order to use this script, you need to specify the target name and optionally, a file name to contain the list of warnings. If you don’t have a target, specify “all”.

# !/bin/sh

# warning_count.sh - Count the warnings in a specified target

# Written by Hai Shalom - hai@rt-embedded.com

#

# usage:

# warning_count.sh [target] 

#

# The stderr file is optional and can be later used to examine all the warnings.

# If the target supports "target_clean" option, it will be used. Otherwise,

# the scripts cleans the whole system.

#

if [ $1 ]

then

    target=$1

else

    echo "$0" [target] \for stderr\>

    echo e.g.: "$0" all ~/stderr.txt

    echo The stderr file is optional and can be later used to examine all the warnings.

    exit 2

fi

echo -e "\n=========== Cleaning $target ===========\n" make "$target"_clean 2> /dev/null if [ $? != 0 ]; then make clean 2> /dev/null; fi

echo -e "\n=========== Building $target ===========\n" if [ $2 ] then stderrfile=$2 else stderrfile=/tmp/warning_count_`whoami`.tmp fi

make $target 2> $stderrfile && \ cat $stderrfile | grep "warning: " |\ echo -e "\n=========== Total warnings:" `wc -l` "===========\n" if [ $? != 0 ]; then echo -e "\n=========== Compilation failed, aborting! ===========\n"; exit 2; fi if [ $2 ] then cat $stderrfile | grep "warning: " |\ echo -e "\n=========== Total warnings:" `wc -l` "===========\n" >> $stderrfile else rm -rf $stderrfile fi

Once the script has completed the run, you’ll see the current warning count in your project, and the warning list will be saved in the specified file. Note that if your Makefile does not specify any warning level to the compiler, many warnings will be inhibited. In case, it is recommended to enable all warnings by adding “-Wall” to the CFLAGS variable in your Makefile. The second article about using gcc provides a nice description about setting the required warning level.

2nd step: Assign warnings to developers

Divide the warnings between your developers in order to reduce the warning count (or eliminate them completely). Pay attention, while most warnings could be a minor issue, such as a missing header file, or an unused variable, some may hide a potential problem. Instruct your staff to consult witheach other or hold brainstorms on warnings they are not sure how to fix. For example, casting a variable may eliminate the warning, but not the problem! In this example, the developer must understand what is the cause for the type mismatch. What if the types are not in the same size? there could be a memory overrun in this case.

3rd step: Perform code reviews

Each warning fix must be reviewed, in order to make sure it was fixed properly.

4th step: Lock clean modules

Once a software module has been cleaned from compilation warnings, it is possible to lock it from accepting new warnings. In order to do that, you can instruct the compiler to treat warnings as errors. Any new warning which will be introduced will fail the compilation process and the developer will have to fix it immediately in order to proceed with the work. This approach has advantages and disadvantages. The advantages are: It will keep the software module clean from warnings forever, it will force the developer to fix any new warning immediately, otherwise, he may postpone it until there are many warnings, and sometimes may even forget to fix them at all. The big disadvantage of this approach is that it is more difficult to develop this way because the compilation will fail for each small issue which may be irrelevant. If you decide not to use this approach, you shall have to run the script once in a while to make sure that your warning count does not raise.

Third party code

In some cases, you may not want to or could not change some code because it is provided by a third party vendor, or has a special license that prevents you from changing it. If you verified that these warnings are not affecting your system, you can instruct the compiler to inhibit all warnings from this software module. Again, this approach has advantages and disadvantages. The main advantages are the look of the compilation process and the total warning count of your project. As a big disadvantage I can say that if a real bug is hiding there, you won’t see it with no warnings.

Resources:
http://www.rt-embedded.com/blog/archives/using-gcc-part-2/

Monitoring process creation, execution and termination

Hai Shalom — Mon, 17 May 2010 07:17:17 +0000

Last week, one of the company’s customers had a major issue with a mysterious process that is periodically spawned, and when it runs, it allocates a resource and terminates without freeing it. Unfortunately, the allocated resource is a shared memory segment, which is not controlled by the kernel. Unlike dynamic memory allocation which gets cleaned up when a process terminates, this resource kept on leaking until it was completely exhausted. Once all the shared memory was consumed, the system was unable to operate correctly, although the kernel was still alive, and there was plenty of free RAM. This is just a single scenario where you might need to get a clear picture about the running processes in the system. How can we log and monitor the creation, execution and termination of processes in the system? Just continue reading.

In many cases, you may encounter a situation when you need to work on a system which you are not completely familiar with. There could be plenty of processes running in the system, which are started using fork and one of the exec functions. It is not a simple task to debug it, at least until you get more familiar with it. In this article I will show how you can easily patch the kernel to display a message each time a new process is created using fork, when a new process is loaded and executed using one of the execfunctions and when a process is about to terminate. These messages could surprise you in the level of details, because the amount of process activity is relatively large, especially during the kernel start-up.

Patching the fork system call

When a process wants to create a new child process it uses fork. This function is actually a system call which eventually ends up in kernel/fork.c file, in the function do_fork. This function creates a clone of the calling process, sets up its stack, assigns a new process id and links the new process with its father. Then, it returns the with the new process id. Note that fork only clones the process, but does not start a new one. After the fork, the process is running concurrently in two copies, with minor differences; the process id, the parent process id and the stack.

To add the patch: Locate the do_fork function in kernel/fork.c file. Look for the copy_process function and place the green code inside the if statement.

p = copy_process(clone_flags, stack_start, regs, stack_size, parent_tidptr, child_tidptr, nr);

/*

 * Do this prior waking up the new thread - the thread pointer

 * might get invalid after that point, if the thread exits quickly.

 */

if (!IS_ERR(p)) {

    struct completion vfork;

    /* Hai: Display a message when a new process is forked */

    printk( KERN_INFO "KERNEL: Process %s (pid = %d) forked a new process,

            pid = %d\n", p->comm, p->parent->pid, p->pid );

Patching the exec system call

In case a new program is needed to be loaded, then in the next step, the process calls one of the exec system calls. This is the common scenario. Otherwise, if the intention was to actually have two copies of the same program, then the exec system call is not required. All these system calls eventually end up in fs/exec.c file, in the function do_execve. This function loads the new program code and schedules its new execution. In the right time, it will call its main function.

To add the patch: Locate the do_execve function in fs/exec.c file. Place the green code in the start of the function, right after the defiinitions of the variables.

/*

* sys_execve() executes a new program.

*/

int do_execve(char * filename,

    char __user *__user *argv,

    char __user *__user *envp,

    struct pt_regs * regs)

{

    struct linux_binprm *bprm;

    struct file *file;

    int retval;

    int i;



    /* Hai: Display a message when a new process is executed */

    printk( KERN_INFO "KERNEL: Executing %s,pid = %d\n",

            filename, current->pid );

Patching the exit system call

When a process is about to terminate, the exit system call is activated. This system call eventually ends up in kernel/exit.c file, in the function do_exit. This function cleans up the process, frees its monitored resources, terminates its services and sends a signal to its father.

To add the patch: Locate the do_exit function in kernel/exit.c file. Place the green code after the in_atomic check.

if (unlikely(in_atomic()))

   printk(KERN_INFO "note: %s[%d] exited with preempt_count %d\n",

          current->comm, current->pid,

          preempt_count());



/* Hai: Display a message when a process terminates */

printk( KERN_INFO "KERNEL: Process %s (pid = %d) is terminating.\n", tsk->comm, tsk->pid );

The results

Here’s the output of a basic “hello world” program, after the patches were applyed on the kernel:

# ./hello

KERNEL: Process sh (pid = 160) forked a new process, pid = 170

KERNEL: Executing ./hello, pid = 170

Hello, world!

KERNEL: Process hello (pid = 170) is terminating.

As you can see, the patches provide detailed logs upon process creation, execution and termination. With this patches, it was possible to debug the system and find which process (among the many in the system) was causing the problem.

Resources:
http://linux.die.net/man/3/exec
http://linux.die.net/man/2/fork
http://lxr.linux.no/linux

Resolving memory leaks in the Linux kernel

Hai Shalom — Mon, 10 May 2010 10:11:50 +0000

The Linux kernel, like any other piece of software, can suffer from memory leaks. However, if you are using a recent version of the kernel, or a “commercial version” of it (such as Red-Hat, or MontaVista), the probability for memory leaks which originate from the kernel itself is relatively low (yes, I have high confidence in it). So what else could eat up the memory in the kernel space? There are a few possibilities for this:

Misuse of standard kernel API.
Local modifications which were added to the kernel in your project.
A loadable kernel module.

How do we know that we have a memory leak? first, follow the instructions in my previous memory leaks post. If your system showed memory leak symptoms, which result in constantly reducing amount of free memory, and on the other hand, the top utility did not show any increasing amount of memory by a specific process, then this leak comes from the kernel space.

Reading the kernel memory consumption (slab)

The kernel provides a file in the proc file system which shows its current memory consumption. The file name is /proc/slabinfo. If you’ll read this file in the first time, you will see a large table with many confusing numbers and you may panic (I know I did in the first time). Don’t worry, it’s not so complicated, and I’m going to explain how to read this file easily. Since this file shows a very long and wide table, it is recommended to redirect the output to a file and analyze it with an editor (use cat /proc/slabinfo > /var/myslabinfo.txt command or similar).

In high level, the memory management subsystem allocates memory in constant block sizes. Each block of memory is referred as an object. The kernel creates pools of objects in various sizes (in embedded systems the standard object sizes vary from 32 bytes to 128K bytes, usually in steps of power of two), and when a kernel code uses kmalloc, it will provide the memory from the appropriate pool (the lowest size pool that satisfies the memory requirement). Furthermore, each driver in the kernel can create its own pool with its specific object memory size, which is optimized for its needs (for example, the networking subsystem creates a pool with objects of 1504 bytes). These pools are also called “caches”, where each driver has its own cache. The kernel keeps track of each and every allocated object, so it knows how many objects were allocated in each pool.

Now let’s see the contents of the slabinfo file. I’ve truncated it a bit and left only the standard sizes, just to make this example easier:

name                 : tunables    : slabdata   

size-131072(DMA)       0      0 131072    1   32 : tunables    8    4    0 : slabdata      0      0      0

size-131072            3      3 131072    1   32 : tunables    8    4    0 : slabdata      3      3      0

size-65536(DMA)        0      0  65536    1   16 : tunables    8    4    0 : slabdata      0      0      0

size-65536             2      2  65536    1   16 : tunables    8    4    0 : slabdata      2      2      0

size-32768(DMA)        0      0  32768    1    8 : tunables    8    4    0 : slabdata      0      0      0

size-32768             0      0  32768    1    8 : tunables    8    4    0 : slabdata      0      0      0

size-16384(DMA)        0      0  16384    1    4 : tunables    8    4    0 : slabdata      0      0      0

size-16384             0      0  16384    1    4 : tunables    8    4    0 : slabdata      0      0      0

size-8192(DMA)         0      0   8192    1    2 : tunables    8    4    0 : slabdata      0      0      0

size-8192              1      1   8192    1    2 : tunables    8    4    0 : slabdata      1      1      0

size-4096(DMA)         0      0   4096    1    1 : tunables   24   12    0 : slabdata      0      0      0

size-4096             16     16   4096    1    1 : tunables   24   12    0 : slabdata     16     16      0

size-2048(DMA)         0      0   2048    2    1 : tunables   24   12    0 : slabdata      0      0      0

size-2048            192    192   2048    2    1 : tunables   24   12    0 : slabdata     96     96      0

size-1024(DMA)         0      0   1024    4    1 : tunables   54   27    0 : slabdata      0      0      0

size-1024             98    100   1024    4    1 : tunables   54   27    0 : slabdata     25     25      0

size-512(DMA)          0      0    512    8    1 : tunables   54   27    0 : slabdata      0      0      0

size-512              64     64    512    8    1 : tunables   54   27    0 : slabdata      8      8      0

size-256(DMA)          0      0    256   15    1 : tunables  120   60    0 : slabdata      0      0      0

size-256             105    105    256   15    1 : tunables  120   60    0 : slabdata      7      7      0

size-192(DMA)          0      0    192   20    1 : tunables  120   60    0 : slabdata      0      0      0

size-192             100    100    192   20    1 : tunables  120   60    0 : slabdata      5      5      0

size-128(DMA)          0      0    128   30    1 : tunables  120   60    0 : slabdata      0      0      0

size-128             794    810    128   30    1 : tunables  120   60    0 : slabdata     27     27      0

size-96(DMA)           0      0     96   40    1 : tunables  120   60    0 : slabdata      0      0      0

size-64(DMA)           0      0     64   59    1 : tunables  120   60    0 : slabdata      0      0      0

size-32(DMA)           0      0     32  113    1 : tunables  120   60    0 : slabdata      0      0      0

size-32             1078   1130     32  113    1 : tunables  120   60    0 : slabdata     10     10      0

size-96              560    560     96   40    1 : tunables  120   60    0 : slabdata     14     14      0

size-64              371    413     64   59    1 : tunables  120   60    0 : slabdata      7      7      0>

Let’s focus on the first 4 columns for the sake of our discussion. These columns describe the following:

Name: The name of the pool/cache. The name could be size-XXXX, where the XXXX stands for the object size in the pool. The name could also be a meaningful name which was selected by a driver in the kernel.
Active objects: Amount of allocated objects in this pool. For example, in the table above, there are curently 1078 allocated objects in the size of 32 bytes.
Number of objects: The highest amount of objects which were allocated in this pool. This number can be used as a “watermark” for understanding how many objects were allocated during the system’s life.
Object size: The net size of the object in bytes.

The rest of columns provide extra information which is not so relevant for memory leaks and I won’t extend the information on them currently.

Detecting a memory leak in the kernel

Assuming we concluded that the memory leak is not originating from the user space, we can use the slabinfo file to detect a memory leak in the kernel. Now, we need to log its output periodically and compare the outputs. You can use a shell script that runs in the background, cats the slabinfo file and sleeps for a minute. In case there is a leak in kernel space, we will see a constantly growing number in the second column of one (or more) of the caches. If this cache has a meaningful name which indicates its origin (name of driver, subsystem, service, etc.) we could try to isolate the leak in this relevant area. However, if the leak comes from one of the general caches, we need some more information in order to proceed.

Enabling slab debug options

The kernel provides some useful debug options to debug the slab allocator. In order to enable these options, select the CONFIG_DEBUG_SLAB and CONFIG_DEBUG_SLAB_LEAK kernel configuration options (In the configuration menu, they can be found in the Kernel Hacking menu). Also, compile the kernel with symbols ( CONFIG_DEBUG_INFO option). Unfortunately, there are kernel versions which do not support the leak option or it simply doesn’t work. The following patch simply extends the slabinfo file with a list of caller functions, which requested an object. The patch is simple and patches the mm/slab.c file, however, it may fail to patch correctly in your Linux version. In this case, it can be easily integrated manually.

*** mm/slab.c 2010-05-10 11:21:03.000687000 +0300

--- newmm/slab.c 2010-05-10 11:28:04.000000000 +0300

***************

*** 186,191 ****

--- 186,192 ----

  #define DEBUG  1

  #define STATS  1

  #define FORCED_DEBUG 1

+ #define SLAB_NCALLERS 128 /* Maximum amount of callers */

  #else

  #define DEBUG  0

  #define STATS  0

***************

*** 372,377 ****

--- 373,379 ----

     int node, int *this_cpu);

  static void enable_cpucache(struct kmem_cache *cachep);

  static void cache_reap(void *unused);

+ static void show_symbol(struct seq_file *m, unsigned long address);  /*

   * This function must be completely optimized away if a constant is passed to

***************

*** 499,504 ****

--- 501,513 ----

    */

   int obj_offset;

   int obj_size;

+

+  /* Structure to hold the caller information */

+  struct caller_info {

+   void *caller;

+   int ncalls;

+  } callers[SLAB_NCALLERS];

+  int slab_ncallers;

  #endif

  };***************

*** 3151,3162 ****

--- 3160,3205 ----

   return objp;

  }
+ #if DEBUG

+ /* A new function to lookup a caller in the list */

+ static int

+ lookup_caller(struct caller_info *cp, void *caller, int ncallers)

+ {

+  int l = 0, u = ncallers, i;

+

+  while (l < u) {

+   i = (l + u) / 2;

+   if (cp[i].caller == caller)

+    return i;

+   if (cp[i].caller < caller)

+    l = i + 1;

+   else

+    u = i;

+  }

+  return u;

+ }

+ #endif

+

  static __always_inline void *__cache_alloc(struct kmem_cache *cachep,

        gfp_t flags, void *caller)

  {

   unsigned long irqflags;

   int this_cpu;

   void *objp;

+ #if DEBUG

+  int n;

+

+  /* Lookup the caller in the list, append a caller if not found */

+  n = lookup_caller(cachep->callers, caller, cachep->slab_ncallers);

+  if (cachep->callers[n].caller == caller)

+   cachep->callers[n].ncalls++;

+  else if (cachep->slab_ncallers < SLAB_NCALLERS) {

+   memmove(cachep->callers + n + 1, cachep->callers + n, sizeof *cachep->callers * (cachep->slab_ncallers - n));

+   cachep->slab_ncallers++;

+   cachep->callers[n].caller = caller;

+   cachep->callers[n].ncalls = 1;

+  }

+ #endif
   cache_alloc_debugcheck_before(cachep, flags);
***************

*** 4127,4132 ****

--- 4170,4187 ----

        allochit, allocmiss, freehit, freemiss);

   }

  #endif

+ #if DEBUG

+  /* Show the list of callers */

+  if (cachep->slab_ncallers) {

+   int n;

+

+   seq_printf(m, " %d callers: ", cachep->slab_ncallers);

+   for (n = 0; n < cachep->slab_ncallers; n++) {

+    show_symbol(m, (unsigned long)cachep->callers[n].caller);

+    seq_printf(m, ":%d ", cachep->callers[n].ncalls);

+   }

+  }

+ #endif

   seq_putc(m, '\n');

   return 0;

  }

Let’s see a single line (as an example) in the slabinfo file with this extra information, once we’ve recompiled the kernel:

name                 : tunables    : slabdata   

names_cache            2      2   4096    1    1 : tunables   24   12    0 : slabdata      2      2      0 : globalstat       2      2     2    0                                  

0    0    0    0    0 : cpustat   1714      2   1716      0 2 callers: mount_block_root+0x28/0x288:1 getname+0x24/0xe0:1715

As we can see, the list of callers was added in the right. Assuming we saw that the number of active objects in the cache in the example is constantly growing, we need to look at the list of callers, and examine each and one of them. It is guaranteed that one of them is causing the memory leak. Per each name in the list, we need to locate its source code. If this function belongs to the kernel, it can be looked up at lxr. Otherwise, if it’s a function of one of your loadable modules, you can easily locate it in your source files.

The number near the function name is the offeset of the return address from within the calling function. In this example, we can locate the function mount_block_root in mm/do_mounts.c file. We can run the nm utility to resolve the address of the mount_block_root function on do_mounts.o, and then run the objdump utility on the do_mounts.o file, and add the offset to the address of the function to find the exact command.

# armeb-linux-uclibceabi-nm -S do_mounts.o | grep mount_block_root

0000012c 00000288 T mount_block_root

The address of the function is at 0x12c. Now let’s add 0×28 to this value, and we got 0×154. Here’s the output of objdump:

# armeb-linux-uclibceabi-objdump -S do_mounts.o | grep 154: -B 14

void __init mount_block_root(char *name, int flags)

{

 140:   e1a09000        mov     r9, r0

 144:   e1a08001        mov     r8, r1

        char *fs_names = __getname();

 148:   e5930000        ldr     r0, [r3]

 14c:   e3a010d0        mov     r1, #208        ; 0xd0

 150:   ebfffffe        bl      0 static void __init get_fs_names(char *page)

{

        char *s = page;        if (root_fs_names) {

 154:   e59f3220        ldr     r3, [pc, #544]  ; 37c

Since 0×154 is the return address, the call was made in the previous address (0×150). We can see that there is a call to kmem_cache_alloc function in this address. The source code shows that the __getname symbol is a macro.

Assuming you’ve found the function that leaks the memory in this method, it still doesn’t mean you found the problem. Next, you’ll need to understand why the leak happens by reviewing the code and all possible flows which use this resource.

Extending your application with Plugins

Hai Shalom — Tue, 04 May 2010 07:46:24 +0000

Nowadays, many software products support a mechanism that allows programmers and 3rd party vendors to enhance their features and capabilities without the need to recompile them, or actually, without the need to change any of their existing executables or libraries. This mechanism is usually called “Plugins”. Plugins are very popular today and can be also used to customize a program according to the requirements of your sysetm. In this article, I will present the Linux version of Plugins, how to support them in your application and how to write plugins. I will also provide examples of an application and a matching Plugin.

What is a Plugin?

A Plugin is a special kind of a shared library. It contains code that the main program can execute upon request. A Plugin library is not linked during compilation, but during run-time. Shared libraries that are linked during the compilation are then loaded automatically by the loader during run-time. Plugin libraries on the other hand, are not loaded automatically, but loaded explicitly upon request of the program which requires them. The Plugin libraries must adhere to the interface which was dictated by the main program and implement all the functions it requires.

What is the use of Plugins?

There are many uses for Plugins and many reasons for using them. Here’s a short list:

The easiest and most classic way to enhance a big program.
Providing users or customers a convenient means for hooks in your program without the need to rebuild it.
Plugins enable your users or customers to customize the program’s behavior upon an event or trigger.
Plugins enable customizations and enhancements even if you do not supply the source code.
Plugins can be supplied with different licensing than the main program. Note that this is in the gray area, because when you compile and link a GPL program, your Plugin is not a part of it (it is not needed, nor linked with the application).

In the next sections I will describe the steps for adding Plugin support and how to write a simple plugin. I will provide some code example as well.

Adding Plugin support in your application

1. Define the interface

When you want to add Plugin support in your application or library, as a first step, you need to define the interface. The interface is the glue between the main program and the Plugin, where all supported Plugins will adhere to this interface in order to function properly. The interface should include the following items:

A well defined Plugin init function name. Each Plugin will implement it and the main program will load and call it.
A data structure that contains:
- Plugin version (could be a single number, or major and minor). This will allow the main program to reject Plugins with unsupported versions, or modify its behavior according to the version (for example, a newly added hook will not be available for older Plugins and the main program will not enable it for them).
- Plugin name: Could be used by the main program for display and debug purposes.
- A list of handlers for all the available hooks. In other words, a list of function pointers for callbacks. These callbacks will be called by the main program.
A list of all data types and macros that are used by the application and the Plugin.
A Plugin register and unregister functionf which allow the plugin to provide its callbacks to the main program.

In this example header file, there is a definition of a plugin interface. All plugins need to include this file:

#ifndef _PLUGIN_INTERFACE_H_

#define _PLUGIN_INTERFACE_H_#define RTE_MAX_PLUGIN_NAME 32

#define RTE_MIN_SUPPORTED_VERSION_MAJOR 1/* Enumeration of events in the main program */

typedef enum

{

  RTE_EVENT_KEY_PRESSED,

  RTE_EVENT_SOMETHING_HAPPENED,

  RTE_EVENT_SOMETHING_ELSE_HAPPENED,
} rte_event_e;
/* The plugin interface structure */

typedef struct rte_plugin_t

{

  /* Plugin version */

  unsigned int version_major;

  unsigned int version_minor;
  /* Plugin name */

  char plugin_name[ RTE_MAX_PLUGIN_NAME ];
  /* Hook handlers */
  /* Plugin start handler */

  int (*start)(void);
  /* Plugin end handler */

  void (*end)(void);
  /* A function that asks the plugin if it wants to handle an event */

  int (*want_to_handle_event)(rte_event_e event);
  /* A function that handles an event */

  int (*handle_event)(rte_event_e event, int value);
  /* More functions can be added here */
} rte_plugin_t;
/* Register function */

extern int plugin_register( const rte_plugin_t *rte_plugin );
/* Unregister function */

extern int plugin_unregister( void );
/* The plugin init function:
Each plugin must have the following function implemented:
int rte_plugin_init( void );
The function prototype is in comment because plugins are optional,

and we don't want to have an unresolved external in link time.

*/
#endif /* _PLUGIN_INTERFACE_H_ */

2. Open, initialize and register the Plugin

The Loader provides functions that allow you to explicitly open and load a shared library. These functions are referred as the dl family. The following actions are required in order to open, initialize and register the plugin:

Call the dlopen function with the plugin name and the required load flags. The recommended load flags are RTLD_GLOBAL which makes the Plugin symbols globally available to the application and RTLD_NOW, which loads all the Plugin symbols immediately. The latter will yield better performance when working with large Plugins. The opposite option which is not recommended, RTLD_LAZY may save some memory when loading a Plugin, but it will require more time to load each Plugin symbol upon request.
Resolve the “well defined Plugin init function” as defined previously using the dlsym function.
Call this init function. The init function needs to register the Plugin, and it will be described later.

3. Start the Plugin

Once the Plugin was loaded and registered, call its start function. The plugin may initialize its data structures, allocated resources, display a message and do all the necessary actions it requires in order to start its work.

4. Changes required in the application Makefile

In order to add Plugin support to your application, you must link with the loader library (-ldl) and use the -rdynamic flag which instructs the linker to add all symbols, not only used ones, to the dynamic symbol table. This is required because there could be functions in your application which are dedicated to Plugins only, and not used by the application itself. Without this flag enabled, these functions will be unresolved when you’ll try to load the Plugin.

5. Install hooks

Modify your application to use the Plugin callbacks in the designated locations a Plugin can differentiate or improve. I leave it to you to decide where to add these in your application.

Here’s a sample code of a program that reads characters from the console. In normal behevior it display the characters on the screen. The program accepts a single parameter which is the name of the plugin you want to use. In case no plugin is required, don’t specify anything. Note that the parameter checking and parsing is simple and assumes no errors for the sake of simplicity.

#include 

#include 

#include 

#include 

#include "plugin-interface.h"/* The plugin interface data type */

typedef void rte_plugin;/* Holds the current active plugin */

static rte_plugin* plugin_ptr = NULL;
/* The plugin interface */

static rte_plugin_t plugin_interface;
/* Plugin init function of the main program */

static int plugin_init_main( char *name )

{

  int (*plugin_init)(void);
  /* Load the plugin */

  plugin_ptr = dlopen( name, RTLD_GLOBAL | RTLD_NOW );

  if( plugin_ptr == NULL )

  {

    fprintf( stderr, "Failed to load plugin %s, Error: %s.\n", name, dlerror());

    return -1;

  }
  /* Initialize the plugin */

  plugin_init = dlsym (plugin_ptr, "rte_plugin_init");

  if( plugin_init == NULL )

  {

    fprintf( stderr, "Failed to initialize plugin %s, Error: %s.\n", name, dlerror());

    dlclose(plugin_ptr);

    return -1;

  }
  /* Initialize the plugin */

  if( plugin_init() < 0 )

  {

    fprintf( stderr, "Plugin %s failed to initialize, Error: %s.\n", name, dlerror());

    dlclose(plugin_ptr);

    return -1;

  }
  return 0;

}
/* Plugin register function, called by the plugins */

int plugin_register( const rte_plugin_t *rte_plugin )

{

  if( !plugin_ptr )

  {

    return 0;

  }

  printf( "Registering plugin: %s\n", rte_plugin->plugin_name );

  if( rte_plugin->version_major < RTE_MIN_SUPPORTED_VERSION_MAJOR )

  {

    fprintf( stderr, "Plugin version (%d.%d) is unsupported\n",

    rte_plugin->version_major, rte_plugin->version_minor );

    return -1;

  }
  /* Copy the plugin information locally */

  memcpy( &plugin_interface, rte_plugin, sizeof( rte_plugin_t ) );

  return 0;

}
/* Plugin unregister function, called by the plugins */

int plugin_unregister( void )

{

  /* Clear plugin info */

  memset( &plugin_interface, 0, sizeof( rte_plugin_t ) );

  plugin_ptr = NULL;

  return 0;

}
int main( int argc, char *argv[] )

{

  /* Check if caller specified a plugin */

  if( argc>=2 )

  {

    /* Initialize the plugin with the first parameter */

    if( plugin_init_main( argv[1] ) < 0 )

    {

      fprintf( stderr, "Failed to initialize plugin %s\n", argv[1] );

      return 1;

    }

  }
  /* Main loop */

  while( 1 )

  {

    char ch = getchar();
    /* Check if a plugin was loaded */

    if( plugin_ptr )

    {

      /* Check if the plugin want to handle the KEY PRESSED event */

      if( plugin_interface.want_to_handle_event

          && plugin_interface.want_to_handle_event( RTE_EVENT_KEY_PRESSED ) )

      {

        /* Let the plugin handle it*/

        plugin_interface.handle_event( RTE_EVENT_KEY_PRESSED, ch );

      }

    }

    else

    {

      /* Perform "normal behavior" */

      if( ch == '\n' )

      {

        printf( "Good bye\n" );
        if( plugin_ptr )

        {

          /* Close the plugin */

          dlclose(plugin_ptr);

        }

        return 0;

      }

      printf( "\tNormal program behavior, key %c pressed\n", ch );

    }

  }
  return 0;

}

Now your application supports Plugins. Let’s see now how to write a compatible Plugin.

Writing a Plugin

The Plugin code is in “the other side”. It should adhere to the interface dictated by the main program and follow its rules. Here are the steps for writing a Plugin:

1. Examine the interface

Include the interface header file in your plugin source code and examine it. Decide which call backs you want to implement and define their prototypes.

2. Implement the hook handlers (callbacks)

For each callback in the interface, implement a function, in the specified prototypes. Some callbacks may be unsupported in your plugin. In this case, you need to find out if the main program allows leaving some callbacks without implementation. If it doesn’t, just create an empty function that does nothing and returns immediately.

3. Implement the init function and fill the interface structure

This is an important step. When the main program loads your plugin, it looks up this function explicitly in order to init the plugin. Note that the name of the function is important and you must provide the function with the designated name only. Once you’ve implemented the handlers, create an interface structure and fill it with your callbacks. In case there is a callback you don’t support, don’t forget to initialize its field with NULL. Note that the application may require that no callbacks will be NULLs. In this case, just write an empty function as described above. Call the plugin register function the main program provides. Once it returns successfully, then your Plugin is loaded and active, and its handler will be called.

Here’s an example of a plugin that extends this program:

#include 

#include 

#include "plugin-interface.h"/* Plugin function prototypes */

static int rte_plugin_start( void );

static void rte_plugin_end( void );

static int rte_plugin_want_to_handle_event( rte_event_e event );

static int rte_plugin_handle_event( rte_event_e event, int value );/* Fill the interface structure */

static const rte_plugin_t rte_plugin =

{

  2, /* Version major */

  5, /* Version minor */

  "rte_plugin", /* Name */

  rte_plugin_start,

  rte_plugin_end,

  rte_plugin_want_to_handle_event,

  rte_plugin_handle_event

};
int rte_plugin_start( void )

{

  printf( "Starting the %s plugin\n", rte_plugin.plugin_name );

  return 0;

}
void rte_plugin_end( void )

{

  printf( "Ending the %s plugin\n", rte_plugin.plugin_name );

}
int rte_plugin_want_to_handle_event( rte_event_e event )

{

  /* We want to handle only key pressed events */

  if( event == RTE_EVENT_KEY_PRESSED )

  {

    return 1;

  }

  return 0;

}



int rte_plugin_handle_event( rte_event_e event, int value )

{

  /* Handle the event */

  if( event == RTE_EVENT_KEY_PRESSED )

  {

    printf( "Plugin event handler, character value = %d\n", value );

  }
  return 0;

}
/* Init function, must be implemented */

int rte_plugin_init( void )

{

  /* Call the register function in the main program */

  plugin_register( &rte_plugin );

  return 0;

}

Compiling and running

Now let’s compile the main program and the plugin:

# gcc rte_main.c -ldl -o rte_main -rdynamic -Wall

# gcc rte_plugin.c -o rte_plugin.so -shared -fPIC -Wall

The main program filename is rte_main and the plugin file name is rte_plugin.so.

Now, let’s run the main program without the plugin:

 # ./rte_main

abcdef

        Normal program behavior, key a pressed

        Normal program behavior, key b pressed

        Normal program behavior, key c pressed

        Normal program behavior, key d pressed

        Normal program behavior, key e pressed

        Normal program behavior, key f pressed

Good bye

Now, let’s run the main program with the plugin:

 # ./rte_main ./rte_plugin.so

Registering plugin: rte_plugin

abcdef

Plugin event handler, character value = 97

Plugin event handler, character value = 98

Plugin event handler, character value = 99

Plugin event handler, character value = 100

Plugin event handler, character value = 101

Plugin event handler, character value = 102

Plugin event handler, character value = 10

We can see that we were able to change the behavior of our program to a customized behavior, without the need to rebuild it and without the sources (except for the interface header file).

Resources:
http://linux.die.net/man/3/dlopen

Resolving memory leaks in user-space applications

Hai Shalom — Thu, 29 Apr 2010 08:27:07 +0000

Memory leaks is one of the RT Embedded major bugs, as I described in the article about the 5 most destructive bugs. Memory leak bugs usually kill your system slowly and painfully. Depending on the total amount of RAM and the extent of the memory leak, the system will continue to run and perform well for a certain amount of time. It could be hours, days, or weeks. During this time, the amount of free memory is constantly decreased. In Linux based systems, when memory is required and there is no free memory, the kernel will start paging out programs and clearing the page cache. Further memory requests will cause performance impact due to paging out living tasks. Eventually, the kernel will trigger the Out-Of-Memory killer and start killing processes. In this stage, the system may be unusable already. The last stage will cause the kernel to panic and hang/reboot. Many consumer electronics products in the field remain active their whole life, therefore you can’t allow any memory leaks to happen, or else the customer will have to power cycle the unit from time to time (this is annoying, isn’t it?). Memory leaks are also hard to notice during unit-tests or lab tests because in these scenarios, the unit is rebooted constantly, as part of the development or testing work.

This article covers a few techniques that will help you isolate and resolve the memory leak in your user-space application. An article about kernel memory leaks will be provided as well.

Memory leaks occur due to continuous allocation of resources without freeing them, thus reducing the amount of free memory.

First step: Do we have a memory leak?

As I described earlier, it is hard to detect memory leaks during development due to the frequent system reboots in the development process. Furthermore, in case the leak is minor it could take plenty of time until the system’s performance starts to degrade to a noticeable state. Therefore, the first action to take during development, is the code review and flow review. Make sure all memory allocations are coupled with memory free. Cover all if-else and other conditional flows to make sure that the coupling still remains. In case the system starts to slow down after it was running for a while, and/or you see strange messages which contain the phrase “oom_killer”, then a memory leak probably exists somewhere. Also, during your tests, you may want to check the amount of free memory in the system periodically (read here if you don’t know how) and check the log if the amount reduces over time. If it reduces over time, then a memory leak exists.

Let’s see the following code example of a program that leaks memory on purpose:

#include 
#include 
#include

#define BLOCK_SIZE      ( 128 * 1024 )

int rte_steal_mem( void )
{
    char *p;

    /* Allocate memory */
    p = malloc( BLOCK_SIZE );

    if( p )
    {
        /* Clear it - force real allocation */
        memset( p, 0, BLOCK_SIZE );

        return 0;
    }
    return -1;
}

int main( int argc, char *argv[] )
{
    while( 1 )
    {
        /* Call this function */
        if( rte_steal_mem() != 0 )
        {
            return 1;
        }
        /* Wait 2 seconds */
        sleep(2);
    }
    return 0;
}

This program will allocate additional 128KB of memory each 2 seconds. Now, lets run it in the background and use free utility to periodically check the amount of free memory, in 10 second intervals:

# memleak &

# (while true; do free; sleep 10; done)&

total         used         free       shared      buffers

Mem:       118820         9500       109320            0          436

Swap:            0            0            0

Total:       118820         9500       109320

total         used         free       shared      buffers

Mem:       118820        10120       108700            0          436

Swap:            0            0            0

Total:       118820        10120       108700

total         used         free       shared      buffers

Mem:       118820        10760       108060            0          436

Swap:            0            0            0

Total:       118820        10760       108060

total         used         free       shared      buffers

Mem:       118820        11400       107420            0          436

Swap:            0            0            0

Total:       118820        11400       107420

total         used         free       shared      buffers

Mem:       118820        12044       106776            0          436

Swap:            0            0            0

Total:       118820        12044       106776

total         used         free       shared      buffers

Mem:       118820        12684       106136            0          436

Swap:            0            0            0

Total:       118820        12684       106136

As we can see in the “free” column, the amount of free memory is reduced in each iteration (marked in red) and the “used” column shows an increasing amount of memory. This is the expected scenario in case there is a memory leak. Again, the extent of the leak depends on the bug itself and its occurrence intervals.

Second step: Isolate the leaking process

The TOP application provides a per-process breakdown of memory allocations. Let’s take a look on the following snapshots, where the first two were taken immediately after the memleak process was activated, and the third one was taken a few minutes later:

##### First top iteration #####

Mem: 9680K used, 109140K free, 0K shrd, 436K buff, 848K cached CPU: 0% usr 0% sys 0% nic 100% idle 0% io 0% irq 0% sirq Load average: 0.00 0.00 0.00 1/36 263 PID PPID USER STAT VSZ %MEM %CPU COMMAND 171 1 root S 572 0% 0% /bin/sh 1 0 root S 564 0% 0% init 263 171 root R 564 0% 0% top 262 171 root S 552 0% 0% memleak 170 1 root S 420 0% 0% watchdog_rt -t 15 /dev/watchdog 101 13 root SW< 0 0% 0% [ti_spi.0] 12 1 root SW< 0 0% 0% [khelper] 144 13 root SW< 0 0% 0% [IRQ 8] 21 13 root SW< 0 0% 0% [khubd] 163 1 root SWN 0 0% 0% [jffs2_gcd_mtd8] 88 1 root SW 0 0% 0% [mtdblockd] 7 1 root SW 0 0% 0% [softirq-block/0] 11 1 root SW< 0 0% 0% [events/0] 5 1 root SW 0 0% 0% [softirq-net-tx/] 2 1 root SW 0 0% 0% [posix_cpu_timer] 3 1 root SW 0 0% 0% [softirq-high/0] 4 1 root SW 0 0% 0% [softirq-timer/0] 6 1 root SW 0 0% 0% [softirq-net-rx/] 59 13 root SW< 0 0% 0% [aio/0] 84 13 root SW< 0 0% 0% [IRQ 41]

##### Second top iteration #####

Mem: 10068K used, 108752K free, 0K shrd, 436K buff, 848K cached

CPU:   0% usr   0% sys   0% nic  99% idle   0% io   0% irq   0% sirq

Load average: 0.00 0.00 0.00 1/36 263

PID  PPID USER     STAT   VSZ %MEM %CPU COMMAND

263   171 root     R      564   0%   0% top

  262   171 root     S      936   1%   0% memleak

  171     1 root     S      572   0%   0% /bin/sh

1     0 root     S      564   0%   0% init

170     1 root     S      420   0%   0% watchdog_rt -t 15 /dev/watchdog

101    13 root     SW<      0   0%   0% [ti_spi.0]

12     1 root     SW<      0   0%   0% [khelper]

144    13 root     SW<      0   0%   0% [IRQ 8]

21    13 root     SW<      0   0%   0% [khubd]

163     1 root     SWN      0   0%   0% [jffs2_gcd_mtd8]

88     1 root     SW       0   0%   0% [mtdblockd]

7     1 root     SW       0   0%   0% [softirq-block/0]

11     1 root     SW<      0   0%   0% [events/0]

5     1 root     SW       0   0%   0% [softirq-net-tx/]

2     1 root     SW       0   0%   0% [posix_cpu_timer]

3     1 root     SW       0   0%   0% [softirq-high/0]

4     1 root     SW       0   0%   0% [softirq-timer/0]

6     1 root     SW       0   0%   0% [softirq-net-rx/]

59    13 root     SW<      0   0%   0% [aio/0]

84    13 root     SW<      0   0%   0% [IRQ 41]

##### top output after a few minutes #####Mem: 65212K used, 53608K free, 0K shrd, 436K buff, 848K cached

CPU:   0% usr   0% sys   0% nic  99% idle   0% io   0% irq   0% sirq

Load average: 0.00 0.00 0.00 1/36 263

PID  PPID USER     STAT   VSZ %MEM %CPU COMMAND

263   171 root     R      564   0%   0% top

144    13 root     SW<      0   0%   0% [IRQ 8]

262   171 root     S    55976  47%   0% memleak

171     1 root     S      572   0%   0% /bin/sh

1     0 root     S      564   0%   0% init

170     1 root     S      420   0%   0% watchdog_rt -t 15 /dev/watchdog

101    13 root     SW<      0   0%   0% [ti_spi.0]

12     1 root     SW<      0   0%   0% [khelper]

21    13 root     SW<      0   0%   0% [khubd]

163     1 root     SWN      0   0%   0% [jffs2_gcd_mtd8]

88     1 root     SW       0   0%   0% [mtdblockd]

7     1 root     SW       0   0%   0% [softirq-block/0]

11     1 root     SW<      0   0%   0% [events/0]

5     1 root     SW       0   0%   0% [softirq-net-tx/]

2     1 root     SW       0   0%   0% [posix_cpu_timer]

3     1 root     SW       0   0%   0% [softirq-high/0]

4     1 root     SW       0   0%   0% [softirq-timer/0]

6     1 root     SW       0   0%   0% [softirq-net-rx/]

59    13 root     SW<      0   0%   0% [aio/0]

84    13 root     SW<      0   0%   0% [IRQ 41]

Take a look at the VSZ and %MEM columns which show the amount of memory allocated and the percentage from the system memory. In the first iteration, the memleak process looks normal, like all the other processes. In the second iteration (marked in orange), it starts to be noticable, and in the third output (marked in red), we can see that the memleak process has allocated about 57MB which is 47% of the system’s memory!

Now we know which process is leaking. In case no process shows growing numbers, then the leak could be in the kernel code. Click here in order to find out how to detect and fix these.

Third step: Open the code, find and fix the leak

The traditional approach – Code reviews

As I mentioned earlier, the first approach is to try to avoid memory leaks by performing code reviews and flow covers. However, sometimes, a post release code review (once we’ve isolated the process) can reveal the leaking location inside the process. Here are some points to consider:

Make sure all memory allocations are coupled with memory free.
Make sure all other resource allocations are coupled with resource release (such as fopen->fclose, open->close, etc.).
Cover all if-else, switch-case and other conditional flows, including erroneous flows, to make sure that the coupling still remains.
There are cases where a function allocates memory for the caller. Make sure that you free the memory once it is not needed anymore.
Make sure that a pointer that holds an address to a dynamically allocated memory is not erased or overrun by another value. If this will happen, you will not be able to free this resource.

If all of that does not help, you can consider using code inspection tools, or the follow open source library.

Replacing the malloc function with a debug version

We can try to apply a nice trick to figure out the caller, by replacing the standard malloc function with a macro. The macro will undefine the original malloc address and will show some useful information about the caller, and how much memory it required. It is most useful to put this macro in a common header file, or in a specific C file that you suspect. Note that the beauty of this macro is in the fact that it will not harm the program, every memory allocation will still be served correctly. However, it will slow down the program due to the prints. The macro that we use is highlighted in the following example code:

#include 
#include 
#include 

#define __malloc malloc
#undef malloc
#define malloc( _size ) \ ({ printf( "%s: Calling malloc(%d) from line %d, filename: %s\n", __func__, _size, __LINE__, __FILE__ ); \   __malloc( _size ); })

int main( int argc, char *argv[] )
{
    void *p;

    p = malloc( 1024 );
    memset( p, 0, 1024 );
    printf( "p=%p\n", p );
    free(p);

    return 0;
}

And here is the output of this file:

$ ./mem
main: Calling malloc(1024) from line 13, filename: mem.c
p=0x9c06008

Any doubt who was the caller…? Note that same macro technique can be applied on free or any other allocation function. If a specific caller is constantly allocating memory (for example; in a loop), it will be very easy to catch it here. However, if the leak is minimal and we need more analysis to apply, the macro can be extended to also print the address of the allocated memory, and so will the free macro replacement. Using an offline processing tool (such as Microsoft Excel), we can then analyze the output of the macros and compare allocations against releases of the same address. Then we could track who was the caller of an allocation which wasn’t freed. If it sounds too complicated, then the next section describes a tool that may automate some of the work.

Using dmalloc to debug your application

The dmalloc is a dynamic memory allocation analysis tool that replaces the traditional malloc (including all its derivatives) and free functions with debug enhanced versions, which will allow you to detect:

Memory leaks.
Memory overruns.
Mutiple calls for free on the same pointer.
Freeing a NULL pointer.
Using a freed resource.

The tool can be configured in run time, and also provides statistics of memory allocation during the program’s life and after it exits. The provided log is detailed and contains file names and line numbers.

Once you’ve downloaded and compiled the dmalloc package, you’ll see that it created an executable and a library. The executable is used to configure the dmalloc in runtime and the library contains the debug enhanced versions of the allocation/release functions.

In order to facilitate the dmalloc library in your program, you need to do some minor modifications to its source files and makefiles. In each source file that you are interested to enable the dmalloc debugging, you need to include the library’s header file. This include directive can be ifdef’ed to be enabled only when you enable debug mode. For example:

#ifdef ENABLE_DMALLOC_DEBUG

#include "dmalloc/dmalloc.h"

#endif

And in the makefile, you need to link with the dmalloc library as in this example:

ifdef ENABLE_DMALLOC_DEBUG

LDFLAGS += -ldmalloc

endif

Note that multithreaded applications require to link with the thread aware version of the library. In this example you need to compile your application with the ENABLE_DMALLOC_DEBUG flag enabled in order to enable the dmalloc infrastructure for debug purposes.

Once you’ve compiled and linked your application with dmalloc enabled, boot the system and configure the tool with your requested settings. When your system finished booting, configure the dmalloc with your required options, such as shell type, monitor intensity, poll intervals, log file and other advanced options according to your requirements. Run the dmalloc application with the –usage flag to see other flags. If you are using an embedded system with BusyBox, use the -b flag, and then copy-and-paste the output text that the dmalloc prints. This procedure is required in order to setup an environment variable which will be visible to the program you want to debug. Here’s an example:

# dmalloc -b high -i 1 -l /dev/console

DMALLOC_OPTIONS=debug=0x4f4ed03,inter=1,log=/dev/console

export DMALLOC_OPTIONS

In this example, the dmalloc is initialized to print an init string for borne shell types, high monitoring in 1 seconds intervals and display the log in the cosole (this could be replaced by any file name). Don’t forget you need to copy-and-paste the output to make this configuration effective. Now you are ready to run your program. As an example, let’s modify our memleak program to do 5 iterations and then exit. Let’s see what statistics are displayed when the program exits:

# memleak

29: 5: Dmalloc version '5.5.2' from 'http://dmalloc.com/'

29: 5: flags = 0x4f4ed03, logfile '/dev/console'

29: 5: interval = 1, addr = 0, seen # = 0, limit = 0

29: 5: starting time = 19

29: 5: process pid = 207

29: 5: Dumping Chunk Statistics:

29: 5: basic-block 4096 bytes, alignment 8 bytes

29: 5: heap address range: 0x4007000 to 0x4194000, 1626112 bytes

29: 5:     user blocks: 165 blocks, 655360 bytes (94%)

29: 5:    admin blocks: 4 blocks, 16384 bytes (2%)

29: 5:    total blocks: 169 blocks, 692224 bytes

29: 5: heap checked 6

29: 5: alloc calls: malloc 5, calloc 0, realloc 0, free 0

29: 5: alloc calls: recalloc 0, memalign 0, valloc 0

29: 5: alloc calls: new 0, delete 0

29: 5:   current memory in use: 655360 bytes (5 pnts)

29: 5:  total memory allocated: 655360 bytes (5 pnts)

29: 5:  max in use at one time: 655360 bytes (5 pnts)

29: 5: max alloced with 1 call: 131072 bytes

29: 5: max unused memory space: 20480 bytes (3%)

29: 5: top 10 allocations:

29: 5:  total-size  count in-use-size  count  source

29: 5:      655360      5      655360      5  /home/hai/rte/memleak.c:13

29: 5:      655360      5      655360      5  Total of 1

29: 5: Dumping Not-Freed Pointers Changed Since Start:

29: 5:  not freed: '0x40ef008|s1' (131072 bytes) from '/home/hai/rte/memleak.c:13'

29: 5:  not freed: '0x4110008|s1' (131072 bytes) from '/home/hai/rte/memleak.c:13'

29: 5:  not freed: '0x4131008|s1' (131072 bytes) from '/home/hai/rte/memleak.c:13'

29: 5:  not freed: '0x4152008|s1' (131072 bytes) from '/home/hai/rte/memleak.c:13'

29: 5:  not freed: '0x4173008|s1' (131072 bytes) from '/home/hai/rte/memleak.c:13'

29: 5:  total-size  count  source

29: 5:      655360      5  /home/hai/rte/memleak.c:13

29: 5:      655360      5  Total of 1

29: 5: ending time = 29, elapsed since start = 0:00:10

Let’s see the useful debug information which the dmalloc provided:

General information
Heap statistics
Total calls for each allocation or free functions
Allocation statistics
Top 10 allocations
And a list of non-freed pointers including file name and line number (This is great for us!).

With this information, we can isolate and fix the memory leak immediately. In case your program does not exist (in case of a daemon for example), you can show these statistics by calling the statistics function dmalloc_log_stats( ), which is actually the same function that is automatically called upon process termination. Your program can call this function periodically or by external request (such as an IPC action).

Although it is not relevant directly to memory leaks, lets also see what happens when the program overflows a buffer. Let’s change the memset call in out memleak program to write BLOCK_SIZE+1 bytes (1 extra byte). Here’s the output:

66: 2: Dmalloc version '5.5.2' from 'http://dmalloc.com/'

66: 2: flags = 0x4f4ed03, logfile '/dev/console'

66: 2: interval = 1, addr = 0, seen # = 0, limit = 0

66: 2: starting time = 64

66: 2: process pid = 209

66: 2:   error details: checking user pointer

66: 2:   pointer '0x40ef008' from 'unknown' prev access '/home/hai/rte/memleak.c:13'

66: 2:   dump of proper fence-top bytes: '\372\312\336i'

66: 2:   dump of '0x40ef008'+131056: '\000\000\000\000\000\000\000\000\000\000\000\312\336i'

66: 2: ERROR: _dmalloc_chunk_heap_check: failed OVER picket-fence magic-number check (err 27)

debug-malloc library: dumping program, fatal error

Error: failed OVER picket-fence magic-number check (err 27)

Aborted

As we can see, the dmalloc caught the overflow and terminated the program. We can see when the buffer was allocated and the data that was overflowing.

You can download the dmalloc source code from here, and consult its detailed documentation here.

Forth step: Validate your fix

Congratulations, you found and fixed the memory leak. The last step is to repeat step 1 and make sure the system does not show memory degradation over time, and your’e done. Resources:
http://dmalloc.com/
http://linux.die.net/man/1/top

Measuring the CPU load

Hai Shalom — Sat, 24 Apr 2010 11:13:39 +0000

The CPU in embedded systems is an important resource. The CPU type, model and processing power is carefully selected during the project’s first steps. A system that its CPU is busy most of the time is not healthy, and a system that its CPU is idle most of the time is wasteful. Assuming we’re past the design stage and the system is running, you and your managers will probably want to know how busy is the CPU while performing various tasks. Operations that require extensive CPU processings include cryptographic calculations, handling and routing network traffic, graphic calculations and many other product specific calculations.

This article covers 3 different utilities which will help you measure the CPU load in Linux environment. Their output defers from each other by the levels of report details.

The VMSTAT utility

The vmstat utility reports summarized information about processes, memory, paging, block I/O, traps, and CPU load. It is distributed as part of the procps package which contains libraries and other utilities that provide information based on the information provided in the proc filesystem. The following window provides an example screenshot of the vmstat utility, which was configured to display 12 readings in 1 second intervals. Each line provides a snapshot of the system in the current sampling time (use vmstat 1 12):

# vmstat 1 12

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------

 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st

 3  0      0 257232  14876 309072    0    0    83     9   65   98  1  2 96  1  0

 0  0      0 257000  14876 309072    0    0     0     0   78  106  2  2 96  0  0

 0  0      0 257124  14884 309072    0    0     0    32   45   62  1  2 92  5  0

 0  0      0 257232  14884 309072    0    0     0     0   74  107  1  2 97  0  0

 0  0      0 257000  14884 309072    0    0     0     0   43   49  2  1 97  0  0

 0  0      0 257124  14884 309072    0    0     0    16   86  124  1  2 97  0  0

 0  0      0 257124  14884 309072    0    0     0     0  151  214  2  3 95  0  0

 0  0      0 256984  14884 309072    0    0     0     0  156  234  2  4 94  0  0

 0  0      0 257108  14884 309072    0    0     0     0  279  420  0  5 95  0  0

 0  0      0 257000  14884 309084    0    0    12     0  201  285  3  5 92  0  0

 0  0      0 257000  14884 309084    0    0     0    12  146  193  2  2 96  0  0

 0  0      0 257000  14884 309084    0    0     0     0   63  110  0  3 97  0  0

The following fields are displayed in the vmstat output:

Procs

r: The number of processes waiting for run time.

b: The number of processes in uninterruptible sleep.

Memory

swpd: the amount of virtual memory used.

free: the amount of idle memory.

buff: the amount of memory used as buffers.

cache: the amount of memory used as cache.

Swap

si: Amount of memory swapped in from disk (or disks).

so: Amount of memory swapped to disk (or disks).

bi: Blocks received from a block device (blocks/s).

bo: Blocks sent to a block device (blocks/s).

System

in: The number of interrupts per second, including the clock.

cs: The number of context switches per second.

CPU

us: %CPU spent by User space applications.

sy: %CPU spent by the System (kernel mode).

id: %CPU spent idle.

wa: %CPU spent waiting for I/O.

st: %CPU stolen from a virtual machine.

The procps package source code can be downloaded from here.

The TOP utility

The TOP utility is another standard utility. It is distributed as part of the procps package and a lighter implementation is provided in Busybox. It provides a summary about the CPU and memory usage in the system, and it also provides detailed information about each task in the system. This information includes the process ID, the parent’s process ID, owner, state and memory consumption. It also shows the CPU usage breakdown per each process in the column before the process name. This detailed output is mainly used for debugging purposes, to detect tasks that consume too much CPU time. The output refresh rate (in seconds) and number of iteration can be configure in the command line. Here’s a snapshot of Busybox’s top output in an idle machine, which was configured to to display 10 readings in 1 second intervals. The top output refreshes the whole screen in each iteration (use top 2 10):

# top 2 10

Mem: 9500K used, 44320K free, 0K shrd, 412K buff, 832K cached

CPU:   0% usr   0% sys   0% nic 100% idle   0% io   0% irq   0% sirq

Load average: 0.00 0.00 0.00 1/32 184

  PID  PPID USER     STAT   VSZ %MEM %CPU COMMAND

  171     1 root     S      568   1%   0% /bin/sh

    1     0 root     S      564   1%   0% init

  184   171 root     R      564   1%   0% top

  170     1 root     S      420   1%   0% watchdog -t 15 /dev/watchdog

  144    13 root     SW<      0   0%   0% [IRQ 8]

  101    13 root     SW<      0   0%   0% [spi.0]

   88     1 root     SW       0   0%   0% [mtdblockd]

   12     1 root     SW<      0   0%   0% [khelper]

    7     1 root     SW       0   0%   0% [softirq-block/0]

   10     1 root     SW<      0   0%   0% [desched/0]

   11     1 root     SW<      0   0%   0% [events/0]

    4     1 root     SW       0   0%   0% [softirq-timer/0]

    5     1 root     SW       0   0%   0% [softirq-net-tx/]

   17    13 root     SW<      0   0%   0% [kblockd/0]

    2     1 root     SW       0   0%   0% [posix_cpu_timer]

    3     1 root     SW       0   0%   0% [softirq-high/0]

   57    13 root     SW       0   0%   0% [pdflush]

   58    13 root     SW<      0   0%   0% [kswapd0]

    6     1 root     SW       0   0%   0% [softirq-net-rx/]

   84    13 root     SW<      0   0%   0% [IRQ 41]

The CPU summay line provides the follwoing information:

Field	Description
usr	%CPU spent by User space applications
sys	%CPU spent by the System (kernel mode)
nic	%CPU spent by Low priority user mode (nice)
Idle	%CPU which is available (idle)
io	%CPU spent by I/O waiting
irq	%CPU spent servicing interrupt requests
sirq	%CPU spent servicing soft irqs

Please note that this output may vary between different top versions or implementations. In Busybox implementation, the CPU measurement feature is disabled by default. In order to enable it, you must change the Busybox configuration to enable this feature (Process Utilities->top->Show CPU per-process usage percentage and Show CPU global usage percentage).

The IOSTAT utility

The iostat utility is used for monitoring system input/output device and can help you estimate the performance of block devices, but it also calculates the CPU load. It can be configured to display the CPU load periodically. Its output is a summary of the CPU load in the sampling time. Here’s the output of iostat, when it was configured to show the CPU usage in 1 second intervals (use iostat c 1):

# iostat c 1

 %user  %system %iowait %idle

  0       0       0     100

  0       50      0     50

  2       50      0     48

  4       50      0     46

  48      10      0     42

  12      0       0     88

  7       0       93    0

  7       6       87    0

Each line provides the follwoing information:

Field	Description
user	%CPU spent by User space applications
system	%CPU spent by System (kernel mode)
io	%CPU spent by I/O waiting
idle	%CPU which is available (idle)

The iostat source code can be downloaded here.

How to interpret the numbers

In high level, all 3 tools provide a summary of the current CPU usage. The four imporant fields are; user, system, i/o and idle, where the sum of all four fields must be 100%. If you are interested to see the CPU load in general, you may run iostat for a summarized information. If you want more details, you can run vmstat. If you need a per-process breakdown, use top.

A system which shows high percentage in the user field, suggests that the CPU is busy mainly with one or more processes in the user space. Usually, a process should not consume too much CPU during its run time due to the scheduling policy. In case a process consumes constant high CPU processing power, it may suggest that it is performing either a heavy calculation taks or that something is wrong. If you know that the specific process should consume a lot of CPU time, then it is OK. Otherwise, you need to check the implementation of the process. It might be stuck in busy-wait state, it might be over prioritized (read here for more details), or perhaps its code implementation is not optimized (for example; performing many iterations of division/modulus calls, performing unneeded calculations, calling too many system calls, etc.).

A system which shows high percentage in the system field, suggests that the CPU is busy mainly running kernel code and drivers. This is typically seen in high bandwidth networking drivers, where the system processes high networking traffic, or high interrupts load. The Linux kernel provides means to reduce the networking overhead with the NAPI networking API. Interrupt handling overhead can be reduced, if the hardware supports it, by interrupt pacing, which allows activation of the servicing function once per a number of interrupts, thus doing all the work once.

A system which shows high percentage in the i/o field, suggests that the CPU is waiting for data to be read or written from or to a storage device (disk, flash, cd, etc). Note that it doesn’t mean that the CPU is busy; it could perform other tasks while the i/o operation completes. However, it might suggest that you need to tune your caches and file systems.

A system which shows high percentage in the idle field, suggests that the CPU is idle most of the time (just sitting and waiting). It is normal in idle state where the system is waiting (for example in the shell, or login screen). However, if this is the situation even in the highest work load possible in the product, then this CPU is too strong for the product, and it is a waste of money and power. You should consider adding additional work for this CPU to do, or put the CPU in low power mode while reducing its clock rate.

Please note that the scenarios which were described are just examples. There could be other reasons which affect your system behavior in a different manner.

Resources:
http://procps.sourceforge.net/
http://linux.die.net/man/8/vmstat
http://linux.die.net/man/1/iostat
http://www.busybox.net/

Embedded software 5 most destructive bugs

Hai Shalom — Wed, 21 Apr 2010 13:30:08 +0000

We all make mistakes, and it is only natural. However, if the product is already running in the field, mistakes we do in the code which are not detected on time, can cause:

Damage to customers,
Massive recalls,
Massive Firmware update requirement,
Failure in field tests,
Loss of brand credibility,
Lot’s of wasted $$$,
Travel to customer’s/OEM site,
You loosing your position, or job ?

The error types I am going to present are generic errors, which are not tied to specific architectures. A bug which belongs to these error types may be hidden, hard to find and hard to reproduce. The system may fail randomly or unexpectedly (Referred as the “Voodoo effect”).

We can never guarantee that the software we provide is error free. What can we do in order to minimize the chance for these errors to pop up? Well, just continue to read.

1. Buffer overflow

Buffer overflow is a large family of errors, and could be the result of various bugs:

Static buffer overflow: Writing, reading or copying data to or from a buffer in a larger amount than its maximum size. A buffer could be a string buffer, a structured variable, an array or a native type variable, either global or on the stack.
Dynamic buffer overflow: Similar to 1, with the difference of the buffer being allocated dynamically.
Array overflow: Misunderstanding or confusion about the last array index. An array in the size of X has the indexes of 0 to X-1.

How to avoid and detect:

When you write new code, pay attention to the size of each variable, or buffer you need to access.
Pay close attention to the memory read/write functions such as memset, memcpy, memmove and similar. Make sure that the destination buffer is large enough to accept the worst case scenario in terms of data size (maximum name length, maximum amount of bytes, largest data type).
When working with string functions such as strcpy, strcat and similar, make sure that the buffer is terminated with a NULL character (‘\0′), otherwise, the string function will continue to process your buffer until a NULL character will be found somewhere in the memory.
Similarly to 2 and 3, when using string format functions like sprintf and similar functions.
When working with strings, try to use the bounded versions of the string functions like: strncpy, strncat, snprintf. These functions accept another parameter in which you can specify your buffer length. They will limit the maximum bytes they write to the destination buffer. Note that in case these functions hit your number, they will stop their processing and will not place a NULL character. Therefore, the correct usage of these functions is; first, clear your buffer using memset, and then, let the string function process your buffer by specifying its length – 1.
When working with arrays, and the array is defined in the size of ARRAY_SIZE:
- Never access the array in the index of ARRAY_SIZE. It’s one index outside of its boundary.
- When using for loops, the end condition for the last entry must be smaller than ARRAY_SIZE and not smaller or equal.

Buffer overflow may result in Segmentation fault crash (in user space) or a kernel panic (for code running in kernel space).

2. Memory (or resource) leaks

Memory leaks usually kill your system slowly and painfully. After a fresh boot, everything looks ok and the system works fine. Depending on the total amount of RAM and the extent of the memory leak, the system will continue to run and perform well for a certain amount of time. It could be hours, days, or weeks. During this time, the amount of free memory is constantly decreased. In Linux based systems, when memory is required and there is no free memory, the kernel will start paging out programs and clearing the page cache. Further memory requests will cause performance impact due to paging out living tasks. Eventually, the kernel will trigger the Out-Of-Memory killer and start killing processes. In this stage, the system may be unusable already. The last stage will cause the kernel to panic and hang/reboot. Many consumer electronics products in the field remain active their whole life, therefore you can’t allow any memory leaks to happen, or else the customer will have to power cycle the unit from time to time. Memory leaks are also hard to notice during unit-tests or lab tests because in these scenarios, the unit is rebooted constantly, as part of the development or testing work.

Memory leaks occur due to continuous resource allocation without freeing it. A system can process numerous amount of data, but if we’ll examine it in snapshots, at any given time its resource allocation is bounded. In order to keep the system functional, the resource must be freed when it is not required anymore, or when the work on it has been done.

Memory leaks could be the results of the following:

Allocating a memory buffer using functions similar to malloc (user) or kmalloc (kernel) and not freeing it.
Opening files, sockets or pipes and not closing them.
A kernel network driver which processes a packet in an skb structure and doesn’t free it when it is finished.

How to avoid and detect:

When a resource is temporary allocated inside a function, go through all flow options and make sure the resource is freed before each return keyword. In some cases, usually error cases when the function exits prematurely, the resource free handling is forgotten. For example, assuming a resource was allocated, and then, an if statement that checks a condition may cause the function to return an error. Usually the if statement continues correctly and the resource is used and freed, but in case of an error, the function may return earlier.
Some functions may allocate memory for you and return a pointer to the allocated buffer. Make sure you free it once your processing is complete.
During long test trials, run “free” or ”top” in the background, and monitor any memory reduction (with free) or memory incrementation of a task (with top).

3. Ignoring compiler warnings or fixing them without understanding

A common bad practice is to ignore compiler warnings. Indeed, there are cases that the warnings are not important, but there are cases where warnings may indicate a real problem. A few examples:

Missing return value.
Incompatible variable assignment.
Mismatch between pointers and variables.
Using an uninitialized varialbe.

Any many other examples.

An even worse practice is to fix warnings without really understanding them, just to make the compiler quiet. A nice story about a developer who “fixed” a warning reported by a code inspection without really understanding what he did can be found here.

How to avoid:

Write a simple well organized code. Writing “clever” code is not fashionable and makes it hard for everybody else to understand and fix.
Enable the -Wall switch in the gcc. It will enable all warnings report.
Never finish a module with remaining warnings. If you won’t fix the first one as it appears, soon there will be more and more until there will be too many. If you coded this module, you know best, right now, how to fix it. Other people might not understand what was your intension, and even you may forget it after a while.
Once the module is warning-free, it is recommended to configure the gcc to treat all warnings as errors using the -Werror switch. This will keep the module warning free for good.
Never fix a warning that you have doubts about. Many warnings are straight forward, but some may require you to ask the person who wrote the module, or to consult with other people.

4. Race conditions

Race condition is a scenario when two or more contexts are using the same resource in parallel without considering each other. Race condition could be a result of:

2 or more contexts are trying to access the same data, variable or memory in the same time.
2 or more contexts are calling a non-reentrant function in the same time.

Contexts could be either kernel threads or user space threads inside a process. A single process is more protected, however, it could be exposed to the same issues in case 2 or more processes are using a shared memory region, or calling a non-reentrant function in a shared library.

Race conditions could result in a randon an unpredictable behavior of the system due to mutual data corruption (one context may corrupt the data or flow other other one), and it may be hard to detect or reproduce such bugs.

How to avoid:

Try to avoid creating and using global variables.
Protect global variables and shared memory regions with semaphores or mutexes. Each context “locks” the resource until it finishes the processing (be careful from deadlocks).
Write your functions to be reentrant. If the function accesses a shared resouce and can’t be reentrant, apply a mutex that will allow only single usage at a time. A second context calling the same function will go to sleep until it is freed.
Try to minimize the use of external non-reentrant functions (such as strtok), use reentrant alternative (such as strtok_r).

5. Alignment traps

Alignment traps are crashes that occur due to misaligned memory access on 32-bit CPUs. Such scenarios could happen when porting code to a new architecture, and when using wrong methods to access raw network packets or data storage with varied length. This error condition occurs when the CPU is requested to read a 32-bit variable from an odd address (which is used by 8-bit variables) or an address that is used by 16-bit variables. For example, suppose you receive an IPv4 packet and you want to read the first 32-bits of the IP header (contains information such as version and total length). The IP portion of the packet is concatenated after the Ethernet header which is composed of 14 bytes (6 source address, 6 destination address, 2 type). If you would try to access the IP packet directly with a 32-bit pointer, you will get an alignment trap because the IP packet data starts at offset 14, and the CPU can read a 32-bit variable from either address 12 or 16 (4 bytes alignment). Another example, suppose there is a function that retreives data from a database. The data size could be from 1 byte to 1024 bytes, and the function returns a void pointer. If you’ll cast this pointer to a 32-bit pointer (such as unsigned integer) in order to read the values, you will get an alignment trap in case the function returns a 1 byte (char) or 2 bytes (short) of data. Whilst the first example is easy to track and fix (happens always), the latter might be difficult, depending on the amount of times the function really returns a pointer to an 8-bit variable.

How to avoid and detect:

Be careful with pointer casting. If there is a chance that the pointer points to an array of bytes or shorts, or comes from a packed structure, you must not cast this pointer to a 32-bit type. Instead, copy the pointer’s value to a local variable, and do the processing on it.
Don’t ignore compiler warnings about pointer mismatch. Make sure that the right pointers are used.

Further actions to take

Understanding these errors and the ways to avoid them is the first step towards a better software. The following actions will also help you avoid critical errors and increase the system stability and robustness:

Create Design Documents. They will help you with the implementation and later with debug.
Hold code reviews. Sometimes, another pair of eyes will see things you missed.
If you have a doubt, consult with other people.
Keep the code simple and organized. “Clever” code is not appreciated.
There are code inspection utilties which you can use, some are open sourced (such as Valgrind), and some are commercial. These tools examine your code and provide a detailed report about potential bugs. Note that there are many false alarms, and some issues could be there on purpose.
And last, but not least; Read my blog :)

Linux Memory Consumption

Hai Shalom — Tue, 06 Apr 2010 05:30:29 +0000

You’ve probably heard the managers asking questions like; how much free memory does the system have? Why is the amount of free memory so low? Where did the entire RAM go?

This article will provide you the right answers that will make the managers and customers smile, and understand the Linux memory consumption concept.

The standard Linux utilities for memory consumption report are “free” and “top”. The “free” utility displays the amount of free and used RAM, and “top” utility also provides a per-process memory usage breakdown. These utilities may often report that the free amount of memory left in the system is rather lower than you would expect, especially after the system was up and running for a while. You may conclude that there are problems related to memory leaks, or that your available RAM is being used inefficiently or even that your system does not have enough RAM to run correctly. Whilst these issues could be real problems with your system, it doesn’t always have to be case.

Linux memory consumption concept

Linux memory consumption concept is all about efficiency. The system’s RAM is a resource that is meant to be used; 100% of it (if possible), all the time (if possible).

Linux utilizes unused RAM to cache data and filesystem meta-data from slower storage devices (Flash or disk) because fetching the information from the RAM is much quicker: There are no bottlenecks such as slow physical media, slow buses or device clocks, and not decompression is required.

Assuming there are no memory leaks, the reason that memory report tools report low amount of free memory is because the RAM is considered to be wasted if it isn’t used. This concept may require some time to digest, because conventional thinking may lead to the conclusion that an efficient system is a system with a lot of free memory. This is not entirely correct. In Linux case, the kernel tries to utilize the most of the RAM to improve the system performance. Keeping the cache means that if the kernel or a task needs the same data again, there’s a good chance it will still be in the fast cache in memory.

Getting the amount of free memory

As mentioned before, there are two standard utilities that report the amount of free memory. For RT Embedded projects, these utilities are provided by “Busybox”, and their output might be slightly different than standard distributions due to memory consumption restraints. Here’s an example for the output of “free” utility:

# free

              total         used         free       shared      buffers

  Mem:       119248        41524        77724            0         3392

 Swap:            0            0            0

Total:       119248        41524        77724

The numbers shown are in Kilo-Bytes, meaning that this system has total memory of about 120MB, 41MB is used and 77MB is free. The buffers column displays the amount of memory that was used for filesystem meta-data (such as the filesystem tree information, file location information, etc.). System with hard disks or flash drives may also have swap memory. This snapshot was taken from a system with a single flash without a disk, and therefore, the total swap memory is 0.

Here’s an example of the output from “top”, taken from Fedora distribution:

top - 15:58:22 up 21 min,  2 users,  load average: 0.00, 0.03, 0.15

Tasks: 127 total,   1 running, 125 sleeping,   0 stopped,   1 zombie

Cpu(s):  0.3%us, 6.9%sy, 0.0%ni, 92.5%id, 0.0%wa, 0.3%hi, 0.0%si, 0.0%st

Mem:    773720k total,   367820k used,   405900k free,    12720k buffers

Swap:  1572856k total,        0k used,  1572856k free,   154052k cached

  PID USER    PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND          

 1899 root    20   0  9980 1124  776 S  4.3  0.1   0:15.41 kerneloops       

 2706 hai     20   0  2428 1064  836 R  1.6  0.1   0:01.16 top              

 2046 root    20   0 38120  18m 6720 S  0.7  2.4   0:22.82 Xorg             

  130 root    15  -5     0    0    0 S  0.3  0.0   0:01.64 ata/0            

  178 root    20   0     0    0    0 S  0.3  0.0   0:00.90 pdflush          

 1683 root    20   0  3628 1032  916 S  0.3  0.1   0:03.05 hald-addon-stor  

 2393 hai     20   0  7508 1424 1076 S  0.3  0.2   0:03.49 VBoxClient       

 2658 hai     20   0  102m  20m  12m S  0.3  2.7   0:04.27 gnome-terminal   

    1 root    20   0  2008  772  564 S  0.0  0.1   0:03.41 init             

    2 root    15  -5     0    0    0 S  0.0  0.0   0:00.06 kthreadd         

    3 root    RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/0      

    4 root    15  -5     0    0    0 S  0.0  0.0   0:00.59 ksoftirqd/0      

    5 root    RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/0       

    6 root    15  -5     0    0    0 S  0.0  0.0   0:00.27 events/0         

    7 root    15  -5     0    0    0 S  0.0  0.0   0:00.21 khelper          

   80 root    15  -5     0    0    0 S  0.0  0.0   0:00.00 kintegrityd/0    

   82 root    15  -5     0    0    0 S  0.0  0.0   0:00.89 kblockd/0

The output shows similar memory reports fields as the “free” utility as well as per-proces detailed information. For the purpose of this article, the following columns are relevant:

VIRT: Virtual Image (KB). The total amount of virtual memory used by the task. It includes code, data and shared libraries as well as pages that have been swapped out.
RES: Resident size (KB). The non-swapped physical memory a task has used.
%MEM: Memory usage (RES). A task’s currently used share of available RAM.

The detailed output of “top” may interest you when you are debugging memory leaks. In this case, the leaky task Resident size (and %MEM) will rise over time. Otherwise, the output of “free” is sufficient.

Getting more information from the proc filesystem

The kernel provides a special proc file called meminfo, which displays the memory consumption figures in details. The meminfo file may differ between distributions, especially between Embedded Linux and Desktop Linux. Here’s a typical output of this proc file, from the same system that was used for the “free” snapshot:

# cat /proc/meminfo

MemTotal:       119248 kB

MemFree:         77544 kB

Buffers:          3392 kB

Cached:          10260 kB

SwapCached:          0 kB

Active:          19612 kB

Inactive:         7660 kB

HighTotal:           0 kB

HighFree:            0 kB

LowTotal:       119248 kB

LowFree:         77544 kB

SwapTotal:           0 kB

SwapFree:            0 kB

Dirty:               0 kB

Writeback:           0 kB

AnonPages:       13640 kB

Mapped:           5428 kB

Slab:             4540 kB

PageTables:       1616 kB

NFS_Unstable:        0 kB

Bounce:              0 kB

CommitLimit:     59624 kB

Committed_AS:    28588 kB

VmallocTotal:   131072 kB

VmallocUsed:      9296 kB

VmallocChunk:   106492 kB

The information fields which are available in meminfo proc file:

MemTotal – Total amount of physical RAM, in kilobytes.
MemFree – The amount of physical RAM, in kilobytes, left unused by the system.
Buffers – The amount of physical RAM, in kilobytes, used for file buffers.
Cached – The amount of physical RAM, in kilobytes, used as cache memory.
Active – The total amount of buffer or page cache memory, in kilobytes, that is in active use. This is memory that has been recently used and is usually not reclaimed for other purposes.
Inactive – The total amount of buffer or page cache memory, in kilobytes, that are free and available. This is memory that has not been recently used and can be reclaimed for other purposes.
LowTotal and LowFree – The total and free amount of memory, in kilobytes, that is directly mapped into kernel space. The LowTotal value can vary based on the type of kernel used.
Mapped – The total amount of memory, in kilobytes, which have been used to map devices, files, or libraries using the mmap command.
Slab – The total amount of memory, in kilobytes, used by the kernel to cache data structures for its own use.
Committed_AS – The total amount of memory, in kilobytes, estimated to complete the workload. This value represents the worst case scenario value, and also includes swap memory.
PageTables – The total amount of memory, in kilobytes, dedicated to the lowest page table level.
VMallocTotal – The total amount of memory, in kilobytes, of total allocated virtual address space.
VMallocUsed – The total amount of memory, in kilobytes, of used virtual address space.
VMallocChunk – The largest contiguous block of memory, in kilobytes, of available virtual address space.

Getting the absolute free memory number

Your manager wants to know how much free memory the system has. What he really wants to know actually is the minimum amount of memory that the system requires in order to function properly. Let’s call this number “The absolute free memory number”. Given this number and the amount of actual installed memory, the manager also wants to know how much RAM is left for the customers’ customizations, or whether a smaller RAM device can be used in order to reduce the hardware cost (or even both…).

The absolute free memory number is not only the value of MemFree, but the sum of the MemFree, Buffers and Cached fields. It means that the majority of the used memory by Buffers and Cached can be reclaimed in case a task will require it, while reducing the amount of buffers and cached. Using unused (clean) cache will not impact the system performance. However, degradation in the system performance will appear once starting to free used cache. In some point (usually in the last free 1MB, will be discussed soon) the degradation will be very noticeable and the system will become very slow. It happens because live tasks are swapped out of the RAM, and execution of commands requires flash/disk access and decompression. If even more RAM is required, the kernel will initiate the Out-Of-Memory Killer and start killing tasks that have the highest “bad” score (See the article about Linux processes for more details). In this stage, the system is not in a stable state and its functionality is not guaranteed. If your system reaches this state on a regular basis, you may either have a serious memory leak, or the actual physical RAM installed is insufficient.

Flush and invalidate system caches

It is possible to artificially increase the value of MemFree (and make your manager happy) by flushing and invalidating the clean (unused) cache and filesystem meta-data. This operation is non-destructive and it does not free the currently used caches (like running processes). Prior to this operation, the “sync” command needs to be run first in order to make sure all cached objects are synchronized.

To free pagecache:
– echo 1 > /proc/sys/vm/drop_caches
To free dentries and inodes:
– echo 2 > /proc/sys/vm/drop_caches
To free pagecache, dentries and inodes (basically everything):
– echo 3 > /proc/sys/vm/drop_caches

Here is the output of the same meminfo file, before and after all the caches were freed:

Memory status before dropping caches Memory status after dropping caches

MemTotal: 119248 kB MemFree: 77544 kB Buffers: 3392 kB Cached: 10260 kB SwapCached: 0 kB Active: 19612 kB Inactive: 7660 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 119248 kB LowFree: 77544 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 13640 kB Mapped: 5428 kB Slab: 4540 kB PageTables: 1616 kB NFS_Unstable: 0 kB Bounce: 0 kB CommitLimit: 59624 kB Committed_AS: 28588 kB VmallocTotal: 131072 kB VmallocUsed: 9296 kB VmallocChunk: 106492 kB MemTotal: 119248 kB MemFree: 85100 kB Buffers: 204 kB Cached: 6268 kB SwapCached: 0 kB Active: 16948 kB Inactive: 3144 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 119248 kB LowFree: 85100 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 13640 kB Mapped: 5428 kB Slab: 4164 kB PageTables: 1616 kB NFS_Unstable: 0 kB Bounce: 0 kB CommitLimit: 59624 kB Committed_AS: 28588 kB VmallocTotal: 131072 kB VmallocUsed: 9296 kB VmallocChunk: 106492 kB

We can see that we gained almost 8MB in of RAM in MemFree field (and also in the “free” report) just by dropping the caches, and that the amount of Buffers and Cached has been reduced.

Other kernel tunables regarding memory

The Kernel provides run-time tunables that can change its behavior in extreme cases:

/proc/sys/vm/min_free_kbytes: Used to force the Linux VM to keep a minimum number of kilobytes free. This number’s value is around 1MB. The Kernel will start swapping out and killing processes in order to meet this requirement.
/proc/sys/vm/overcommit_memory: Controls over-commit of system memory, possibly allowing processes to allocate (but not use) more memory than is actually available.

Resources:
http://www.linuxinsight.com/proc_sys_vm_hierarchy.html
https://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/en-US/Reference_Guide/s2-proc-meminfo.html
http://wiki.linuxquestions.org/wiki/FAQ_-_Linux_problems
http://linux.die.net/man/1/top
http://linux.die.net/man/1/free

Linux Loadable Kernel Modules

Hai Shalom — Thu, 01 Apr 2010 20:21:01 +0000

Linux loadable Kernel Modules (also referred as LKMs, DLMs (dynamically loaded modules), or just Kernel modules) can be viewed as “kernel extensions”. They are programs that run in the Kernel’s execution environment and extend its basic functionality.

Some more facts about LKMs:

Most modules in the Kernel tree provide: device drivers, networking support, filesystem support, and more.
Modules can be loaded and removed in runtime, without needing to neither recompile nor reboot the system.
Modules are not user space programs. They can access the Kernel’s API directly, but they can cause a system crash as well (A kernel panic).
A module can overwrite kernel system calls (replace system calls with its own implementation).
A module can export API’s as well.
Some drivers in the Kernel can be compiled as an Integral mode or module mode.
A module source code does not have to reside in the Kernel tree. It could reside outside of it as well.
The Loadable Kernel module extension name is “.ko”.
The compiled modules reside in the filesystem under the /lib/modules directory.

Module utilities

Module utilities are a set of tools that provide module-related services. The utilities include:

insmod: Insert (load) a module in runtime. It can also pass run time configuration parameters to the module.
rmmod: Remove (unload) a module in runtime.
lsmod: List all active modules and their usage count.
modprobe: Handle the loading of modules and their dependencies.

Kernel built-in modules

Some drivers in the Kernel can be compiled as an Integral mode or module mode. These drivers are marked with 3-state mode variables: Disabled, Modulized, and Enabled. In order to configure the driver to be compiled in your desired mode, you need to activate the kernel’s configuration menu (read the kernel configuration article for more details how to do it) and find the appropriate driver configuration option.

In the following example, you can see a screen shot of an I2C driver in each of the 3 states:

Switching between the modes is done either by pressing , or to enable, to modulize and to disable.

In case a driver was selected to be modulized, it will not be built when the kernel is built, and an explicit make call is required in order to build them. The first command is “make modules”, which builds all the modules, and the second command is “make modules_install INSTALL_MOD_PATH=/”, which installs the modules in the target’s filesystem.

Module licensing

The linux/module.h header file defines the following licensing options:

/*

* The following license idents are currently accepted as indicating free

* software modules

*

*  "GPL"               [GNU Public License v2 or later]

*  "GPL v2"            [GNU Public License v2]

*  "GPL and additional rights" [GNU Public License v2 rights and more]

*  "Dual BSD/GPL"          [GNU Public License v2

*                   or BSD license choice]

*  "Dual MIT/GPL"          [GNU Public License v2

*                   or MIT license choice]

*  "Dual MPL/GPL"          [GNU Public License v2

*                   or Mozilla license choice]

*

* The following other idents are available

*

*  "Proprietary"           [Non free products]

*

* There are dual licensed components, but when running with Linux it is the

* GPL that is relevant so this is a non issue. Similarly LGPL linked with GPL

* is a GPL combined work.

*

* This exists for several reasons

* 1.   So modinfo can show license info for users wanting to vet their setup is free

* 2.   So the community can ignore bug reports including proprietary modules

* 3.   So vendors can do likewise based on their own policies

*/

According to GNU GPLv2, since the module runs in the same memory space as the kernel itself, and the kernel is licensed under GPLv2, then the module is considered to be a “combined work”, therefore, any non-GPL license is not a valid license. When running in the kernel space, the GPLv2 license applies even if the license is a dual license. Theoretically, you can ignore the module licensing or define it as “Proprietary”, but this will cause the following issues (other than the legal issues):

The kernel will display a warning that this module taints the kernel. The open source communities ignore error reports or support requests by companies who taint the kernel.
Non-GPL licensed modules are denied from calling some of the Kernel API which is export by EXPORT_SYMBOL_GPL macro). Therefore, when trying to load them, they will fail to link and load in case they are calling one of these functions.

Writing your own Loadable Kernel Module – Kernel configuration

A few configuration options must be enabled in order to support Loadable Kernel Modules in your Kernel:

CONFIG_MODULES: Enables the module handling ability in the kernel, therefore, when modules are used, this option must be enabled.
CONFIG_MODULE_UNLOAD: Enables the module unloading ability in the kernel. Theoretically, modules can be removed from the system, and their resources freed. However, some modules do not support it correctly, and others can’t be removed because they are used by other drivers or programs. It is recommended to enable this option anyway.
CONFIG_MODULE_FORCE_UNLOAD: Allows the kernel to forcibly remove modules upon request, even if the kernel believes it is not safe. This could result by other drivers keeping reference or using this module. This option is mainly used for debug and should be disabled.
CONFIG_MODVERSIONS: Usually, each module is compiled for the kernel you are using. In case a precompiled module is supposed to run in another kernel environment, this option adds extra kernel information inside the module which could make it possible to detect incompatibilities. If this scenario is not a part of your setup, you can disable this option.
CONFIG_MODULE_SRCVERSION_ALL: Calculates a checksum from all the module’s sources, and adds it in the module’s binary. This option is used to detect version variations even if the module’s version string is the same.
CONFIG_KMOD: Allows the kernel to load its modules automatically, upon request. If you manage your modules insertion, then this option could be disabled, otherwise, it is recommended to enable it.

Writing your own Loadable Kernel Module – Source code

Your module can be made of one or more source files which perform the task it was made for. However, one source file must be used for interfacing with the Kernel. Let’s call this file; the “main file”. The main file:

Must include linux/kernel.h, linux/module.h and linux/init.h header files.
Must contain an init and an exit functions.
Should declare the module licensing.
Can declare author’s name and module description.
Can declare initialization parameters.
Can export API to be used by other modules.

The init function initializes the module’s data, allocates resources and registers the module in the kernel (if necessary). It can also print a message in case it succeeds or fails. Returning 0 means that the init process was successful. Otherwise, a negative value means a failure which will cause the module to be unloaded and insmod command to fail.

The exit function cleans up, frees the resources and unregisters the module from the kernel. The exit function does not return any value.

It is possible to declare variables to be initialized by insmod. Parameter types are: byte, short, ushort, int, uint, long, ulong and charp. It is also possible to define an array of each type (except charp which is already an array).

/* Declare the variables */

static char rte_byte = 128;

static short int rte_short = 1000;

static int rte_int = 128 * 1024;

static long int rte_long = 128 * 1024 * 1024;

static char *rte_string = "RT Embedded";

static int rte_int_array[4] = { 0, 1, 2, 3 };

static int rte_array_len = 0;

/* Setup modules parameters */

module_param(rte_byte, byte, 0000);

MODULE_PARM_DESC(rte_byte, "A byte");

module_param(rte_short, short, 0000);

MODULE_PARM_DESC(rte_short, "A short integer");

module_param(rte_int, int, 0000);

MODULE_PARM_DESC(rte_int, "An integer");

module_param(rte_long, long, S_IRUSR);

MODULE_PARM_DESC(rte_long, "A long integer");

module_param(rte_string, charp, 0000);

MODULE_PARM_DESC(rte_string, "A string");

module_param_array(rte_int_array, int, &rte_array_len, 0000);

MODULE_PARM_DESC(rte_int_array, "An array of integers");

Here’s an example for a basic module:

/*
 *  rte_lkm.c - Linux Loadable Kernel Module example
 */

#include 
#include  
#include 

/* Macro definitions */
#define LKM_AUTHOR          "Hai Shalom, http://www.rt-embedded.com" 
#define LKM_DESCRIPTION     "An example LKM"

static int __init init_lkm(void)
{
	printk(KERN_INFO “Hello, world!\n”);
	return 0;
}

int rte_lkm_func( int i )
{
	printk( “%s: i=%d\n”, __FUNCTION__, i );
	return 0;
}

static void __exit exit_lkm(void)
{
	printk(KERN_INFO “Goodbye, world…\n”);
}

module_init(init_lkm);
module_exit(exit_lkm);

/*
* Define the module’s license. Important!
*/

MODULE_LICENSE(“GPL”);

/*
* Optional definitions
*/

MODULE_AUTHOR(LKM_AUTHOR);  /* The author’s name */
MODULE_DESCRIPTION(LKM_DESCRIPTION);    /* The module description */

/*
* API the module exports to other modules
*/

EXPORT_SYMBOL(rte_lkm_func);

This module will write “Hello, world!” after the insmod command, and “Goodbye, world…” after the rmmod command.

Writing your own Loadable Kernel Module – Makefile

There is no need to understand the Make syntax in order to build a kernel module. All you need is a 5 line Makefile and it works. However, there are some options you may want to explore, and later, I will provide a simple template which addresses them nicely.

The make process is divided to two stages. In the first stage, the Makefile calls the Kernel build system in order to build the module. In the second stage, the build system uses the same Makefile in order to configure and build the required module. Therefore, there should be two parts in the Makefile, to handle each stage. The kernel provides a variable to detect if we are in the first or second stage. The variable name is KERNELRELEASE. Therefore, we shall use it to distinguish between the two parts. In the first part, we need to provide all the targets required in our build system, and call the kernel’s build system. In the second part, we define all the module’s parameters which are required in order to build the module. Please read the example Makefile. The remarks explain each line.

################################################################################

#

# Copyright(c) 2010, Hai Shalom, http://www.rt-embedded.com

#

# This program is free software; you can redistribute it and/or modify it

# under the terms of the GNU General Public License as published by the Free

# Software Foundation; either version 2 of the License, or (at your option)

# any later version.

#

################################################################################

# Target module name

TARGET := rtelkm



# Kernel directory - This works only for the host.

# The target kernel is different, and it can't be detected.

ifndef KERNEL_DIR

KERNEL_DIR := /lib/modules/$(shell uname -r)/build

endif# Target development home directory

ifndef TARGET_HOME

TARGET_HOME := /usr/target

endif# File system path, where to install the module

ifndef FILESYSTEM_PATH

FILESYSTEM_PATH := $(TARGET_HOME)/project/fs

endif

# Include export directory

ifndef TARGET_INCLUDE_PATH

TARGET_INCLUDE_PATH := $(TARGET_HOME)/$(TARGET_INCLUDE_PATH)

endif



# Include directories

INCLUDES := -I$(CURDIR)/../include \

-I$(KERNEL_DIR)/include/linux \

-I$(KERNEL_DIR)/include/asm \

-I$(TARGET_INCLUDE_PATH)



export EXTRA_CFLAGS += $(INCLUDES)



# Module extra compilation flags

EXTRA_CFLAGS +=



# Installation module directory: fill in the driver type

export INSTALL_MOD_DIR := /drivers/



# Filesystem path

export INSTALL_MOD_PATH=$(FILESYSTEM_PATH)



# Kernel module compilation – part 2

ifneq ($(KERNELRELEASE),)

obj-m += $(TARGET).o



# Target objects - add as many as required

$(TARGET)-objs := \

rte_lkm.o

else # Makefile targets – part 1



all: build install



build: #note that the MAKE commands must start with a 

@$(MAKE) -C $(KERNEL_DIR) M=`pwd` modules



install:

@$(MAKE) -C $(KERNEL_DIR) M=`pwd` modules_install



uninstall:

@find $(FILESYSTEM_PATH)/lib/modules/ -name $(TARGET).ko | xargs rm -rf



clean:

@$(MAKE) -C $(KERNEL_DIR) M=`pwd` clean

@rm -f Module.symvers



endif

Once your Makefile is complete, all you need to do in order to compile and install your module is to run “make” in the module’s directory.

Resources:
http://tldp.org/LDP/lkmpg/2.6/html/lkmpg.html
http://en.wikipedia.org/wiki/Loadable_kernel_module

Resolving/Debugging user space crashes and segmentation faults

Hai Shalom — Mon, 29 Mar 2010 19:09:52 +0000

What is a segmentation fault?

Segmentation fault is the most common error condition where your program tries to access either an invalid memory location, or a memory location it is not allowed to access.

A few examples for this could be:

Dereferencing a NULL pointer.
Dereferencing an uninitialized pointer.
Accessing memory with a wrong alignment.
Writing to a read only area.
Writing or reading beyond program allocated resources (buffer overflow).
Memory corruption/overrun.

For our examples, let’s use the following function that simply writes the character ‘R’ to a given location provided by ptr:

void rte_test_ptr( char *ptr )
{
    *ptr = 'R';
}

All the following code examples will cause the program to terminate and the message “Segmentation Fault” to appear on the console.

Example 1: Dereference a NULL pointer:

int main( int argc , char *argv[] )
{
    rte_test_ptr(NULL);
    return 0;
}

Example 2: Write to a read-only location:

char *ro_ptr = "RT-Embedded";

int main( int argc , char *argv[] )
{
    rte_test_ptr(ro_ptr);
    return 0;
}

Example 3: Dereference an uninitialized pointer:

int main( int argc , char *argv[] )
{
    char *uninit_ptr;

    rte_test_ptr(uninit_ptr);
    return 0;
}

What other crash types we usually see?

Another common error is alignment trap (also referred as bus error). A bus error occurs when the CPU tries to access a 32-bit or 16-bit variable in an unaligned memory address.

See the article about alignment traps for more details.

Another type of crash is caused be an illegal instruction. Normally, this cannot happen because the compiler generates only legal instructions. However, in case your program uses callbacks, this scenario could happen in case the program is trying to jump to an uninitialized callback address.

The last type of error which is covered in this article is the Floating point exception. If your program performed an illegal arithmetic operation (such as division by zero), the program will terminate and the message “Floating point exception” will appear on the console. Although the words “Floating point” are used, this type of error also refers to errors cause by arithmetic operations with integers.

What happens when the program performs an illegal operation?

When one of the above happens, the kernel sends a fault signal (or an exception) to the program. A fault signal is a special signal that tells the program that something bad has happened, and it needs to be terminated. There are four fault signals in the system:

SIGSEGV: In case of Segmentation fault
SIGBUS: In case of Alignment trap
SIGILL: In case of Illegal instruction
SIGFPE: In case of Illegal arithmetic operation.

Each signal needs to be handled and acknowledged. There are also the signals SIGQUIT and SIGINT which are not caused by an error, but also must be handled because the program needs to be terminated. In case the program did not install such handlers, there are default handlers for the fault signals that just write a short error message; “Segmentation fault“, and then terminate the program. Unfortunately, this is insufficient information in order to find and fix the bug, and the system may become unstable or unusable once the program was terminated. Furthermore, this short error message might get lost within the many other messages which appear on the console while the system is running. Note that this behavior is usually unaccepted in a project which is meant for mass production (the user must manually power cycle the unit in this case).

Enabling core dump

During debug, it is possible to enable core dump, which can be parsed off line on a host machine using gdb. The core dump contains useful information about the last known whereabouts of the program. Further information is available in the Enabling core dumps post.

Handling fault signals, and extracting information from them

In each system, it is crucial to install exception handlers for these fault signals, because you can never know when a program might crash (trust me; usually it happens in the customer’s premises or during some qualification tests). The PCD provides an easy and convenient way for registering its exception handlers, which provide a lot of useful debug information (See the next paragraph for more details). In case you want to write your own exception handler, you must consider the following issues:

The exception can occur anytime, therefore, the handler must be written carefully.
Many ANSI-C functions are not signal-safe (including printf( )…), therefore, your signal handler must use only signal safe functions. A list of signal safe functions can be found here.
Don’t try to call your main( ) function from your signal handler. It may appear that you’ve revived your program, but it is unclear what will be the consequences, because your program’s stack or heap could have been corrupted.
Your exception handler must call exit( ) once it has completed its work.

The next step is to write the exception handler. There are two types of exception handlers you can use; a standard exception handler which receives only the signal number, or an enhanced exception handler, which can also receive some more information, which may defer between architectures: a pointer to a siginfo_t and a pointer to a ucontext_t (casted to void *). Let’s take a look at the sigaction structure, which is defined in signal.h:

/* Structure describing the action to be taken when a signal arrives.  */

struct sigaction
{
    /* Signal handler.  */

    union     {
         /* Used if SA_SIGINFO is not set.  */
         /* Type of a signal handler.  */
         typedef void (*sa_handler) (int);

         /* Used if SA_SIGINFO is set.  */
         void (*sa_sigaction) (int, siginfo_t *, void *);

    }  __sigaction_handler;

    /* Additional set of signals to be blocked.  */
    __sigset_t sa_mask;

    /* Special flags.  */
    int sa_flags;

    /* Restore handler.  */
    void (*sa_restorer) (void);

};

In case we want a simple handler, we can specify our handler in the sa_handler field, and in case we want the enhanced handler, we define SA_SIGINFO flag in sa_flags and specify our handler in sa_sigaction. The following macros can be used for registering exception handlers:

#define SETSIG(sa, sig, func) \
    {    memset( &sa, 0, sizeof( struct sigaction ) ); \
         sa.sa_handler = func; \
         sa.sa_flags = SA_RESTART; \
         sigaction(sig, &sa, 0L); \
    }

#define SETSIGINFO(sa, sig, func) \
    {    memset( &sa, 0, sizeof( struct sigaction ) ); \
         sa.sa_sigaction = func; \
         sa.sa_flags = SA_RESTART | SA_SIGINFO; \
         sigaction(sig, &sa, 0L); \
    }

The first macro is for the simple handler and the second macro is for the enhanced handler. The macros use the sigaction( ) function.

Once your handler has been activated, it means that an illegal operation has occurred. If you enabled the enhanced handler, it is possible to read the siginfo_t structure which contains information about the crash. This structure is large and contains a number of unions. I would like to mention only the most relevant members:

typedef struct siginfo
{
    int si_signo;   /* Signal number.  */
    int si_errno;   /* If non-zero, an errno value associated with this signal,
                       as defined in .  */
    int si_code;    /* Signal code.  */

    /* SIGILL, SIGFPE, SIGSEGV, SIGBUS.  */
 struct {
       void *si_addr;  /* Faulting insn/memory ref.  */

    } _sigfault;

} siginfo_t;

This structure provides information about the signal number and its code (See the siginfo.h header code for textual information about each signal code), the last known errno and the address which caused the error condition in the si_addr pointer. This address in this case is the address that the CPU was trying to access and not an address of the instruction (which is held by the Program Counter). The ucontext_t structure contains a list of the core register sets and their last known values (such as the Program Counter). It varies between architectures.

What do we do with this information?

We can now understand the nature and root cause of the error. We can also understand what the fault address that caused the error is. From the ucontext_t structure, we can extract the Program Counter (for the last known execution address) and the Link register (for the return address in ARM architecture). Theoretically speaking, we could print this information to the console using printf( ) function. However, if you remember, this function is not signal-safe, and therefore, it cannot be used, although I have seen some signal handlers that do use printf to print it. So how can we do it right? The solution is to have a crash daemon which listens on a socket. We can use our signal handler to send this information to the socket, and the daemon will print it for us. The socket API is signal safe and could be used without a problem. There is and easier way to do it, and I’ll present it next. Let’s say for now that we extracted the value of the Program Counter from the ucontext_t structure, and that the CPU was executing the instruction in address 0×8548. Assuming we compiled our program with debug symbols, we can use the objdump utility and ask it to show the mixed assembly and C code (using the –S option) around this address. Here’s a snippet from the output:

# armeb-linux-uclibceabi-objdump -S segv

...
00008544 :
#include 
#include

void rte_test_ptr( char *ptr )
{
    *ptr = 'R';
    8544:       e3a03052        mov     r3, #82 ; 0x52
    8548:       e5c03000        strb    r3, [r0] }
    854c:       e12fff1e        bx      lr
...

The line in red caused the Segmentation Fault crash. Object file with symbols will be presented in mixed-mode of C and assembly. In case the object was compiled without symbols, the only information we could extract near the address is the function name.

It is also possible to extract the file name and line number from the address number using the addr2line utility. Here’s an example:

# armeb-linux-uclibceabi-addr2line -e segv -f 8548
rte_test_ptr
/home/hai/rte/segv.c:7

There are cases where the fault address is not inside the program’s code, but inside a shared library’s code. A shared library code address will be mapped much higher than the area of the program code. We could use the maps file in the proc filesystem to determine where the last command came from. However, in order to print it, we’ll have to reboot the system because once the program has crashed, its proc entry is already gone. After we have rebooted and figured out the new PID, we need to print its map file by the command “cat /proc//maps”. Here’s an example output:

# cat /proc/204/maps
00008000-00009000 r-xp 00000000 1f:07 59         /usr/sbin/segv
00010000-00011000 rw-p 00000000 1f:07 59         /usr/sbin/segv
04000000-04005000 r-xp 00000000 1f:06 231        /lib/ld-uClibc-0.9.29.so
04005000-04007000 rw-p 04005000 00:00 0
0400c000-0400d000 r--p 00004000 1f:06 231        /lib/ld-uClibc-0.9.29.so
0400d000-0400e000 rw-p 00005000 1f:06 231        /lib/ld-uClibc-0.9.29.so
0400e000-04023000 r-xp 00000000 1f:06 175        /lib/libticc.so
04023000-0402a000 ---p 04023000 00:00 0
0402a000-0402c000 rw-p 00014000 1f:06 175        /lib/libticc.so
0402c000-04067000 r-xp 00000000 1f:06 200        /lib/libuClibc-0.9.29.so
04067000-0406e000 ---p 04067000 00:00 0
0406e000-0406f000 r--p 0003a000 1f:06 200        /lib/libuClibc-0.9.29.so
0406f000-04070000 rw-p 0003b000 1f:06 200        /lib/libuClibc-0.9.29.so
04070000-04075000 rw-p 04070000 00:00 0
04075000-0407f000 r-xp 00000000 1f:06 137        /lib/libgcc_s.so.1
0407f000-04086000 ---p 0407f000 00:00 0
04086000-04087000 rw-p 00009000 1f:06 137        /lib/libgcc_s.so.1
0ece0000-0ecf5000 rwxp 0ece0000 00:00 0          [stack]

Look for the “x” symbol in the permission column for code sections. In this example, possible Program counter locations could theoretically reside in the ranges of 0×8000-0×9000, 0×4000000-0×4005000, 0x400e000-0×4023000, 0x402c000-0×4067000, and 0×4075000-0x407f000. Matching the Program Counter and Link register to one of these ranges will result with the faulty code section. We’ll see an example later.

In this example, we can see what the last executed command was, but not cause of the problem. The signal info structure also contains the signal code and the registers which could help us figure out what is the root cause. How can we extract this information easily? Well, just continue to read.

How can PCD help debugging, resolving and preventing crashes?

When a program crashes due to an exception, it will be terminated once the error message is displayed. This crash will not trigger any recovery action, and your system will probably because unstable or unusable. The PCD can help here in two fields:

Enhanced debugging capabilities and system recovery. Once registering to the PCD exception handlers, they will provide more information about this crash, including the Program counter, Link register (return address), all the other registers, last value of errno and the maps file of the process, right before it was terminated, without the need to reboot the system. The latter will help you analyze the location of the error just be looking at the PCD’s error report. It will also trigger a recovery action once the crash was detected, and return the system to functional mode. The crash information is also saved on a non-volatile storage for later/offline analysis. Let’s take the piece of code from example 2, and instrument it with the PCD exception handlers (See how easily it is done):

#include 
#include

void rte_test_ptr( char *ptr )
{
    *ptr = 'R';
}

char *ro_ptr = "RT-Embedded";

int main( int argc , char *argv[] )
{
    /* Register to PCD's exception handlers */
    PCD_API_REGISTER_EXCEPTION_HANDLERS();

    /* Crash test */
    rte_test_ptr(ro_ptr);

    printf(ro_ptr);

    return 0;
}

Let’s configure a simple PCD rule to start an monitor a program:

RULE = TEST_SIGSEGV
START_COND = NONE COMMAND = /usr/sbin/segv
SCHED = NICE,0
DAEMON = YES END_COND = NONE END_COND_TIMEOUT = -1
FAILURE_ACTION = REBOOT ACTIVE = YES

Here is the output on the console once PCD has started this rule. Note that the selected recovery action here was “Reboot”, that’s why the system is rebooting right after the crash. Pay attention to the bolded red line:

pcd: Starting process /usr/sbin/segv (Rule TEST_SIGSEGV).
pcd: Rule TEST_SIGSEGV: Success (Process /usr/sbin/segv (204)).

**************************************************************************
**************************** Exception Caught ****************************
**************************************************************************

Signal information:

Time: Thu Jan  1 00:00:12 1970
Process name: /usr/sbin/segv
PID: 204
Fault Address: 0x00008590
Signal: Segmentation fault
Signal Code: Invalid permissions for mapped object
Last error: Success (0)
Last error (by signal): 0

ARM registers:

trap_no=0x0000000e
error_code=0x0000081f
oldmask=0x00000000
r0=0x00008590
r1=0x0ecf4ba4
r2=0x00000000
r3=0x00000052
r4=0x00010690
r5=0x00000000
r6=0x0000846c
r7=0x00008418
r8=0x00000000
r9=0x00000000
r10=0x00000000
fp=0x00000000
ip=0x00000000
sp=0x0ecf4cf0
lr=0x0000856c
pc=0x00008548
cpsr=0x40000010
fault_address=0x00008590

Maps file:

00008000-00009000 r-xp 00000000 1f:07 59         /usr/sbin/segv
00010000-00011000 rw-p 00000000 1f:07 59         /usr/sbin/segv
04000000-04005000 r-xp 00000000 1f:06 231        /lib/ld-uClibc-0.9.29.so
04005000-04007000 rw-p 04005000 00:00 0
0400c000-0400d000 r--p 00004000 1f:06 231        /lib/ld-uClibc-0.9.29.so
0400d000-0400e000 rw-p 00005000 1f:06 231        /lib/ld-uClibc-0.9.29.so
0400e000-04023000 r-xp 00000000 1f:06 175        /lib/libticc.so
04023000-0402a000 ---p 04023000 00:00 0
0402a000-0402c000 rw-p 00014000 1f:06 175        /lib/libticc.so
0402c000-04067000 r-xp 00000000 1f:06 200        /lib/libuClibc-0.9.29.so
04067000-0406e000 ---p 04067000 00:00 0
0406e000-0406f000 r--p 0003a000 1f:06 200        /lib/libuClibc-0.9.29.so
0406f000-04070000 rw-p 0003b000 1f:06 200        /lib/libuClibc-0.9.29.so
04070000-04075000 rw-p 04070000 00:00 0
04075000-0407f000 r-xp 00000000 1f:06 137        /lib/libgcc_s.so.1
0407f000-04086000 ---p 0407f000 00:00 0
04086000-04087000 rw-p 00009000 1f:06 137        /lib/libgcc_s.so.1
0ece0000-0ecf5000 rwxp 0ece0000 00:00 0          [stack]

**************************************************************************
pcd: Error: Process /usr/sbin/segv (204) exited unexpectedly (Rule TEST_SIGSEGV).
pcd: Terminating PCD, rebooting system...
starting pid 205, tty '': '/bin/umount /var /sys'
The system is going down NOW!
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Restarting system.

As we can see, the details of this crash provided by the PCD can help you find and resolve this issue easily and quickly. Let’s extract the file name and line number from the address number using the mentioned addr2line utility:

# armeb-linux-uclibceabi-addr2line -e segv -f 8548
rte_test_ptr
/home/hai/rte/segv.c:7

Now let’s get back to the objdump’s output for a more detailed output:

00008544 :
#include 
#include

void rte_test_ptr( char *ptr )
{
    *ptr = 'R';
    8544:       e3a03052        mov     r3, #82 ; 0x52
    8548:       e5c03000        strb    r3, [r0] }
    854c:       e12fff1e        bx      lr

00008550 :

char *ro_ptr = "RT-Embedded";

int main( int argc , char *argv[] )
{
    8550:       e92d4010        push    {r4, lr}
    /* Register to PCD's exception handlers */
    PCD_API_REGISTER_EXCEPTION_HANDLERS();

    /* Crash test */
    rte_test_ptr(ro_ptr);
    8554:       e59f4020        ldr     r4, [pc, #32]   ; 857c 
char *ro_ptr = "RT-Embedded";

int main( int argc , char *argv[] )
{
    /* Register to PCD's exception handlers */
    PCD_API_REGISTER_EXCEPTION_HANDLERS();
    8558:       e5910000        ldr     r0, [r1]
    855c:       e3a01000        mov     r1, #0  ; 0x0
    8560:       ebffffb8        bl      8448 <_init+0x30>

    /* Crash test */
    rte_test_ptr(ro_ptr);
    8564:       e5940000        ldr     r0, [r4]
    8568:       ebfffff5        bl      8544 
    printf(ro_ptr);
    856c:       e5940000        ldr     r0, [r4]     8570:       ebffffb1        bl      843c <_init+0x24>

    return 0;
}
    8574:       e3a00000        mov     r0, #0  ; 0x0
    8578:       e8bd8010        pop     {r4, pc}
    857c:       00010690        .word   0x00010690

We can see that the red line is the last instruction that the CPU performed before the crash, and the return address (by the Link Register) is marked by the orange line. In many cases, a function is called from various locations; the Link Register value tells us what the specific location is, and therefore, is also important. From the code, we can see that r4 is loaded with address 0x857c and that the value in this address is sent to our rte_test_ptr function. We can use the objdump to lookup the variable’s name by grepping on the value of the word:

# armeb-linux-uclibceabi-objdump -x segv | grep 10690
00010690 g     O .data  00000004 ro_ptr

Now we also know the variable name, although it was possible because this variable was global and not on the stack.

Suppose the faulty function is inside a shared library and not in our program. How can we debug this?
Let’s repeat the example, and place the rte_test_ptr inside a shared library. After we compile and link, we run the program again. We’ll reexamine the crash log:

**************************************************************************
**************************** Exception Caught ****************************
**************************************************************************

Signal information:

Time: Thu Jan  1 00:01:11 1970
Process name: /usr/sbin/segv
PID: 206
Fault Address: 0x000085f4
Signal: Segmentation fault
Signal Code: Invalid permissions for mapped object
Last error: Success (0)
Last error (by signal): 0

ARM registers:

trap_no=0x0000000e
error_code=0x0000081f
oldmask=0x00000000
r0=0x000085f4
r1=0x0eb72b74
r2=0x00000000
r3=0x00000052
r4=0x04078000
r5=0x00000000
r6=0x000084ac
r7=0x0000844c
r8=0x00000000
r9=0x00000000
r10=0x00000000
fp=0x0eb72cd4
ip=0x0402c4a0
sp=0x0eb72cc0
lr=0x000085c0
pc=0x0402c4a4
cpsr=0x00000010
fault_address=0x000085f4

Maps file:

00008000-00009000 r-xp 00000000 1f:06 315        /usr/sbin/segv
00010000-00011000 rw-p 00000000 1f:06 315        /usr/sbin/segv
04000000-04005000 r-xp 00000000 1f:06 232        /lib/ld-uClibc-0.9.29.so
04005000-04007000 rw-p 04005000 00:00 0
0400c000-0400d000 r--p 00004000 1f:06 232        /lib/ld-uClibc-0.9.29.so
0400d000-0400e000 rw-p 00005000 1f:06 232        /lib/ld-uClibc-0.9.29.so
0400e000-04023000 r-xp 00000000 1f:06 175        /lib/libticc.so
04023000-0402a000 ---p 04023000 00:00 0
0402a000-0402c000 rw-p 00014000 1f:06 175        /lib/libticc.so
0402c000-0402d000 r-xp 00000000 1f:06 212        /lib/libsegv.so
0402d000-04034000 ---p 0402d000 00:00 0
04034000-04035000 rw-p 00000000 1f:06 212        /lib/libsegv.so
04035000-04070000 r-xp 00000000 1f:06 200        /lib/libuClibc-0.9.29.so
04070000-04077000 ---p 04070000 00:00 0
04077000-04078000 r--p 0003a000 1f:06 200        /lib/libuClibc-0.9.29.so
04078000-04079000 rw-p 0003b000 1f:06 200        /lib/libuClibc-0.9.29.so
04079000-0407e000 rw-p 04079000 00:00 0
0407e000-04088000 r-xp 00000000 1f:06 137        /lib/libgcc_s.so.1
04088000-0408f000 ---p 04088000 00:00 0
0408f000-04090000 rw-p 00009000 1f:06 137        /lib/libgcc_s.so.1
0eb5e000-0eb73000 rwxp 0eb5e000 00:00 0          [stack]

**************************************************************************
pcd: Error: Process /usr/sbin/segv (206) exited unexpectedly (Rule TEST_SIGSEGV).
pcd: Terminating PCD, rebooting system...
starting pid 207, tty '': '/bin/umount -l /nvram /var /sys'
The system is going down NOW!
Sent SIGTERM to all processes
Requesting system reboot
Restarting system.

Now we can see that unlike the previous example, the Program Counter is in a high address (0x0402c4a4 according to the log). We understand that it resides outside of the program’s code and it is somewhere in one of the linked shared libraries. As we already saw, we can determine which library executed the bad instruction by matching the PC value to the executable address ranges in the maps file. In this example, this address is in the execution range of /lib/libsegv.so, which is the library made for the purpose of this example. In order to understand where we can find the problem inside the library, we need to calculate the offset by reducing the library’s base address from the PC value. In this example, we do: 0x0402c4a4 – 0x402c000 = 0x4A4, and this is the address of the problematic code inside the library. Let’s use objdump utility again, but now we’ll specify the library name, and not the program’s name. We can truncate the output by using the grep utility, to match the address we calculated and a few lines before and after the match (use –A and –B options):

# armeb-linux-uclibceabi-objdump -S libsegv.so | grep 4a4 –B 4 –A 4

000004a0 :
void rte_test_ptr( char *ptr )
{
    *ptr = 'R';
 4a0:   e3a03052        mov     r3, #82 ; 0x52
 4a4:   e5c03000        strb    r3, [r0]
}
 4a8:   e12fff1e        bx      lr

In red we see the instruction that caused the crash, as expected, inside the rte_test_ptr function we moved to a shared library.

For conclusion, now we know how to:

Extract and understand the crash information provided in to a fault signal handler.
Find a bad instruction inside our program and inside a shared library.

Now we have all the required information to fix this crash. It can be done manually, and it can be done using the PCD.

Memory corruption

Crashes and segmentation faults may be also a result of memory corruption. There could be a case where some code unintentionally changes a memory portion which it does not own thus causing a mess to the rightful owner. Read here how to debug such errors.

Resources:

http://linux.die.net/man/2/signal
http://linux.die.net/man/2/sigaction
http://sourceforge.net/projects/pcd/

http://www.rt-embedded.com/blog/archives/enabling-core-dumps-in-embedded-systems/

Resolving Alignment Traps

Hai Shalom — Thu, 25 Mar 2010 18:46:04 +0000

What is an Alignment Trap?

The memory of the system is used for two main purposes: Store code, and store data. Regarding code, each 32-bit opcode must be stored in a 32-bit aligned address, meaning, the address of each opcode must divide by 4. A 16-bit processor (or ARM in Thumb mode), can access opcodes only in 16-bit aligned addresses, meaning the address must divide by 2. Storing the code differently in the memory will cause an unpredictable behavior because the CPU will read the opcodes incorrectly.

An alignment trap occurs when the CPU is required to access a 32-bit variable which is not 32-bit aligned (same for 16-bit).

Some examples for alignment of variables

When you define a variable globally, on the stack, or allocate memory for it on the heap, it will be positioned in a memory location according to its size. A 32-bit variable address must divide by 4, a 16-bit variable address must divide by 2 and an 8-bit variable address could be anywhere. The CPU has opcodes to access words, half-words and bytes, accordingly. Data structures which contain variables with various sizes will be padded to make their variables aligned according to their sizes, unless forcing it to skip padding with packed attribute. In this case, the compiler generates code that reads the nearest aligned address and isolates the required value using register shift, logical and and multiple byte access functions. This of course makes the code larger, and therefore, it is always recommended to work with the native-size variables of the CPU (in this case; int or unsigned int) to get the optimal performance, even if we don’t need their full range, and avoid packed structures as much as possible (note that there are cases that packed structures must be used, for example, hardware address or network packets). For conclusion, it looks like we are ok, and all kinds of memory accesses should be covered. Alignment traps occur when the CPU tries to perform a memory access on an unaligned address. But if we are covered, why do we still get alignment traps?

Let’s take a look on the following code snippet which defines some types of variables, a structure and a packed structure:

int global_i;

char global_c1;

short global_s;

struct rte_unpacked_struct_t

{

   char c1;

   int i;

   char c2;

   short s1;

 char c3;

};



struct rte_packed_struct_t

{

    char c1;

    int i;

    char c2;

    short s1;

    char c3;

} __attribute__((__packed__));



struct rte_unpacked_struct_t unpacked;

struct rte_packed_struct_t packed;

Now let’s print each variable’s address, using a simple program:

Address of unpacked = 0x109d8

Address of unpacked.c1 = 0x109d8

Address of unpacked.i = 0x109dc

Address of unpacked.c2 = 0x109e0

Address of unpacked.s1 = 0x109e2

Address of unpacked.c3 = 0x109e4

Address of packed = 0x1097a

Address of packed.c1 = 0x1097a

Address of packed.i = 0x1097b

Address of packed.c2 = 0x1097f

Address of packed.s1 = 0x10980

Address of packed.c3 = 0x10982

Address of global_i = 0x10994

Address of global_c1 = 0x10979

Address of global_s1 = 0x10978

As expected, the addresses of the unpacked structure, as well as the integers are 32-bit aligned, the addresses of the short integers are 16-bit aligned and the addresses of the 8-bit variables could be odd or even. Pay attention to the packed structure, where the addresses of its members are not aligned – but access to each one of them is expensive in terms of code size and performance.

An example of bad code which causes an alignment trap

Let’s use the above data types, and write a function that returns the address of s1 variable inside a rte_unpacked_t structure. From the main, we will use an integer variable (on purpose) which is incompatible with the size of the short integer (int is 32-bit in length, and short is 16-bit in length). From the output above, we know that the address of s1 is aligned to 16-bit (0x109e2). The following code produces a warning from the compiler, about some incompatible pointer type. In many cases, these warnings are ignored, or even worse, a casting is added in the main function to make it shut up…

short *rte_get_s1( void )

{

return &unpacked.s1;

}

int main( int argc, char *argv[] ) { int *val; val = rte_get_s1(); printf( "val = %d\n", *val );
return 0; }

As a result, our program crashes immediately, and we get the following error:

Alignment trap: alignment (268) PC=0x00008680 Instr=0xe5901000 Address=0x000109e2 FSR 0x011

Bus error

Well, this is easy because it’s a very simple and short program. It could be a major problem when it is hidden inside thousands of code lines, especially if someone used casting to avoid the warning. It could be even worse if the function returns a void pointer address to a database which contains various data types with different sizes.

We have an alignment trap error. Now what?

The error message provides a lot of useful information we could use in order to isolate the bug. We can see the process name, process id, the current value of the Program Counter register, what was the instruction that caused the error, what is the address which was not aligned and what is the value of FSR (in ARM platforms).

The process name and id gives us a general location of the problem. In a system with many programs, it’s a good first step towards the solution. As we suspected, the address in the error message matches the address of our packed.s1 variable, but usually we can’t conclude any information from this field because we don’t know the address of each variable. Now, we’ll take a closer look at the PC and Instr. The value of the Program Counter points to the process’s virtual memory, so the physical address is not 0×8680 but something else. We don’t really care about that. This value can help us to determine whether the instruction was executed from within the program or from a shared library.

In the next step, will reboot (or restart the process) and look at its maps file:

# cat /proc/276/maps

00008000-00009000 r-xp 00000000 00:0b 757 /var/alignment

00010000-00011000 rw-p 00000000 00:0b 757 /var/alignment

04000000-04005000 r-xp 00000000 1f:02 122 /lib/ld-uClibc-0.9.29.so

04005000-04007000 rw-p 04005000 00:00 0

0400c000-0400d000 r--p 00004000 1f:02 122 /lib/ld-uClibc-0.9.29.so

0400d000-0400e000 rw-p 00005000 1f:02 122 /lib/ld-uClibc-0.9.29.so

0400e000-04049000 r-xp 00000000 1f:02 117 /lib/libuClibc-0.9.29.so

04049000-04050000 ---p 04049000 00:00 0

04050000-04051000 r--p 0003a000 1f:02 117 /lib/libuClibc-0.9.29.so

04051000-04052000 rw-p 0003b000 1f:02 117 /lib/libuClibc-0.9.29.so

04052000-04057000 rw-p 04052000 00:00 0

0e94a000-0e95f000 rwxp 0e94a000 00:00 0 [stack]

Since it’s just an example, there are not many libraries. Let’s take a look at the address ranges in the first column, and compare against the reported value of the Program Counter. We can clearly see that the error comes from within the program’s code (See the first line, with the Read/Execute permissions). If there was a problem in a linked library (such as the libc library), the PC value would be mapped much higher, and for the libc library example, it would be something in the range of 0x400e000 and 0×4049000. In order to calculate the instruction address inside the library, we reduce the corresponding library’s base address from the reported PC value.

OK, so now we know where to look for. Assuming we compiled our program with debug symbols, we can use the objdump utility and ask it to show the mixed assembly and C code (using the –S option) around this address. Here’s a snippet from the output:

# armeb-linux-uclibceabi-objdump -S alignment

...

val = rte_get_s1();

867c: ebffff6b bl 8430 

8680: e5901000 ldr r1, [r0]

8684: e59f0018 ldr r0, [pc, #24] ; 86a4 

8688: ebffff26 bl 8328 <_init+0x24>

printf( "val = %d\n", *val );

return 0;

}

...

The highlighted numbers in the objdump output show us both the reported address and instruction, right next to the problematic code! This trick works even for the most complicated programs. Assuming you can’t figure out the correct address, you can try to grep the objdump’s output with the reported instruction. In high probability, you’ll get a few matching results, one of them hold the key to your error.

If the address belongs to a shared library, we use the objdump on the shared library’s binary code and not on the program’s code.

In case your program was not compiled with symbols, you can recompile the program with symbols and try to reproduce again. It is recommended to always compile the program with symbols. Symbols are removed anyway in the final target’s filesystem.

In order to find the exact assembly line when the problem comes from a library, we’ll subtract the correct base address from the reported PC value. This will be the offset of the problematic code inside the library.

For conclusion

Try to use 32-bit variables. Short and Char variables do not save memory or perform faster anyway because the CPU loads them into 32-bit registers in any case, and has even more work to deal with them correctly.
Use casting with caution. Beware of arrays of bytes; don’t try to access this array with a 32-bit pointer.
Clean the warnings. Don’t leave warnings in your code.
Don’t fix warnings that you are not sure how to fix. Consult with others.

How can PCD help debugging, resolving and preventing alignment traps?

When a program crashes due to an alignment trap, it will be terminated once the error message is displayed. This crash will not trigger any recovery action, and your system will probably because unstable or unusable. The PCD can help here in two fields:

Enhanced debugging capabilities and system recovery. Once registering to the PCD exception handlers, they will provide more information about this crash, including the Program counter, Link register (return address), all the other registers, last value of errno and the maps file of the process, right before it was terminated. The latter will help you analyze the location of the error just be looking at the PCD’s error report. It will also trigger a recovery action once the crash was detected, and return the system to functional mode. The crash information is also saved on a non-volatile storage for later/offline analysis. Let’s configure a simple PCD rule to start and monitor the alignment program:

# PCD Rule for alignment program

RULE = TEST_ALIGNMENT

START_COND = NONE

COMMAND = /usr/sbin/alignment 315305 12 42

SCHED = NICE,0

DAEMON = YES

END_COND = NONE

END_COND_TIMEOUT = -1

FAILURE_ACTION = REBOOT

ACTIVE = YES

Here is the output on the console once PCD has started this rule. Note that the selected recovery action here was “Reboot”, that’s why the system is rebooting right after the crash:

pcd: Starting process /usr/sbin/alignment (Rule TEST_ALIGNMENT).

Alignment trap: alignment (157) PC=0x000087cc Instr=0xe5901000 Address=0x00010b3a FSR 0x011

pcd: Rule TEST_ALIGNMENT: Success (Process /usr/sbin/alignment (157)).

************************************************************************** **************************** Exception Caught **************************** ************************************************************************** Signal information: Time: Thu Jan 1 00:03:10 1970 Process name: /usr/sbin/alignment PID: 157 Fault Address: 0x00000000 Signal: Bus error Signal Code: Kernel Last error: Success (0) Last error (by signal): 0

ARM registers:
trap_no=0×00000000
error_code=0×00000000
oldmask=0×00000000
r0=0x00010b3a
r1=0x0ea5ab84
r2=0×00000000
r3=0xfffff000
r4=0x0000000c
r5=0x0ea5aeb4
r6=0x0004cfa9
r7=0x00010b24
r8=0×00000000
r9=0×00000000
r10=0×00000000
fp=0×00000000
ip=0×00000000
sp=0x0ea5acd0
lr=0x000087cc
pc=0x000087cc
cpsr=0×40000010
fault_address=0×00000000

Maps file:

00008000-00009000 r-xp 00000000 1f:07 57 /usr/sbin/alignment
00010000-00011000 rw-p 00000000 1f:07 57 /usr/sbin/alignment
04000000-04005000 r-xp 00000000 1f:05 282 /lib/ld-uClibc-0.9.29.so
04005000-04007000 rw-p 04005000 00:00 0
0400c000-0400d000 r–p 00004000 1f:05 282 /lib/ld-uClibc-0.9.29.so
0400d000-0400e000 rw-p 00005000 1f:05 282 /lib/ld-uClibc-0.9.29.so
0400e000-04023000 r-xp 00000000 1f:05 213 /lib/libpcdapi.so
04023000-0402a000 —p 04023000 00:00 0
0402a000-0402c000 rw-p 00014000 1f:05 213 /lib/libpcdapi.so
0402c000-04067000 r-xp 00000000 1f:05 241 /lib/libuClibc-0.9.29.so
04067000-0406e000 —p 04067000 00:00 0
0406e000-0406f000 r–p 0003a000 1f:05 241 /lib/libuClibc-0.9.29.so
0406f000-04070000 rw-p 0003b000 1f:05 241 /lib/libuClibc-0.9.29.so
04070000-04075000 rw-p 04070000 00:00 0
04075000-0407f000 r-xp 00000000 1f:05 171 /lib/libgcc_s.so.1
0407f000-04086000 —p 0407f000 00:00 0
04086000-04087000 rw-p 00009000 1f:05 171 /lib/libgcc_s.so.1
0ea46000-0ea5b000 rwxp 0ea46000 00:00 0 [stack]
**************************************************************************
pcd: Error: Process /usr/sbin/alignment (157) exited unexpectedly (Rule TEST_ALIGNMENT).
pcd: Terminating PCD, rebooting system…
The system is going down NOW!
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Restarting system.
.

The details of this crash provided by the PCD can help you find and resolve this issue easily and quickly. The fact that it is also saved in the non-volatile storage of the target makes it even possible to fix it remotely or later during post-mortem analysis.

Process information in the proc filesystem

Hai Shalom — Mon, 22 Mar 2010 18:18:53 +0000

The proc filesystem is a pseudo/virtual filesystem which is used as a communication interface with the kernel. It is commonly mounted at /proc and the files and directories which reside there are created in run-time by the kernel, drivers and loadable modules. The directories usually mark an area or an item, and the files, which are usually read-only, provide information about this area or item. Files that allow writing can change the behavior of the kernel or the driver which provided the file.

In this article, I will present most of the files which you can find for each process in the proc filesystem. It might be slightly different in each and every Linux distribution, but most of it is the same. The following text was taken from the proc manual, and I added some more information when I felt necessary. Wherever you see a [pid], you need to replace it with an actual process ID number. To see the information, use the cat command (e.g. cat /proc/225/cmdline).

The list is long because there is a lot of information. Remember that this is only a partial list, which covers only the Process related files. There are more interesting files in the proc filesystem which will be covered in a different article.

The following files are available in the proc filesystem per each process in the system:

/proc/[pid]

There is a numerical subdirectory for each running process; the subdirectory is named by the process ID. Each contains the following pseudo-files and directories.

/proc/[pid]/cmdline

This holds the complete command line for the process, unless the whole process has been swapped out, or unless the process is a zombie. In either of these later cases, there is nothing in this file: i.e. a read on this file will return 0 characters. The command line arguments appear in this file as a set of null-separated strings, with a further null byte after the last string.

/proc/[pid]/cwd

This is a link to the current working directory of the process.

/proc/[pid]/environ

This file contains the environment for the process. The entries are separated by null characters, and there may be a null character at the end.

/proc/[pid]/exe

The exe file is a symbolic link containing the actual path name of the executed command. The exe symbolic link can be dereferenced normally – attempting to open exe will open the executable. You can even type /proc/[pid]/exe to run another copy of the same process as [pid].

/proc/[pid]/fd

This is a subdirectory containing one entry for each file which the process has opened, named by its file descriptor, and which is a symbolic link to the actual file (as the exe entry does). Thus, 0 is standard input, 1 standard output, 2 standard error, etc.

/proc/[pid]/maps

A file containing the currently mapped memory regions and their access permissions.

An example for the format:

address        perms offset  dev   inode      pathname
00008000-0000a000 r-xp 00000000 1f:06 294        /usr/sbin/gptimer
00011000-00012000 rw-p 00001000 1f:06 294        /usr/sbin/gptimer
00012000-00013000 rwxp 00012000 00:00 0          [heap]
04000000-04005000 r-xp 00000000 1f:06 231        /lib/ld-uClibc-0.9.29.so
04005000-04007000 rw-p 04005000 00:00 0
04008000-04009000 rw-s 00000000 00:07 65538      /SYSV03030004 (deleted)
0400c000-0400d000 r--p 00004000 1f:06 231        /lib/ld-uClibc-0.9.29.so
0400d000-0400e000 rw-p 00005000 1f:06 231        /lib/ld-uClibc-0.9.29.so
0400e000-04023000 r-xp 00000000 1f:06 175        /lib/libticc.so
04023000-0402a000 ---p 04023000 00:00 0
0402a000-0402c000 rw-p 00014000 1f:06 175        /lib/libticc.so
0402c000-04036000 r-xp 00000000 1f:06 193        /lib/libpthread-0.9.29.so
04036000-0403d000 ---p 04036000 00:00 0
0403d000-0403e000 r--p 00009000 1f:06 193        /lib/libpthread-0.9.29.so
0403e000-0403f000 rw-p 0000a000 1f:06 193        /lib/libpthread-0.9.29.so
0403f000-04041000 rw-p 0403f000 00:00 0
04041000-0407c000 r-xp 00000000 1f:06 200        /lib/libuClibc-0.9.29.so
0407c000-04083000 ---p 0407c000 00:00 0
04083000-04084000 r--p 0003a000 1f:06 200        /lib/libuClibc-0.9.29.so
04084000-04085000 rw-p 0003b000 1f:06 200        /lib/libuClibc-0.9.29.so
04085000-0408a000 rw-p 04085000 00:00 0
0408a000-04094000 r-xp 00000000 1f:06 137        /lib/libgcc_s.so.1
04094000-0409b000 ---p 04094000 00:00 0
0409b000-0409c000 rw-p 00009000 1f:06 137        /lib/libgcc_s.so.1
0409c000-0409e000 r-xp 00000000 1f:06 199        /lib/libdl-0.9.29.so
0409e000-040a5000 ---p 0409e000 00:00 0
040a5000-040a6000 r--p 00001000 1f:06 199        /lib/libdl-0.9.29.so
040a6000-040a7000 rw-p 00002000 1f:06 199        /lib/libdl-0.9.29.so
040a8000-04125000 rw-s 00000000 00:07 0          /SYSVf00b0251 (deleted)
04125000-04126000 ---p 04125000 00:00 0
04126000-04925000 rw-p 04126000 00:00 0
0eace000-0eae3000 rwxp 0eace000 00:00 0          [stack]

Where; address is the address space in the process that it occupies, perms is a set of access permissions:

     r = read
     w = write
     x = execute
     s = shared
     p = private (copy on write)

offset is the offset into the file/whatever, dev is the device (major:minor), and inode is the index of the associated file structure in the filesystem, for that device. 0 indicates that no inode is associated with the memory region.

The maps file also specify the address and range of the process heap (how much memory was dynamically allocated with malloc( )), and the address and range of the process stack.

/proc/[pid]/smaps

Provides an even more detailed information about the process mapped memory regions.

Here’s an example of the same process, which I had to truncate:

00008000-0000a000 r-xp 00000000 1f:06 294        /usr/sbin/gptimer
Size:                 8 kB
Rss:                  8 kB
Shared_Clean:         0 kB
Shared_Dirty:         0 kB
Private_Clean:        8 kB
Private_Dirty:        0 kB
00011000-00012000 rw-p 00001000 1f:06 294        /usr/sbin/gptimer
Size:                 4 kB
Rss:                  4 kB
Shared_Clean:         0 kB
Shared_Dirty:         0 kB
Private_Clean:        0 kB
Private_Dirty:        4 kB
00012000-00013000 rwxp 00012000 00:00 0          [heap]
Size:                 4 kB
Rss:                  4 kB
Shared_Clean:         0 kB
Shared_Dirty:         0 kB
Private_Clean:        0 kB
Private_Dirty:        4 kB
04000000-04005000 r-xp 00000000 1f:06 231        /lib/ld-uClibc-0.9.29.so
Size:                20 kB
Rss:                 20 kB
Shared_Clean:        20 kB
Shared_Dirty:         0 kB
Private_Clean:        0 kB
Private_Dirty:        0 kB
04005000-04007000 rw-p 04005000 00:00 0
Size:                 8 kB
Rss:                  8 kB
Shared_Clean:         0 kB
Shared_Dirty:         0 kB
Private_Clean:        0 kB
Private_Dirty:        8 kB
04008000-04009000 rw-s 00000000 00:07 65538      /SYSV03030004 (deleted)
Size:                 4 kB
Rss:                  4 kB
Shared_Clean:         4 kB
Shared_Dirty:         0 kB
Private_Clean:        0 kB
Private_Dirty:        0 kB
0400c000-0400d000 r--p 00004000 1f:06 231        /lib/ld-uClibc-0.9.29.so
Size:                 4 kB
Rss:                  4 kB
Shared_Clean:         0 kB
Shared_Dirty:         0 kB
Private_Clean:        0 kB
Private_Dirty:        4 kB
0400d000-0400e000 rw-p 00005000 1f:06 231        /lib/ld-uClibc-0.9.29.so
Size:                 4 kB
Rss:                  4 kB
Shared_Clean:         0 kB
Shared_Dirty:         0 kB
Private_Clean:        0 kB
Private_Dirty:        4 kB

Where; Size is the total size of the region, Rss is the resident size of the region (the actual used memory size), Shared Clean is the amount of shared memory that was not changed by the process (like the code of a shared library), Shared dirty is the amount of shared memory that was changed by the process (like a code of a duplicated shared library – wrong), Private clean¸ is the amount of memory (usually global data) that was not changed by the process (it is 0 most of the time) and Private dirty, which is the amount of memory that was modified by the process (like shared library’s global data which each process has its own copy of them). All the entries in this list are rounded to the system’s Page Size, in this example, it is 4KB.

/proc/[pid]/mem

The mem file provides a means to access the process memory pages, using open, fseek and read commands.

/proc/[pid]/root

This is a link to the root directory which is seen by the process. This root directory is usually “/”, but it can be changed by the chroot command.

/proc/[pid]/stat

This file provides status information about the process. This is used by the Process Show utility. It is defined in fs/proc/array.c source file and may differ from one distribution to another.

The fields, in order, with their proper scanf format specifiers, are:

pid %d The process ID.
comm %s The filename of the executable, in parentheses. This is visible whether or not the executable is swapped out.
state %c One character from the string “RSDZTW” where R is running, S is sleeping in an interruptible wait, D is waiting in uninterruptible disk sleep, Z is zombie, T is traced or stopped (on a signal), and W is paging.
ppid %d The PID of the parent.
pgrp %d The process group ID of the process.
session %d The session ID of the process.
tty_nr %d The controlling terminal of the process. (The minor device number is contained in the combination of bits 31 to 20 and 7 to 0; the major device number is in bits 15 t0 8.)
tpgid %d The ID of the foreground process group of the controlling terminal of the process.
flags %u The kernel flags word of the process. For bit meanings, see the PF_* defines in . Details depend on the kernel version.
minflt %lu The number of minor faults the process has made which have not required loading a memory page from disk.
cminflt %lu The number of minor faults that the process’s waited-for children have made.
majflt %lu The number of major faults the process has made which have required loading a memory page from disk.
cmajflt %lu The number of major faults that the process’s waited-for children have made.
cutime %ld Amount of time that this process’s waited-for children have been scheduled in user mode, measured in clock ticks.
cstime %ld Amount of time that this process’s waited-for children have been scheduled in kernel mode, measured in clock ticks.
priority %ld For processes running a real-time scheduling policy, this is the negated scheduling priority, minus one; that is, a number in the range -2 to -100, corresponding to real-time priorities 1 to 99. For processes running under a non-real-time scheduling policy, this is the raw nice value as represented in the kernel (The kernel stores nice values as numbers in the range 0 (high) to 39 (low), corresponding to the user-visible nice range of -20 to 19.
nice %ld The nice value, a value in the range 19 (low priority) to -20 (high priority).
num_threads %ld Number of threads in this process (since Linux 2.6).
itrealvalue %ld The time in jiffies before the next SIGALRM is sent to the process due to an interval timer. Since kernel 2.6.17, this field is no longer maintained, and is hard coded as 0.
starttime %llu The time in jiffies the process started after system boot.
vsize %lu Virtual memory size in bytes.
rss %ld Resident Set Size: number of pages the process has in real memory. This is just the pages which count towards text, data, or stack space. This does not include pages which have not been demand-loaded in, or which are swapped out.
kstkesp %lu The current value of ESP (stack pointer), as found in the kernel stack page for the process.
kstkeip %lu The current EIP (instruction pointer).
signal %lu The bitmap of pending signals, displayed as a decimal number. Obsolete, because it does not provide information on real-time signals; use /proc/[pid]/status instead.
blocked %lu The bitmap of blocked signals, displayed as a decimal number. Obsolete, because it does not provide information on real-time signals; use /proc/[pid]/status instead.
sigignore %lu The bitmap of ignored signals, displayed as a decimal number. Obsolete, because it does not provide information on real-time signals; use /proc/[pid]/status instead.
sigcatch %lu The bitmap of caught signals, displayed as a decimal number. Obsolete, because it does not provide information on real-time signals; use /proc/[pid]/status instead.
wchan %lu This is the “channel” in which the process is wait ing. It is the address of a system call, and can be looked up in a namelist if you need a textual name
nswap %lu Number of pages swapped (not maintained).
cnswap %lu Cumulative nswap for child processes (not maintained).
exit_signal %d Signal to be sent to parent when we die.
processor %d CPU number last executed on.
rt_priority %u Real-time scheduling priority, a number in the range 1 to 99 for processes scheduled under a real-time policy, or 0, for non-real-time processes.
policy %u Scheduling policy.

/proc/[pid]/statm

Provides information about memory status in pages. The columns are:

sizetotal	program size
resident	resident set size
share	shared pages
trs	text (code)
drs	data/stack
lrs	library
dt	dirty pages

/proc/[pid]/status

Provides much of the information in /proc/[pid]/stat and /proc/[pid]/statm in a readable form.

Here’s an example:

Name:   pcd
State:  S (sleeping)
SleepAVG:       0%
Tgid:   219
Pid:    219
PPid:   1
TracerPid:      0
Uid:    0       0       0       0
Gid:    0       0       0       0
FDSize: 32
Groups:
VmPeak:     1172 kB
VmSize:     1172 kB
VmLck:         0 kB
VmHWM:       356 kB
VmRSS:       356 kB
VmData:      148 kB
VmStk:        84 kB
VmExe:        28 kB
VmLib:       380 kB
VmPTE:         6 kB
Threads:        1
SigQ:   0/1024
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: fffffffffffabab1
SigCgt: 000000000001444e
CapInh: 0000000000000000
CapPrm: 00000000fffffeff
CapEff: 00000000fffffeff

/proc/[pid]/oom_adj

This file can be used to adjust the score used to select which process should be killed in an out-of-memory (OOM) situation. The kernel uses this value for a bit-shift operation of the process’s oom_score value: valid values are in the range -16 to +15, plus the special value -17, which disables OOM-killing altogether for this process. A positive score increases the likelihood of this process being killed by the OOM-killer; a negative score decreases the likelihood. The default value for this file is 0; a new process inherits its parent’s oom_adj setting. A process must be privileged (CAP_SYS_RESOURCE) to update this file.

/proc/[pid]/oom_score

This file displays the current score that the kernel gives to this process for the purpose of selecting a process for the OOM-killer. A higher score means that the process is more likely to be selected by the OOM-killer. The basis for this score is the amount of memory used by the process, with increases (+) or decreases (-) for factors including:

whether the process creates a lot of children using fork(2) (+);
whether the process has been running a long time, or used a lot of CPU time (-);
whether the process has a low nice value (i.e., > 0) (+);
whether the process is privileged (-); and
whether the process is making direct hardware access (-).

The oom_score also reflects the bit-shift adjustment specified by the oom_adj setting for the process.

Resources:
http://linux.die.net/man/5/proc

Processes – Background and Priority

Hai Shalom — Thu, 18 Mar 2010 20:29:30 +0000

This article is the first amongst a series of articles about processes in Linux. The first article provides some background about processes, and covers process priority and scheduling policies.

What is a process?

A process is a program, an independent entity, which performs a dedicated task in the system. A process is composed of one or more binary objects which contain function implementations and an entry point function called main( ). Each process has a father, which created it, and can have many children processes which it can create. Each process can belong to a certain group (similar to a family).

Context and Process ID

In Linux, each process has its own unique Process ID (or pid). Each process has a context of its own. It means that each process has its own copy of registers, data and instruction memory, which can be accessed by the process itself. The process show utility (ps) provides a list of all currently active processes in the system, as well as their owner and their virtual memory allocation:

# ps -a
  PID USER       VSZ STAT COMMAND
    1 root       564 S    init
    2 root         0 SW   [posix_cpu_timer]
    3 root         0 SW   [softirq-high/0]
    4 root         0 SW   [softirq-timer/0]
    5 root         0 SW   [softirq-net-tx/]
    6 root         0 SW   [softirq-net-rx/]
    7 root         0 SW   [softirq-block/0]
    8 root         0 SW   [softirq-tasklet]
    9 root         0 SW   [softirq-rcu/0]
   10 root         0 SW<  [desched/0]
   11 root         0 SW<  [events/0]
   12 root         0 SW<  [khelper]
   13 root         0 SW<  [kthread]
   17 root         0 SW<  [kblockd/0]
   21 root         0 SW<  [khubd]
   56 root         0 SW   [pdflush]
   57 root         0 SW   [pdflush]
   58 root         0 SW<  [kswapd0]
   59 root         0 SW<  [aio/0]
   84 root         0 SW<  [IRQ 41]
   88 root         0 SW   [mtdblockd]
  121 root         0 SW<  [IRQ 29]
  122 root         0 SW<  [IRQ 24]
  132 root         0 SW<  [IRQ 8]
  151 root       420 S    watchdog_rt -t 15 /dev/watchdog
  152 root       572 S    /bin/sh
  154 root       564 R    ps -a

The last item in the list is always the process show utility itself, and its state is Running because it was running during the time it was generating the list. Lines with square brackets mark kernel threads, which are also considered as processes, with the one difference that they run in Kernel space.

Process memory space

In Linux, unlike other embedded operating systems, each process has its own virtual memory space, meaning it is living in its own world and no other process can read or write its data and code memory. This model provides enhanced security and robustness and avoids memory overflow or corruption which another task is creating. On the other hand, it makes the programmer’s life harder when debugging is required. It is also different in the scenario where one program wants to call another program’s function. To resolve this issue, a logical software module can export its API as a shared library, thus allowing other processes to link with it and perform certain functionalities it exports. There are also a few mechanisms of Inter-Process-Communication which will be discussed in the IPC section. The proc filesystem provides a lot of information regarding the memory map of the process, as well as area sizes, permissions, and library mappings. An example will be provided in the proc section.

Process life cycle

The first process in the system is called init. We can see it in the ps output, and we can also see that its pid is 1. It is the father of all processes in the system and it is created by the kernel during the boot up process. A new child process is created by the fork( ) function (or one of its permutations), followed by an exec( ) function (or one of its permutations). Both functions are declared in unistd.h header file. The fork( ) function creates a new identical process which has a different process id. The child inherits his father’s properties, including working directory, signal masks, environment and also duplicates the father’s resources. When the father calls the fork( ) function, the function returns to two different contexts, which are executing the same code. Therefore, in order to distinguish between the father and the child, the function returns different return value in each case. When it returns at the father side, the return value will be the new child’s process id (or an error value, in case of failures). When it returns at the child side, the return value will be 0. So we could write our code to handle this situation correctly. See the following code snippet:

pid_t pid;

pid = fork();

if( pid == 0 ) {
    /* Child context */
    /* Do the child's work; usually we call exec here */
} else if( pid < 0 ) {
    /* This is an error */
    fprintf( stderr, "failed to fork\n" );
    exit(1);
} else {
    /* Father context */
    /* Do the father's work */
}

In most cases, once a child process is created, we use the exec( ) function to load a new program code and start its main( ) function, instead of running two identical copies of the same program. The exec( ) function has a few variations, which configure the parameters which are passed to the child process in various methods. See the manual page for more details. A process can be terminated by sending a termination signal. The father which created the child knows its process id, and can request it to terminate. In case a child terminates on its own, it must be picked up by its father, or else it will remain a zombie. Since zombies are dead already, you can’t kill them, and they won’t go away until their father picks them up. Picking up a child process is done by the wait( ) or waitpid( ) functions. Once the father is notified that the child was terminated and used the above mentioned functions, the child will disappear from the system completely. The wait functions are declared in sys/wait.h header file.

Scheduling policy and task priority

How can I guarantee that a high priority program will always get enough CPU time? What is the best scheduling policy and priority for my program? How do I change my program’s scheduling policy and priority?

In most embedded Linux projects, there are two scheduling policies:

Standard time sharing policy: Scheduling policy which is meant for most tasks which share the CPU resources in fair turns, according to their given priority. This policy is not preemptive, and each process gets a CPU time slice in its turn. In case a low priority process does not get CPU time, its priority is temporarily raised until it is processed, then it gets its original priority back. This scheduling policy is also referenced as nice. The higher priority the program has, the nicer it is. This term was given because interactive processes must be responsive to their users, otherwise, the user will get suspicious that something is wrong in the system.
FIFO (First-in-First-out) Real-Time preemptive policy: Scheduling priority which is meant for real-time, high priority tasks. This policy is preemptive, meaning that if a higher priority FIFO task needs the CPU time, it will preempt any lower priority FIFO task, as well as all other nice programs. Therefore, the work that these tasks do must be quick and short; otherwise, it will starve the rest of the system.

The total amount of “priorities” in the Linux OS is 120, as seen in the following diagram:

Some more facts about scheduling:

The default scheduling policy and priority is nice 0.
FIFO lowest priority is higher than nice highest priority.
The nice policy has a “reversed” numbering scheme. The lowest the priority value is, the highest its priority is.
FIFO 99 is the highest priority in the system.
Nice 19 is the lowest priority in the system.
Each process inherits its father’s scheduling policy and priority.
Each process can change its priority according to its needs.
A process controller program (such as PCD) can configure the appropriate priority per each program it controls.

Reading and changing a nice-policy priority is done using the getpriority( ) and the setpriority( ) functions accordingly. These functions are declared in sys/resource.h header file. Note that these functions have no effect for processes running in Real-Time-policy. Reading the scheduling policy is done using the sched_getscheduler( ) function, and the sched_setscheduler( ) function can be used to both switch to a different scheduling policy and to set a FIFO-policy priority. Both functions are declared in linux/sched.h header file. It is also possible to retrieve the process scheduling policy and priority in the system’s shell, by the proc filesystem. An example will be provided in the article about the proc filesystem.

Assigning the correct scheduling and priority to your task

Here is a set of groups that can help you assign the right scheduling and priority to your task:

Low priority tasks (range 19-1): Tasks will be processed when the system is idle. Mainly used for logging and garbage collection daemons.
Normal priority tasks (0): The default middle balanced priority. The init process is created with priority 0, and all its children inherit it. This priority level is used by tasks which can compete on the CPU time and are not response critical.
Above average priority tasks (range (-1) – (-20)): Tasks will be processed more often than lower/normal priority tasks. This priority range is used by tasks which require higher responsiveness on one hand and lengthy amount of CPU time on the other hand (remember that the normal scheme is based on fare sharing). Examples for this kind of tasks are user space drivers and command line interfaces (like shells) which must react quickly. Otherwise (in case of shells), the user will think that the system is unstable due to lack of responsiveness.
Real-Time lower range (1-49): Tasks will be processed upon request while preempting lower priority tasks. Uses by tasks which require a fast response (usually less than a few milliseconds), and do not consume a lot of CPU time, otherwise, a system starvation will occur.
Real-Time average priority (50): In the Embedded Linux distro with RT patch enabled, this priority value is used by the kernel’s soft IRQ tasks and interrupt handlers. It is recommended not to use this level, but consider either lower or higher levels.
Real-Time high priority range (51-99): Similar idea as the RT lower range, but differs that it also preempts the kernel soft IRQ tasks and interrupt handlers. It should be used carefully and assigned only to super high time-critical tasks which seldom wake up and consume a very short amount of CPU time.

Process Stack

Process stack grows automatically upon request. The only limitation is the amount of available memory. Note that this doesn’t mean that you can allocate all your memory on the stack, because once it is allocated, it can’t be freed are reused by other process.

Resources:

http://oreilly.com/catalog/linuxkernel/chapter/ch10.html
http://www.opengroup.org/onlinepubs/007908799/xsh/fork.html
http://www.opengroup.org/onlinepubs/007908799/xsh/exec.html
http://www.opengroup.org/onlinepubs/007908799/xsh/getpriority.html
http://www.opengroup.org/onlinepubs/007908799/xsh/sched_getscheduler.html