• http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=tree;f=Documentation/trace
• Debugging the kernel using Ftrace – part 1: http://lwn.net/Articles/365835/
• Debugging the kernel using Ftrace – part 2: http://lwn.net/Articles/366796/
• Secrets of the Ftrace function tracer: http://lwn.net/Articles/370423/
• http://elinux.org/Kernel_Trace_Systems

In this example, I am working on a new audio driver.  The typical experience with a new driver is that you install it and nothing happens because something is not registered correctly with the Linux driver model.  So, the first thing I do is start with with the platform_device_add() function in my drivers init function.  To observe the kernel activity around the kernel platform code, I can do the following:

cd /sys/kernel/debug/tracing/
echo 0 > tracing_on (keep trace from filling up until we set filter)
echo function_graph > current_tracer
echo platform* > set_ftrace_filter
echo 1 > tracing_on
cat trace_pipe (leave running in a different shell)
<insmod my driver>

After executing the above, we see the following.  For this example, trace_pipe is preferred because the trace is then emptied and only new information is shown.

0) + 30.518 us   |  platform_device_alloc();
0)               |  platform_device_add() {
0)   0.000 us    |    platform_uevent();
0) + 30.518 us   |  platform_uevent();
0)   0.000 us    |  platform_uevent();
0) + 30.518 us   |    platform_match();
0) + 30.518 us   |    platform_match();
0)   0.000 us    |    platform_match();
0)   0.000 us    |    platform_match();

...

0) + 30.518 us   |    platform_match();
0)   0.000 us    |    platform_match();
0)   0.000 us    |    platform_match();
0)   0.000 us    |    platform_match();
0)   0.000 us    |    platform_match();
0) ! 3936.767 us |  }
0) + 30.518 us   |  platform_uevent();
0) + 30.518 us   |  platform_device_alloc();

From the above, I can conclude that the platform_match() is not succeeding, because I would expect some more activity.  At this point I chose to add a printk:

diff --git a/drivers/base/platform.c b/drivers/base/platform.c
index 7a24895..f9ce0c7 100644
--- a/drivers/base/platform.c
+++ b/drivers/base/platform.c
@@ -662,6 +662,8 @@ static int platform_match(struct device *dev, struct device_driver *drv)
        struct platform_device *pdev = to_platform_device(dev);
        struct platform_driver *pdrv = to_platform_driver(drv);

+       trace_printk("pdev->name = %s, drv->name = %s", pdev->name, drv->name);
+
        /* Attempt an OF style match first */
        if (of_driver_match_device(dev, drv))
                return 1;

Now, if I re-run the trace, I see the following:

 0)               |      /* pdev->name = soc_audio, drv->name = davinci_emac */
 0)   0.000 us    |    }
 0)               |    platform_match() {
 0)               |      /* pdev->name = soc_audio, drv->name = snd-soc-dummy */
 0)   0.000 us    |    }
 0)               |    platform_match() {
 0)               |      /* pdev->name = soc_audio, drv->name = soc-audio */
 0)   0.000 us    |    }
 0)               |    platform_match() {
 0)               |      /* pdev->name = soc_audio, drv->name = omap-pcm-audio */
 0)   0.000 us    |    }
 0) ! 4241.943 us |  } /* platform_device_add */

From the above, it looks like we have a simple mismatch between “soc_audio” and “soc-audio.”  Fixing this problem, and re-installing the module, we now have:

 0)               |    platform_match() {
 0)               |      /* pdev->name = soc-audio, drv->name = snd-soc-dummy */
 0)   0.000 us    |    }
 0)               |    platform_match() {
 0)               |      /* pdev->name = soc-audio, drv->name = soc-audio */
 0)   0.000 us    |    }
 0) + 91.553 us   |    platform_drv_probe();
 0) ! 4241.943 us |  } /* platform_device_add */

Now we can see that the names match, and the probe function is now being called.  At this point, we may want to turn on tracing of some additional functions to try to determine what is happening next.

echo "platform* snd* mydriver*" > set_ftrace_filter

And the result:

 0)               |      /* pdev->name = soc-audio, drv->name = snd-soc-dummy */
 0)   0.000 us    |    }
 0)               |    platform_match() {
 0)               |      /* pdev->name = soc-audio, drv->name = soc-audio */
 0) + 30.517 us   |    }
 0)               |    platform_drv_probe() {
 0)               |      snd_soc_register_card() {
 0) + 30.518 us   |        snd_soc_instantiate_cards();
 0) ! 17852.78 us |      }
 0) ! 17883.30 us |    }
 0) ! 22125.24 us |  } /* platform_device_add */

With the above additional information, we can continue to learn more about the flow through the kernel.

While all of the above could have been done with printk’s, it would have been more time consuming.  The kernel function tracing capabilities allow us to quickly get a high level view of the flow through the kernel without manually adding a bunch of printk statements.  The kernel tracing features are completely contained in the kernel without requiring additional user space utilities which makes it very convenient to use in embedded systems.  The low overhead is also important in resource constrained embedded systems.

Just recently, I discovered that screen can be used as a serial terminal program:

screen /dev/ttyUSB0 115200

A few notes on using:

• to exit screen: Ctrl-a k
• to write a hardcopy of the screen image: Ctrl-a h
• to get help: Ctrl-?

All the neat features of screen are two numerous to list here, but one more that I’ll point out is the scrollback/copy feature (activated by Ctrl-a [ ).  This allows you to scroll back and the navigation works much like VI — what could be nicer?

Fortunately, both modems function very similar to previous modems, so with the drivers available in the Linux kernel, and standard pppd support in OpenEmbedded, they worked fine.

The Verizon UML290 modem provides a challenge in that it must be manually switched between 4G and 3G modes.  Typically this is done automatically by the vzaccess program Verizon supplies with the modem that runs on Windows.  The solution for this system was to manually set the modem to 3G mode as detailed on the following page:

http://www.evdoinfo.com/content/view/3492/64/

It appears that some embedded systems such as the Cradlepoint routers have implemented automatic 3G/4G switching support for the UML290, so this is no doubt possible with a little effort.

The Sprint U600 modem appears to default to 3G, or automatically switch inside the modem.

The same pppd scripts can be used with both modems:

# /etc/ppp/peers/verizon_um290
user a
password v
connect "/usr/sbin/chat -v -f /etc/ppp/peers/verizon_um290_chat"
defaultroute
usepeerdns
ttyACM0
921600
local
usepeerdns
debug
-detach

# /etc/ppp/peers/verizon_um290_chat
'' 'ATZ'
'OK' 'ATDT#777'
'CONNECT' ''

To initiate a connection:

pppd call verizon_um290

With Verizon cellular modems, it appears that port 22 is often blocked, so if you need to access a remote device via ssh, you may need to run ssh on a higher port number.  With 4G networks, it appears that the networking setup may be different in that a public IP address may not be assigned.  From the above evdoinfo.com page, we find the following text:

This fix will also work for users looking to use their device for remote based applications because it assigns a public facing IP address (3G ONLY). With eHRPD you’re assigned a private IP in either 3G or 4G mode, which has prevents UML290 users from accessing remote applications.

Perhaps the Rev A HDR Service mode will also work in 4G mode, but its seems as cellular networks become more complicated, there will be more issues to deal with in using USB Cellular modems for remote access in embedded systems.

Past articles on USB Cellular modems:

• http://bec-systems.com/site/353/sprint-598u-usb-broadband-modem-in-embedded-systems
• http://bec-systems.com/site/203/using-a-verizon-usb720-modem-in-an-embedded-linux-system

The “many repository” paradigm has been partly driven by the distributed development paradigm.  Git solves the problem of multiple developers working in multiple repositories very well.  Because we want to use and customize projects like the Linux kernel, U-boot, and OpenEmbedded in our projects, then we naturally find ourself in the situation where we need to manage multiple repositories.  Yes, you can check the Linux kernel into your company Subversion repository, but you are much better off long term if you bite the bullet and implement your own Git infrastructure.

As we consider the product development process, we need to consider the life cycle of a product.  Most products live for at least several years, and will go through several software iterations.  If we can update the software components we use, then we can add value to the product in the form of new or updated drivers to support new peripherals, new libraries, performance improvements, etc.  But, we are dealing with millions of lines of source code, so we must have an efficient way to deal with software projects of this size.  The below Figure 2 illustrates how you might organize a typical project.  Notice we can pull updates from the U-boot and Kernel source trees at any time in the development process, and merge the changes with our own modifications.  We might have an outside team working an application, and we then easily synchronize the repositories when it makes sense.

There are many other design flows possible.  Once you have the ability to support multiple branches and repositories easily, it becomes trivial to implement a staging/testing repository for a QA processes, maintenance repositories for supporting old releases, etc.

Even at a personal developer level, Git’s distributed capabilities offers many advantages.  Each Git workspace is actually a full Git repository.  This means you can check changes in locally, re-organize your changes, be able to track changes when off-line, etc.  For this reason, many developers are now using git-svn when they need to work with Subversion repositories.  With git-svn, you have all the benefits of using git locally, and yet you can easily synchronize with Subversion repositories.  And this leads us to our next topic: cheap branches (coming soon).

• http://internal.omtp.org/Lists/ReqPublications/Attachments/59/OMTP CCLDC V1.1 Final.pdf
• http://www.usb.org/developers/devclass_docs/Battery_Charging_V1_2.zip

There is also a Chinese standard titled the “Telecommunications Industry Standard of the PRC” that has been instrumental in influencing these standards.

The fundamental problem is a battery powered device needs to know if it is plugged into a computer USB port, or a dedicated charger.  A standard USB port on a computer is only specified to provide 500mA.  Typically a dedicated charger that plugs into a wall outlet provides more current than 500mA so that the battery can be charged quicker.  To differentiate between a computer USB port, and a dedicated charger, the dedicated charger shorts the D+ and D- USB signals together with a resistance of less than 200 ohms (specified by USB-IF).

One of the goals of these standards is that any charger can be used with any battery powered device.  The next question is how do we handle the case where every USB charger has a different rated current?  How can a device that charges its battery at 1.7A be compatible with a charger that only outputs 0.7A?  One theoretical solution would be for the device to query the current capacity of the charger and then only use that much current.  It turns out a much simpler approach is used.  The USB-IF provides the following chart in the above specification:

taken from http://www.usb.org/developers/devclass_docs/Battery_Charging_V1_2.zip

As long as the charger outputs power in the Required Operating Range, the battery powered device must be able to use whatever power is available to charge its battery.  If the device uses more current than the charger can supply, the charger simply goes into a constant current mode and the battery charges slower than it would with a higher capacity charger.  Thus we have a very simple scheme where we can theoretically use any charger with any device.

As a simple test, I charged a Nokia N900 phone (which came with a 1.2A charger) with a smaller LG charger that is rated for 0.7A.  I monitored the LG charger a couple times during charging to make sure it was not getting hot.  It seemed to charge the N900 battery just fine.

When purchasing after-market USB chargers, it is sometimes difficult to determine if they have the USB D+ and D- lines shorted together.  In once case I purchased a car USB charger that did not work with my phones, but was able to “fix” it by disassembling it, and putting a solder blob between the D+ and D- signals on the USB connector.

While shorting the D+ and D- pins usually makes a charger work, it is nice to know if the charger is compliant with the USB-IF specification and is designed to work in the constant current mode without shutting down, catching fire, etc.

This article discusses the trend in software configuration management toward multiple repositories, rather than one large repository.  In the past when many companies used Subversion or comparable systems, there was typically one huge company repository (I’ve seen them in the 10′s of GB in size) that held a hierarchical tree of all the company source code.  This worked fairly well, and is comfortable in that the organization is very similar to a file system directory structure.  However, this model is not very flexible in that it does not have a consistent way to re-use components between different projects.  Some people simply copy source code.  Some have a “common” project that is included in all their other projects.  Subversion externals can be used.  With Git, typically a separate repository is created for each software component.  There are perhaps several reasons for this, but one reason is that Git simply does not scale to huge multi-GByte repositories.  However, this turns out to be a blessing in disguise as I think it is a better model in many cases.  What we end up with is more of a catalog of software components rather than a rigid hierarchy.

There is much emphasis these days on modular, re-usable software components (Object Oriented Programming, plugins, etc.).  Why not keep things modular at the repository level?  Another example of this type of organization is the Internet itself.  It is not a hierarchy of information, but rather a flat system that is organized by hyperlinks.

One of the benefits of organizing your source code this way is that it encourages clean boundaries between software components.  Each software component needs to stand on its own without being propped up by header files located in an unrelated source tree 3 levels up in the directory hierarchy.  This type of organization forces us to make better use of standard build system practices.

How do you implement this type of repository infrastructure?  Build systems such as OpenEmbedded, Gentoo, and the Android build system manage this fairly well.  However, Git also includes a feature named “submodules” that provides a mechanism for one repository to reference source code from another.  What you end up using really depends on your needs, and what you are trying to accomplish.

A screencast is also available that covers this topic.