When considering Embedded Linux development, I often use the iceberg analogy; what you see going into a project is the part above the water.  These are the pieces your application interacts with, drivers you need to customize, the part you understand.  The other 90% is under water, and herein lies a great deal of variability.  Quality issues with drivers or not being able to find a driver for something you may want to support in the future can easily swamp known parts of the project.  There are very few people who have a lot of experience with both WinCE and Linux solutions, hence the tendency to go with what is comfortable (or what managers are comfortable with), or what we have experience with.  Below are thoughts on a number of aspects to consider:

SYSTEM SOFTWARE DEVELOPMENT

Questions in this realm include CPU support, driver quality, in field software updates, filesystem support, driver availability, etc.  One of the changes that has happened in the past two years, is CPU vendors are now porting Linux to their new chips as the first OS.  Before, the OS porting was typically done by Linux software companies such as MontaVista, or community efforts.  As a result, the Linux kernel now supports most mainstream embedded cpus with few additional patches.  This is radically different than the situation 5 years ago.  Because many people are using the same source code, issues get fixed, and often are contributed back to the mainstream source.  With WinCE, the BSP/driver support tends to be more of a reference implementation, and then OEM/users take it, fix any issues, and that is where the fixes tend to stay.

From a system perspective, it is very important to consider flexibility for future needs.  Just because it is not a requirement now does not mean it will not be a requirement in the future.  Obtaining driver support for a peripheral may be nearly impossible, or be too large an effort to make it practical.

Most people give very little thought to the build system, or never look much beyond the thought that “if there is a nice gui wrapped around the tool, it must be easy”.  OpenEmbedded is very popular way to build embedded Linux products, and has recently been endorsed as the technology base of MontaVista’s Linux 6 product, and is generally considered “hard to use” by new users.  While WinCE build tools look simpler on the surface (the 10% above water), you still have the problem of what happens when I need to customize something, implement complex features such as software updates, etc.  To build a production system with production grade features, you still need someone on your team who understands the OS and can work at the detail level of both the operating system, and the build system.  With either WinCE or Embedded Linux, this generally means companies either need to have experienced developers in house, or hire experts to do portions of the system software development.  System software development is not the same as application development, and is generally not something you want to take on with no experience unless you have a lot of time.  It is quite common for companies to hire expert help for the first couple projects, and then do follow-on projects in-house.  Another feature to consider is parallel build support.  With quad core workstations becoming the standard, is it a big deal that a full build can be done in 1.2 hours versus 8?  How flexible is the build system at pulling and building source code from various sources such as diverse revision control systems, etc.

Embedded processors are becoming increasingly complex.  It is no longer good enough to just have the cpu running.  If you consider the OMAP3 cpu family from TI, then you have to ask the following questions: are there libraries available for the 3D acceleration engine, and can I even get them without committing to millions of units per year?  Is there support for the DSP bridge?  What is the cost of all this?  On a recent project I was involved in, a basic WinCE BSP for the Atmel AT91SAM9260 cost $7000.  In terms of developer time, this is not much, but you have to also consider the on-going costs of maintenance, upgrading to new versions of the operating system, etc.

APPLICATION DEVELOPMENT

Both Embedded Linux and WinCE support a range of application libraries and programming languages.  C and C++ are well supported.  Most business type applications are moving to C# in the WinCE world.  Linux has Mono, which provides extensive support for .NET technologies and runs very well in embedded Linux systems.  There are numerous Java development environments available for Embedded Linux.  One area where you do run into differences is graphics libraries.  Generally the Microsoft graphical APIs are not well supported on Linux, so if you have a large application team that are die-hard windows GUI programmers, then perhaps WinCE makes sense.  However, there are many options for GUI toolkits that run on both Windows PCs and Embedded Linux devices.  Some examples include GTK+, Qt, wxWidgets, etc.  The Gimp is an example of a GTK+ application that runs on windows, plus there are many others.  The are C# bindings to GTK+ and Qt.  Another feature that seems to be coming on strong in the WinCE space is the Windows Communication Foundation (WCF).  But again, there are projects to bring WCF to Mono, depending what portions you need.  Embedded Linux support for scripting languages like Python is very good, and Python runs very well on 200MHz ARM processors.

There is often the perception that WinCE is realtime, and Linux is not.  Linux realtime support is decent in the stock kernels with the CONFIG_PREEMPT option, and real-time support is excellent with the addition of a relatively small real-time patch.  You can easily attain sub millisecond timing with Linux.  This is something that has changed in the past couple years with the merging of real-time functionality into the stock kernel.

DEVELOPMENT FLOW

In a productive environment, most advanced embedded applications are developed and debugged on a PC, not the target hardware.  Even in setups where remote debugging on a target system works well, debugging an application on a workstation works better.  So the fact that one solution has nice on-target debugging, where the other does not is not really relevant.  For data centric systems, it is common to have simulation modes where the application can be tested without connection to real I/O.  With both Linux and WinCE applications, application programing for an embedded device is similar to programming for a PC.  Embedded Linux takes this a step further.  Because embedded Linux technology is the same as desktop, and server Linux technology, almost everything developed for desktop/server (including system software) is available for embedded for free.  This means very complete driver support (see USB cell modem and printer examples above), robust file system support, memory management, etc.  The breadth of options for Linux is astounding, but some may consider this a negative point, and would prefer a more integrated solution like Windows CE where everything comes from one place.  There is a loss of flexibility, but in some cases, the tradeoff might be worth it.  For an example of the number of packages that can be build for Embedded Linux systems using Openembedded, see http://cgit.openembedded.org/cgit.cgi/openembedded/tree/recipes.

GUI TRENDS

It is important to consider trends for embedded devices with small displays being driven by Cell Phones (iPhone, Palm Pre, etc).  Standard GUI widgets that are common in desktop systems (dialog boxes, check boxes, pull down lists, etc) do not cut it for modern embedded systems.  So, it will be important to consider support for 3D effects, and widget libraries designed to be used by touch screen devices.  The Clutter library is an example of this.

REMOTE SUPPORT

Going back to the issue of debugging tools, most people stop at the scenario where the device is setting next to a workstation in the lab.  But what about when you need to troubleshoot a device that is being beta-tested half-way around the world?  That is where a command-line debugger like Gdb is an advantage, and not a disadvantage.  And how do you connect to the device if you don’t have support for cell modems in New Zealand, or an efficient connection mechanism like ssh for shell access and transferring files?

SUMMARY

Selecting any advanced technology is not a simple task, and is fairly difficult to do even with experience.  So it is important to be asking the right questions, and looking at the decision from many angles.  Hopefully this article can help in that.  For additional assistance, please do not hesitate to contact BEC Systems — we’re here to help.

Sparse File Support

Most Unix file systems support sparse files.  This means that sections of data that is zeros is not stored on the disk.  Consider the following example where we create a 200MB file that is all zeros:

root@cm-x270:/media/mmcblk0p1# df
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/mmcblk0p1         3917212    202448   3515776   5% /media/mmcblk0p1

root@cm-x270:/media/mmcblk0p1# dd if=/dev/zero of=sparse-file bs=1 \
count=1 seek=200M

root@cm-x270:/media/mmcblk0p1# df
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/mmcblk0p1         3917212    202456   3515768   5% /media/mmcblk0p1

root@cm-x270:/media/mmcblk0p1# ls -l
-rw-r--r--    1 root     root    209715201 May 22 08:54 sparse-file

root@cm-x270:/media/mmcblk0p1# time od sparse-file
0000000  000000 000000 000000 000000 000000 000000 000000 000000
*
1440000000
real    0m 17.23s
user    0m 14.21s
sys     0m 2.66s

root@cm-x270:/media/mmcblk0p1# du sparse-file
68      sparse-file

It this case we created a 200MiB file on a SD card formatted as ext3. Even though the file is 200M in size, it only uses a few KiB of disk space.  This same test also worked fine with a JFFS2 filesystem.  With sparse file support, we get a cheap form of run length compression (at least for zeros) with no effort.

mmap()

The mmap() system call is used to map a file, or portions of a file into memory.  The data in the file can then be accessed directly in memory.  Because Linux is a demand paged system, portions of the file are paged in as needed, so the entire file does not need to be present in RAM at one time.  There are several advantages to using mmap():

1. mmap() avoids the extraneous copy that occurs when using read() or write() as the data does not need to be copied to a user space buffer.
2. there is very little overhead
3. you can directly access any part of the file without doing a lseek() and keeping track of where you are.
4. the operating system takes care of paging in sections of the file you are using, and discarding sections that are not in use when memory is low.  This includes flushing dirty portions to disk.

What this means, is mmap gives you an easy way to work on large, file backed data structures, and the OS takes care of loading the portions you need, and saving the modified portions back to disk.  As most embedded systems are 32-bit, there is a limit to the file size you can use as virtual memory space is limited.

Test Application

Next I wrote a small application that is used to create, and modify files using mmap().  I wanted to experiment with creating files of various sizes, writing non-zero data at various intervals, and test the performance of this.  The test application source is located here.

Usage: sparse_file_mmap_test [OPTION]

  -s, --filesize   File size to allocate
  -o, --offset     Will write a few bytes every offset bytes
  -m, --modify     Modify existing file
  -d, --data       Data to write to file at offset location (0-255)

The application creates a file of size filesize, and writes the value of data to the file every offset bytes.  There is also an option to modify existing files, so we can test the performance of opening a large file, making a small change, and then closing it.

Test Results

The results are pretty amazing, and the performance is beyond my expectations. This type of problem is where you learn to appreciate the performance of an advanced operating system, and file system.  There is a reason for the complexity!  There are 3 things I wanted to test: 1) does the sparse file support work as expected? 2) can mmap be used to easily modify large files? 3) can mmap be used to work on data structures that are larger than physical RAM?

1. Sparse File Support

The basic tests above confirm that sparse file support works for an empty file, but what about a file that has some data every X bytes?  Below are the test results for creating a 10MB file, and writing data at various offset intervals.

As soon as the offset was greater than the MMU page size (4KB), then the sparse file effect started to kick in and worked as expected.

2. Can mmap() be used to efficiently modify large files?

The test here was to open a large file, make a small change in the middle, and then close it.  In this case a 100MB file was created with a data value of 1 written every 1MB, and then re-opened the same file and wrote a data value of 2 every 2MB.  The od utility was used to examine the file to verify the contents were correct.

root@cm-x270:/media/mmcblk0p1# time sparse_file_mmap_test_arm -s104857600 \
-o1048576 -d1
Sparse File mmap() test
Filesize = 104857600, offset = 1048576, data = 1
size = 102400 KB
size on disk (KB):
508     sparse-file
real    0m 1.10s
user    0m 0.00s
sys     0m 0.43s

root@cm-x270:/media/mmcblk0p1# time sparse_file_mmap_test_arm -s104857600 \
-o2097152 -d2 -m
Sparse File mmap() test
Filesize = 104857600, offset = 2097152, data = 2
size = 102400 KB
size on disk (KB):
508     sparse-file
real    0m 0.37s
user    0m 0.01s
sys     0m 0.05s

root@cm-x270:/media/mmcblk0p1# time od -x sparse-file
0000000     0002    0000    0000    0000    0000    0000    0000    0000
0000020     0000    0000    0000    0000    0000    0000    0000    0000
*
4000000     0001    0000    0000    0000    0000    0000    0000    0000
4000020     0000    0000    0000    0000    0000    0000    0000    0000

....
*
614000000     0001    0000    0000    0000    0000    0000    0000    0000
614000020     0000    0000    0000    0000    0000    0000    0000    0000
*
620000000
real    0m 12.93s
user    0m 7.33s
sys     0m 1.70s

Creating and modifying large file-backed data structures is very fast and efficient, and takes on the order of 1 second for a 100MB file that contains data every 1MB.  Dumping the data with od took considerably longer (13 seconds) as 100MB of data needed to be processed.

Possible Issues

There are several things to watch out for when using sparse files and mmap()

1. With sparse files, there is the potential to run out of disk space as the files are using less space on disk than the file size.  It is a good idea to monitor disk space when working with sparse files, so you don’t use up all of the disk space.
2. mmap() requires virtual memory space for the size file it maps.  With embedded systems, this is less of an issue, because the physical RAM size tends to be much less than the 4GB virtual address space.  With a system that only has 64MB of RAM, mmap()’ing files in the 10’s of MB in size makes a lot of sense because it insures that you will not run the system out of memory, and yet there should be plenty of virtual address space to map in files of this size.

Conclusion

mmap() and sparse file support provide a very convenient solution for dealing with large, file-backed data structures.  One example of this type of data structure is any type of large matrix such as maps used in mapping applications.  Writing a “from scratch” solution to solve this problem would be a fairly large and difficult task.  Processing large amounts of data efficiently is becoming more and more important in many embedded systems. This example provides another compelling reason why implementing modern, data-centric embedded systems using Linux makes a lot of sense.

Below are few a notes from watching a few videos on the MV web site:

• Discussion with Jim Ready
  • around 2006 things changed
  • Semiconductor vendors now porting Linux to all their CPUs, so MV has moved up the stack
  • MVL6 first embedded linux distro produced in post-victory phase (Linux has won)
  • very friendly with other open source pieces (I assume OpenEmbedded)
  • enable open source
  • The decision the use Linux is the right decision technically.  What is the right business decision to accompany the technical decision?
• MVL6 Demonstration
  • toolchains pre-packaged
  • sources are all downloaded from MV servers
• Interview with Joe Green
  • manager of developer tools team
  • MLV6 is new approach
  • previous editions were consistent for all targets
  • MLV6 more flexible model.  Market specific distros depending on market.
  • source driven product.  Previous products based on binary RPMs.  MLV6 is a lot easier to build the whole environment.  Very customized/fine-tuned distro.
  • key problems MLV6 solves
    • Complete starting point for target hardware
    • complete build system, so you have total control vs pre-canned distribution that sets all the rules
  • Integration platform based on Bitbake.  Working closely with community.  Enhanced in a number of ways.  Customers have the option of bringing in software developed by the OpenEmbedded community and bringing it into their project.
  • Gives you a system that you can configure at almost any level.
  • Enhancements to make it easy to make changes, as well as provide a pre-built starting point.
  • Features to control sources, and environment so the host environment does not contaminate your build.
  • Essentially hardening, and commercializing the bitbake and openembedded projects.
  • Why MVL6 vs roll-your-own.
    • Qualified starting point
    • Support.  Community or Semi-conductor vendor support is hit or miss.
    • Consistent environment accross multiple systems.
    • Includes devrocket graphical tools

Overall, this seems like a very good approach.  I’m glad they are not re-inventing the wheel, but instead have chosen to build on top of a proven open source solution, and leverage the work that is being done by a very active community of OpenEmbedded developers.  The OpenEmbedded project currently has 61 developers (not all are active), and weekly contribution rate is very good.  This type of effort would be very difficult to match by any but the largest commercial organizations.  This should be a win-win situation for all involved.  As a common base technology, the Bitbake and OpenEmbedded projects should improve due to MV’s involvement.  Through the MVL6 product, the OpenEmbedded project and way of doing things will get a lot more exposure and there will be a lot more developers who understand how it works.  Developers who are familiar with the OpenEmbedded build system will be able to apply their knowlege to a greater number of projects.   And developers building products will have more options for getting the support and techology they need.

1. the target system has Ethernet support built into the kernel
2. the target system is on the same network as a Linux workstation
3. said network includes a DHCP server

This configuration emphasises “simple” and does not require you to spend 4 hours trying to get a bootp server configured on a test network.

Kernel Support

Make sure you have the following options turned on in the kernel:

• CONFIG_IP_PNP_DHCP=y
• CONFIG_ROOT_NFS=y

Then, add the following to your kernel CMDLINE:

• ip=dhcp root=/dev/nfs nfsroot=<nfs server IP>:/path/to/nfsroot

Workstation Setup

The following setup is for Ubuntu.

apt-get install nfs-user-server

(edit /etc/exports to contain something like the following)

/path/to/nfsroot 192.168.1.0/255.255.255.0(no_root_squash,sync,rw)

/etc/init.d/nfs-user-server restart

If you are using OpenEmbedded, then instruct OE to generate a rootfs tar image, and extract to your nfsroot directory:

cd /path/to/nfsroot
sudo tar -xvf <path to OE images dir>/Angstrom-image...rootfs.tar

And the result

eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1
Sending DHCP requests ., OK
IP-Config: Got DHCP answer from 192.168.1.1, my address is 192.168.1.112
IP-Config: Complete:
     device=eth0, addr=192.168.1.112, mask=255.255.255.0, gw=192.168.1.1,
     host=192.168.1.112, domain=hq.bec-systems.com, nis-domain=(none),
     bootserver=192.168.1.1, rootserver=192.168.1.11, rootpath=
Looking up port of RPC 100003/2 on 192.168.1.11
Looking up port of RPC 100005/1 on 192.168.1.11
VFS: Mounted root (nfs filesystem) readonly.
Freeing unused kernel memory: 220k freed
Write protecting the kernel text: 2792k
Write protecting the kernel read-only data: 788k
INIT: version 2.86 booting

W2CBW003 Overview

The W2CBW003 module integrates both WiFi and Bluetooth functionality in a 12mm x 12mm x 1.6mm package. The WiFi portion is based on the Marvell 88W8686, and the Bluetooth on the CSR BC04. Both of these components are well supported by Open Source software. Some other features include:

• Separate Antennas for WiFi and BT.
• ROHS compliant
• Single Supply at 3.3V
• 802.11g support (54Mbps)
• Both SPI and SDIO interfaces for WiFi
• UART interface for BT
• PCM audio interface for BT

Pictures of the module and the demo board are shown below.

Evaluation System Configuration

The test software included with the Wi2Wi eval board is for a Windows PC, is provided in binary format only, and was not used in this review. For this review, I plugged the W2CBW003 demo board into a Compulab cm-x270 board (PXA270 cpu) running a 2.6.29-rc7 Linux kernel and a recent OpenEmbedded Angstrom distribution. With the exception of the Marvell Wifi Firmware, all software in this setup is Open Source and is available in the Linux kernel, and as packages in the OpenEmbedded Project.

When booting the kernel, you will see the following messages:

mmc0: new SDIO card at address 0001
libertas_sdio mmc0:0001:1: firmware: requesting sd8686_helper.bin
libertas_sdio mmc0:0001:1: firmware: requesting sd8686.bin
libertas: 00:19:88:06:0b:2e, fw 9.70.3p25, cap 0x00000303
eth2 (libertas_sdio): not using net_device_ops yet
libertas: PREP_CMD: command 0x00a3 failed: 2
libertas: PREP_CMD: command 0x00a3 failed: 2
libertas: eth2: Marvell WLAN 802.11 adapter

The “command 0×00a3 failed” messages are harmless, and have to do with features that are not supported. After the system boots, you will now see a new ethX network device:

root@cm-x270:~# ifconfig -a
...
eth2      Link encap:Ethernet  HWaddr 00:19:88:06:0B:2E
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:65902 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1758 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:13550002 (12.9 MiB)  TX bytes:251627 (245.7 KiB)

The “iwlist eth2 scanning” command will list available access points.

Connecting to Open WiFi Networks

A connection to an open wifi network can be accomplished by placing the following in /etc/network/interfaces:

/etc/network/interfaces:
iface eth1 inet dhcp
    wireless_mode managed
    wireless_essid any

And now execute “ifup eth2″:

root@cm-x270:~# ifup eth2
udhcpc (v1.13.2) started
run-parts: /etc/udhcpc.d/00avahi-autoipd exited with code 1
Sending discover...
Sending select for 192.168.1.115...
Lease of 192.168.1.115 obtained, lease time 86400
run-parts: /etc/udhcpc.d/00avahi-autoipd exited with code 1
adding dns 208.67.222.222
adding dns 208.67.220.220

root@cm-x270:~# iwlist eth2
iwlist: unknown command `eth2' (check 'iwlist --help').
root@cm-x270:~# iwconfig eth2
eth2      IEEE 802.11b/g  ESSID:"bec3"
          Mode:Managed  Frequency:2.437 GHz  Access Point: 00:18:39:C1:AD:4A
          Bit Rate:1 Mb/s   Tx-Power=13 dBm
          Retry short limit:8   RTS thr=2347 B   Fragment thr=2346 B
          Encryption key:off
          Power Management:off
          Link Quality=84/100  Signal level=-37 dBm  Noise level=-87 dBm
          Rx invalid nwid:0  Rx invalid crypt:14707457  Rx invalid frag:0
          Tx excessive retries:58  Invalid misc:3   Missed beacon:0

WPA Secured WiFi Networks

The OpenEmbedded console image includes the WPA Supplicant packages which is used to manage wireless connections to secured networks. To set up the system for WPA encryption, modify the following files:

/etc/network/interfaces:
iface eth2 inet dhcp
   wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf
   wpa-driver wext

/etc/wpa_supplicant/wpa_supplicant.conf:
ctrl_interface=/var/run/wpa_supplicant
ctrl_interface_group=0
eapol_version=1
ap_scan=1
fast_reauth=1

network={
      ssid="bec"
      proto=WPA2
      key_mgmt=WPA-PSK
      pairwise=CCMP TKIP
      group=CCMP TKIP
      scan_ssid=1
      psk="ascii passphrase"
      priority=10
}

Then, execute “ifup eth2″, and you should see something like:

root@cm-x270:~# ifup eth2
WPA: Configuring Interface
udhcpc (v1.13.2) started
run-parts: /etc/udhcpc.d/00avahi-autoipd exited with code 1
Sending discover...
Sending select for 192.168.1.115...
Lease of 192.168.1.115 obtained, lease time 86400
run-parts: /etc/udhcpc.d/00avahi-autoipd exited with code 1
adding dns 208.67.222.222
adding dns 208.67.220.220

root@cm-x270:~# iwconfig eth2
eth2      IEEE 802.11b/g  ESSID:"bec"
          Mode:Managed  Frequency:2.412 GHz  Access Point: 00:40:10:10:00:03
          Bit Rate:1 Mb/s   Tx-Power=13 dBm
          Retry short limit:8   RTS thr=2347 B   Fragment thr=2346 B
          Encryption key:<too big>   Security mode:open
          Power Management:off
          Link Quality=64/100  Signal level=-68 dBm  Noise level=-89 dBm
          Rx invalid nwid:0  Rx invalid crypt:-1463809279  Rx invalid frag:0
          Tx excessive retries:22524  Invalid misc:3   Missed beacon:0

Other Observations

With the above networks, the bec access point was much further away than the bec3 AP, so you will notice the difference in link quality. “iwlist eth2 rate” can be used to list the current connection rate. When the network is idle, it sits at 1Mb/s. When downloading a large file, it will climb to 36 or 54Mb/s, depending on link quality.

Production Issues

The review demonstrates that it is fairly simple to set up a demo quality Embedded Linux system with WiFi. Some of the issues that would likely need addressed for a production system include link management, test software for certification, and power management.

There are several ways to programatically control WPA Supplicant including linking to the wpa supplicant control interface, or using DBus. There are several WiFi management applications available including Gnome NetworkManager (used in desktop systems), and connman which seems a little better suited for embedded systems.

Unless you use a completely pre-certified module + antenna solution, you will likely need to do some level of agency certification. As many products use a custom antenna, this is an important issue to consider and plan for. While Marvell provides test software and firmware, it will likely require some work, as their software is designed to be used with their in-house drivers rather than the libertas driver which is available with modern kernels.

Also, if you want to minimize the power usage, some work will be required to figure out the power modes supported, and how to implement/control them. With this module, the Wifi and BT portions run off the same crystal, so if you only want the BT active, you will need to actively power manage the Wifi portion to a low power state instead of completely disabling it.

Conclusion

TheW2CBW003 module is an attractive solution for products that need WiFi functionality. With the availability of modules like this, and mainstream open source software, the technology is available to about anyone, including low volume manufacturers. Standard interfaces such as SDIO make it possible to interface this module with about any modern ARM processor that can run Linux. Software support in the Linux kernel, wpa supplicant, and the Linux wireless tools provide the needed software support to implement a very complex system with relatively little effort.