I often write and talk about how useful Simics is for debugging concurrency bugs and glitches in multithreaded and multicore systems. Recently, we had a case where we proved this on a very complex application, namely Simics itself. This nicely demonstrated both the recursive completeness of Simics, and its usefulness for conquering tricky bugs in complex software.

The beginning of this story is a bug in Simics, triggered by a certain Simics configuration. The Simics target is a Power Architecture machine, running some bare-metal test code testing the processor simulation. Occasionally, this setup would crash Simics, due to some bug in Simics or the models. It was a difficult bug to track down, as it only happened in one run out of 50 or so. When attaching a debugger to try to diagnose it, it invariably did not happen (a classic Heisenbug).

Simics is the perfect tool to diagnose these kinds of issues, but in order to do that, we had to get the failing program into Simics. I.e., running Simics on Simics. The first step was to create a duplicate of the development host, inside of Simics. This was fairly simple, just a matter of installing a Fedora 16 standard Linux on an 8-core Intel target. Once the Linux was installed and booted, a checkpoint of the system was taken.

Next, the development code tree from the host was packaged up as a tar file and put on a DVD image file. Simics was started from the checkpoint of the booted target system, and the DVD image inserted into the virtual DVD drive and mounted by the Fedora Linux running on Simics. The tar file was copied to the file system on the target, and unpacked. A new checkpoint was taken after the Simics installation was thus completed and Simics could run on Simics. The result at this point was a completely self-contained, controllable, and repeatable environment.

The screenshot below shows Simics running on Simics, with the same desktop wallpaper being used for both the host and outer Simics Fedora system:

The next step was to replicate the bug inside of Simics. To this end, a shell command was used that repeatedly ran the inner Simics until the bug hit (obviously, this session was started from the checkpoint after the Simics installation).

The result was this setup, ready to run Simics until the bug hit:

To recap, we have Simics running on Simics. The “inner Simics” is configured with the Power Architecture setup that resulted in a crash on the host, and the “outer Simics” is running Fedora 16, providing a virtual replica of the development host (but inside of Simics).  

Additional scripting in the outer Simics was used to make the search for and replication of the bug more efficient.

• The Simics script varied the time slices given to the processors in the IA target system. This caused greater variation in scheduling of concurrent processes and threads in the Simics-simulated Fedora 16 OS, which in turn helped provoke the bug so that it appeared faster (after fewer runs of the inner Simics).
• A checkpoint was taken after the inner Simics had been started and the timing variation applied to the IA processors – but before it had started executing the test case. This meant that a checkpoint would be available that led straight to the bug, with no need to do any warm-up of the target or particular configuration of Simics. The checkpoint would in effect be a self-contained bug report for the issue.
• A magic instruction (blue star) was planted in the segfault handler of the inner Simics, making it very simple to catch the crash of the inner Simics. Often, using a magic instruction like this is simpler than trying to capture the right page fault or putting a breakpoint at the right place. A magic instruction is a live marker in the code that will always trigger, regardless of debug information or OS awareness. Furthermore, it has no overhead until it hits.

Eventually, after some 20 runs of the inner Simics, the bug was triggered. Thanks to the checkpoint and Simics repeatability, reproducing the bug is trivial. The Simics crash could now be reproduced any number of times, and it was time to go debug and figure out why Simics crash. An occasional Heisenbug had been converted into a 100% reproducible Bohrbug.

The first step of debugging was to figure out the mapping of the many dynamically loaded modules in the inner Simics. This was done by running the outer Simics and sending a Ctrl-Z to the Fedora shell, pausing the inner Simics. Then, the /proc file system on the Fedora Linux running on Simics was interrogated to find the load addresses. Since the checkpoint was taken after Simics was started, we know that this is the mapping in the software setup found in the checkpoint. Every time the checkpoint is opened, the same mapping applies – so the information was saved and used to setup symbolic debug information for the Simics modules used.

The next step of debugging was to open the checkpoint again, turn on reverse execution, and run forward until the magic instruction hit. Then, OS awareness was used to back up until the last time that the inner Simics was running prior to hitting the segfault handler. This placed the execution of the outer Simics at the precise instruction where the inner Simics crashed.

It turned out that Simics was trying to execute code in a location (BCDE) where no code was to be found.

Stepping back one instruction led to a JMP instruction to the location BCDE.

So where did this JMP BCDE come from? It was clearly not part of the static code of Simics, but something that was generated at run time by Simics itself (Simics contains a JIT compiler and thus modifying running code at run time is perfectly expected behavior).

To find out how the bad JMP was created, a memory write breakpoint was put on the instruction (JMP BCDE), and execution reversed. Simics stopped at the point where the “JMP” part of the instruction was written to memory. Doing a stack back trace at this point showed the code that was trying to write a five-byte “JMP XYZQ” instruction into the JIT-generated code stream. Since the breakpoint had hit on the write of the byte containing the JMP instruction code, indicating that the other four bytes (containing the actual JMP target location of XYZQ) were yet to be written when the instruction got executed and Simics crashed.

Stepping forward (on the processor) revealed that a thread switch happened in the inner Simics, and that the incoming thread immediately executed the five-byte JMP instruction, such as it was. Since only the JMP byte had been written, this was a jump to location BCDE, rather than the intended XYZQ (it would also have been OK to execute the original ABDCE code). Thus, the issue was diagnosed to be a read-write race condition, with the twist that the read was an execution of the memory as code and the write a regular data write. As soon as the problem was identified it was of course very easy to fix.

With the same setup, another race condition in Simics was also found and fixed, involving the more common case of multiple concurrent threads updating and reading a shared data structure without sufficient synchronization.

In summary, this blog post has described one instance where Simics was used to find and fix concurrency bugs in a real-world complex software system called Simics. The key to the success was the repeatability that Simics provides, even for timing-related occasional events, along with checkpoints, scripting, reverse execution, and debug facilities.

In a recent Simics seminar, I was asked about repeatability, variability, determinism and Simics. This is a question that comes up almost every time I present about Simics in front of an audience with testing experience. The people asking the question intuitively think that determinism is a bad thing - since it sounds like it will limit the execution scenarios that will be explored in testing. For a tester, variation is a good thing. However, determinism is not in conflict with variation. And I think I found a perfect illustration of this in a setting that is a bit more accessible and easy to understand than computer simulators. In a computer game, Bad Piggies, from Rovio.  

Bad Piggies is a game based on a physics simulation (not particularly realistic, but still reasonably consistent with everyday experience), where you put together strange contraptions to guide an endearing little pig pilot from the starting point to a goal. What struck me about the game is its incredible sensitivity to inputs, while still being 100% deterministic. To me, this is absolutely analogous to how Simics works (and I guess drawing that analogy indicates a pretty warped mind too).

There are two types on inputs in the game that matter: the (static) configuration of the vehicular contraption that you build, and the interactive inputs you provide during a level to turn on engines, pop balloons, or fire off boxes of TNT. An utterly minor change in the timing of an action can mean the difference between success and failure. On the static side, moving the center of gravity of a vehicle by putting the pilot pig somewhere else can have a huge impact on the behavior. There are levels where all you do is just move the center of gravity around to see which location finally hits the perfect balance.

Still, the game is clearly deterministic. This is particularly obvious in levels where no interactive input is needed. All you do is put the vehicle together and watch it go (roll down a hill, float away on a balloon, or throw the pig by exploding a box of TNT underneath it). In these cases, each game plays out the exact same way, as the physics engine is intentionally and precisely crafted to be deterministic. Whether I play an HD version on an Android tablet or a regular version on a iPod touch, the outcome is the same and the way you clear the levels identical. This is strong determinism.

Is not predetermined, however - when you put a new vehicle together and let it go, you really cannot predict what is going to happen. You might have an idea in mind, and you might hope it does the right thing. But before you play it through, you do not know. It most likely fails on the first attempt, and you go back and tweak the design (in computer programming, this is known as debugging). And try again. Each time, something different will most surely happen, since the setup input is different. There is huge variability, and if we see the game as a way to test the contraptions we put together, there is infinite room for variation on top of the fundamentally deterministic game engine.

Imagine how this would be if the game was really nondeterministic. It would be pretty much unplayable. It would turn into a die-rolling exercise, where you would create the exact same initial conditions,  let it run, and hope that on some run, you would be lucky and get to the goal. Not my idea of fun, exactly. Determinism is clearly good and helpful for testing and development, as long as there is also variability and sensitivity to inputs. But if inputs do not changed, you do not want behavior to change. Normally, computer system are indeed random, and since this is a fact of life, many people make lemonade out of lemons and think of this randomness as a good tool for testing. But when you can do it, avoiding randomness is clearly superior as a way to approach systematic testing.  And as a way to fly computer game pigs to the goal.

Simics works in the same way as the Bad Piggies game. For a static test case like a non-interactive boot of an OS, it will play out the same each time you run it (provided no configuration changes are made). Change the target OS image, and you will get something different. Change the number of processors or the size of memory, and something different will happen. Change the host machine (the machine on which Simics runs), but not the target, and you will get the same result.

If we add uncontrolled interactive inputs to the mix, we will get different results each time. We can make interactive inputs deterministic by scripting or recording them. And given deterministic inputs, the execution will be the same.  This is how Simics achieves perfect repeatability.  Not by limiting what executes and how, but by controlling inputs to a system so that any particular variation can be reproduced.

Taking this one step further, a simulator also allows you to systematically explore issues by programming input variations into scripts. At one extreme, this is fault injection, but even just automatically varying inputs to a program within legal bounds can be very interesting. For example, Simics has been used to provoke OS bugs, or test how a program runs as the number of processor cores in a system is varied.

Repeatability and determinism just mean that we can repeat any execution we happen to chance upon - not that we limit the space that can be explored in testing. For any tester who ever had to try to replicate and report a failed test case that only was seen once, repeatability is a wonderful thing indeed.

Recently, we introduced a synthetic simulation-only Simics target machine called QSP (Quick-Start Platform) and it's included in the latest version of Wind River Simics. The idea of QSP is to give every user a useful Simics target that allows them to immediately start using Simics and begin reaping the benefits of simulation for software development. In this blog post, we'll be taking a look under the hood of QSP to see how it works.

The idea behind QSP is to design a piece of virtual-only hardware that is as simple as possible, while still running real operating systems similar to how they work on ordinary hardware.

This is not a new idea. The Simics team and other simulation providers have done similar things in the past - but never with the bottom-up design done with Simics QSP. Usually, designs have started with a system controller and core complex from an existing platform, and then using synthetic devices for various IO. Looking back at my own work history, I ran a student project back in 2006 to port SMP Linux onto a cut-down Simics PowerPC platform. More recently, some colleagues of mine have ported VxWorks to various simple uniprocessor Simics models that basically just contained a processor, memory, serial port, and interrupt controller (modeled closely to existing PICs). The VxWorks team has used a highly-simplified platform in their OS porting to new architectures.

What is truly new with Simics QSP is the scope and depth of the design effort. Instead of cutting down an existing hardware model to something less complex, we started from a blank page and designed each device in the system to be as simple and easy to use as possible. We also wanted the devices to be flexible and scalable and not impose unnecessary accidental limitations on the system setup. After providing the specifications to engineering, I expected to get back a platform with at most a handful of simple devices. However, it turned out that a modern OS requires a surprising number of hardware services in order to work, and the resulting initial QSP design contains nine types of devices in addition to the processor cores.

The picture above shows a block diagram of the QSP that we are now shipping. We can see that it contains quite a few pieces of hardware. To make an OS run, we need a processor. A processor needs RAM to store code and data. To get periodic interrupts, it needs a timer (even though the basic needs are taken care of by internal features within the Power Architecture, timers are needed for ARM and possibly other architectures). To handle interupts from devices and inter-processor interrupts in a multi-core design, you need an interrupt controller. To start non-primary processors in a multi-core system, you need a system controller. A real-time clock (RTC) is needed for sane dates, especially important when using disks to store a large file system. 

Apart from this core set of devices, serial, Ethernet, and LEDs are needed to provide input and output facilities. Disks and flash disks were added to support large file systems and flash-based applications. Not all of the I/O units will be present in all configurations, as that is up to the user to decide.

For added flexibility, we made sure that the number of processors and important functional devices can be varied in case some application needs several of these elements.  It is a virtual platform, and not subject to the physical limitations of real hardware. We wanted to make sure this benefit was passed on to our users. This is where writing both the BSP and the target hardware was most beneficial. Most real multicore hardware designs have arbitrary limitations in system controller registers or interrupt controller designs that limit how many cores can be used, even when a simulator should allow you to use any number of cores. By designing a very scalable hardware interface and the BSP to drive it, we can scale the system to theoretically any number of cores. The current hard stop is at 128, but this might change in the future.

If we look at the design of the devices themselves, it turned out that we needed more details than what I expected at the start of the project.

For instance, Ethernet requires both receive and transmit interrupts to be present.  In an ideal world, transmit would seem unncessary, right?  But we have to model finite bandwidth, as infinite bandwidth is very confusing to existing software, and that means that packets can take time to transmit, and thus transmit-completed interrupts are interrupts (or equivalently, a transmit complete flag along with a polling driver).  We also have descriptor tables in RAM rather than a simple buffer inside the device, as that is what the existing driver stacks for networking would expect. 

Another surprise to me was the fact that simulating (NOR) flash memory is best done by actually implementing a flash command set.  File systems and flash driver stacks expect a flash to follow a certain pattern of operation, such as requiring several writes to turn into write mode.  Just modeling flash as persistent RAM memory does not work, even though it seems the obvious thing to do.

Overall, as we worked on QSP, we discovered just how much the structure of operating systems is shaped by the design of contemporary hardware interfaces.  The BSP or HAL (hardware abstraction layer) in an OS like Linux or VxWorks does not really allow any hardware to run underneath the OS - it expects certain behavioral patterns that are not necessarily what one might think of as the simplest way to express a functionality in hardware. Indeed, if we had made QSP devices more lightweight, the BSP side of the QSP would have been much more complex and would have required much deeper changes to the OS kernels and device driver structures.

Ultimately, what we've released is a solid piece of virtual hardware that allows application development and other system development tasks to progress without the need to model or use any particular hardware model. Obviously, many tasks that Simics is used for does require a model of a specific hardware board, and in those cases QSP is intended as a companion, not a replacement, for traditional Simics models.

For additional information from Wind River, visit us on Facebook.

We just released a new Simics feature, the QSP (Quick-Start Platform). This is a synthetic simulation-only Simics target machine that is included with the Simics base product. QSP provides every user with a useful Simics target that allows you to immediately start using Simics and reap the benefits of simulation for software development, without waiting for a target model to be ready. It provides a quick way to realize our old Simics slogan: "Resistance is Futile, you will be Simulated."

With QSP, we provide a shortcut by offering a ready-to-use integration of a model and target software stack that can be used to do application-level development from the moment a user receives Simics. QSP also makes simulation features like checkpointing, reverse execution, unintrusive debug, repeatability, automation, and scripting easily available for application developers without having to first setup an OS and select a target machine.

A key use case for QSP is developing user-level application software. QSP presents an application developer with an operating system on top of a certain architecture, with a set of functions available in the hardware and supported by the software stack. All application software is compiled to the target architecture, not to the host, and application binaries from a real target should run on the QSP, provided that the OS APIs in the QSP are sufficient. 

Let's go over some examples of how QSP can be configured and used.

Our first example will be Linux on a dual-core Power Architecture machine with Ethernet and serial, as shown in the picture below.

This type of target would be used to run a networked application. By instantiating multiple QSPs in a single simulation, the network application can be tested with communication between the boards.  It does not matter which Ethernet adapter is used or the precise nature of the board, as long as the application can run with two threads tied to a core each and communicate over Ethernet to its peer.

Thanks to Simics repeatability and control capabilities, network scenarios involving all the machines can be perfectly replicated, analyzed, and debugged, across the entire system. An instantaneous system state, including traffic sessions in progress, can be saved as Simics checkpoints, allowing the developer to come back later to a particular point in the execution. Checkpoints can also be used to capture and communicate bugs between team members, replacing the tedious process of describing system setups in bug reporting systems. The Simics system-level debugger allows source-code debugging on all target systems at once, at the application level, with synchronous stepping of the entire system, not just an application at a time. Reverse execution and reverse debugging can be applied to the entire system, following the flow of control and packets between the machines backwards in time. Simics Analyzer makes it easy to see how programs on both sides of the network connection wait for data and get activated.

Another crucial function is using a flash disk. In the picture above, we see an ARM machine with a flash disk running VxWorks.  A serial setup is added to have an interactive connection to the target. This setup would allow a programmer to develop and test software interacting with a flash-based file system, without caring about the make and model of the flash memory unit, or how to configure a driver stack (for a real flash-based system you tend to need to know the type of the flash to correctly configure the OS to drive it, while with the QSP, there is just a single type of flash and the driver configuration is given). The QSP flash disk is also much faster than a real one, as there is no need to wait for contents to stabilize after a write.

In Simics, saving and restoring multiple versions of the flash disk contents is trivial. Since disk changes are handled as diffs, any bad operations can be easily undone by restarting the simulation session (which discards all changes). They can also be interactively undone by using reverse execution in the simulation session.

QSP was designed with multicore in mind, and it offers a more flexible target in terms of the number of cores available in hardware than any physical ARM or Power Architecture system available today (with the exception of IBM pSeries servers). As illustrated in the picture above, this can be used to test the scaling of and debug the behavior of multithreaded multicore software, easily varying core counts to see what happens. With Simics repeatability and reverse execution and reverse debugging, concurrency bugs are much easier to find and fix than when using physical hardware.

At this point, it is worth remembering that application binaries developed on a QSP will run on real machines too.  Indeed, a key value of using QSP for application development is that the same cross-build tools and the same OS API and ABI are used on the physical target. Thus, the application binaries compiled for and tested on a QSP will run unchanged on a physical target (with the same OS and architecture). If the application uses third-party binary-only libraries, these can be used as-is.  In this way, QSP reduces the distance between the development environment and the production environment, compared to using host-based development. 

There is a real and relevant difference between Linux on Power and Linux on ARM and Linux on an x86 host. With QSP you find the target-related issues during application development rather than much later at hardware-software integration. If Simics is also used to model a real board before physical hardware appears, integration issues can even be found and resolved before the actual target exists.

Above, we show an example of how QSP can be used to shorten the development time and time to market for a system based on custom hardware. The eventual goal of the project is to have the real OS running on a model of the actual physical hardware board used in a system, and after that on the physical board itself. With QSP configured with the relevant functions, application development can start concurrently with the development of the model of the physical platform. Once the platform model is finished and the OS runs on it, the application is available for execution and integration test. Without QSP, we would have had to wait until the virtual platform was complete enough to use before doing application development.

QSP is also a great teaching platform. It is an obvious candidate for setups like the one presented in my previous blog on teaching networking with Simics - you want plenty of Ethernet ports and a certain OS, along with ease of use and ease of portability across target architectures. The QSP design is a natural match for such use cases. QSP is also perfect for university-level OS courses. It reminds me of a course that I used to teach where students were given the task to develop an OS kernel from scratch. In QSP, you have all the relevant interesting behaviors such as interrupts from devices, timers, multicore, etc., but none of the arbitrary complexities of real hardware platforms. For example, configuring an i8254 interrupt controller to get serial port interrupts is pretty complex, distracting students from the core subject material of how an OS kernel works.

This blog post has really only scratched the surface of what QSP can be used for. It offers a very quick way to get going with Simics simulation, and to apply the benefits of simulation for software developers for all application development - regardless of whether there is an actual platform model available for the actual target hardware.

For additional information from Wind River, visit us on Facebook.

Wind River Education Services provides user training for a variety of topics, including Wind River operating systems and tools, as well as more general topics like networking. Training always includes hands-on labs, which can complicate logistics for training sessions. Shipping boards and configuring networks is time-consuming and error-prone. For that reason, we are looking into using Simics as an alternative to physical hardware to streamline training logistics. It also makes it simple to encapsulate configurations, while maintaining the reality of cross-development. In this blog post, we will look at how Simics is used as a tool to facilitate network labs in the context of education.

To make an interesting lab, we need a few machines, connected over a few networks. The starting setup currently used is shown below:

We have eight machines inside the Simics process, four used as routers ("Router N" in the picture, with yellow borders) and four used as end points for communications ("Target N" in the picture). These eight machines are connected using six Ethernet network links. In the real world, each such network link would have been an Ethernet switch, for a total of fourteen physical units.

One of the target machines connects out to the host to allow host-based software such as Wind River Workbench to control and monitor the operations. To monitor the traffic, we attach live Wireshark capture points to two of the simulated networks. Each Wireshark capture point is connected to a Wireshark process on the host, which gets the packets flowing in the simulation and presents them just as if they had been captured from an Ethernet interface on the host. This lets students monitor and investigate the traffic flows in the labs. Compared to using a regular network sniffer on a host interface, the Simics Wireshark trace does not require any administrative rights on the host - it is just a stream of data from one user-level process to another.

The target machines shown here are all Power Architecture-based Wind River SBC8548 boards. In their physical incarnation, these boards only expose two of the four Ethernet controllers on the MPC8548 SoC to the outside. In Simics, all four Ethernets are available for connection, as making that change in a model is very simple. In this setup, most of the targets run VxWorks, but a few also run Linux to make the network more heterogeneous and realistic. The software stacks running on the targets are complete OS stacks, the same as would be running had this lab been configured from physical boards and cables. This includes aspects like IPv4 and IPv6 network stacks, SNMP, routing protocol handlers, and higher-level software like web servers. This makes the labs very realistic, and allows us to teach how to use a particular real software stack if that is desired.

Since this setup is created by Simics scripts, it is very easy to change the topology or nature of the machines. For example, adding in more machines and networks, changing their architecture from the Freescale MPC8548 to another SoC or another instruction set architecture altogether. We could also mix different architectures in the setup to emulate the effects of a mixed-endian network or mixed-OS network.

With this setup, many different labs exercises can be performed, such as:

• Policy based routing from the Linux targets to the VxWorks targets, making routing decisions based on network protocols used for traffic rather than IP addresses. For example, TCP could be routed over "Router 6" and UDP over "Router 5", by making routing changes on "Router 7".  
• Dynamic routing protocols between subnets, by artificially disconnecting the link between "Router 7" and "Router 5". Such a network disconnect is trivial to achieve in Simics.
• Watching the ARP broadcast distribution in the system when one of the Linux targets is trying to reach one of the VxWorks targets for the first time. 
• Injecting packet streams from the host, running the pktgen utility, and inserting them in the Ethernet port of "Router 7" that is connected to the host machine. Other external traffic sources could also be used, such as IXIA and Agilent traffic generators. 
• Simulating the effects of long-distance WAN links by introducing jitter, delays, and dropped packets on the network traffic on certain links. Simics can be used to provide a model of the world, not just a local lab, to add realism to the training.

With Simics, you get a "network in a box" that is ready to go at any training session, anywhere, without needing anything else but a laptop and a virtual machine image with Simics and the lab setups installed. This is using Simics as a tool to achieve the goal of teaching networking - users do not need to learn much about Simics at all, they can just start the session and get going on the labs with minimal overhead. The Simics setup is stable and does not need to be double-checked as a physical setup would.

 In action, the setup can look like the above screenshot (using Simics 4.6 and the stand-alone GUI). Note the commands to set up network adapters and their IP addresses in the router serial consoles along the bottom (which is a task that is best scripted so that Simcis does it for you, although in a teaching setting it might be pedagogic to have the students do this manually once just to think about what they do). After this, a user would enter the routing commands manually to perform the configurations which are part of the labs.

For additional information from Wind River, visit us on Facebook.