As noted before, one common use case for Simics is doing fault injection.  Fault injection is typically added to an existing system without changing it, since the goal is to maintain a fundamentally fast simulator when not injecting faults.  There are a range of mechanisms available in Simics to implement fault injection. This blog post will take a look at two such mechanisms and how they are used in the context of injecting faults on a serial port.

We will look at two ways to induce bad behavior into a serial port in the system shown below:

First, we introduce faults in the interface between the serial port and the processor using it.  I.e., on the register bank of the serial port that the software on the system processor(s) use to send and receive data from the port.  This simulates a device with flaky interface circuitry, or a bus that would sometimes fail to correctly pass the data from the device to the processor.  It does not directly correspond to a faulty device, since the device operation is not affected (unless of course an error is injected on a transaction that targets the configuration or control registers of the device and have unintended side effects).

On the surface, this might seem trivial to implement: just pass all transaction from the main memory map through an intercept module, and have that module corrupt the transactions as it sees fit.  However, this would burden all transaction with a cost and remove the ability of the simulator to cache values and optimize accesses to memory and other devices which are not affected by the fault injection. Instead, in Simics, we can introduce a level of indirection and intercept only accesses to the serial device.

As shown in the picture below, we do this by introducing a private little memory map for the serial device. The mapping in the main memory map is set to point to this new memory map, and it in turn maps the serial device.  The new memory map has an intercept module attached to it which can see and modify both read and write transactions.

This target system modification can be done in a few lines of Simics scripting, even in the middle of a simulation. Simics allows arbitrary reconfiguration of the simulation system at runtime, and the changing of the memory map to point to something new and the introduction of a new memory map and fault injection module can both be done and undone at runtime.  Using a module instead of just a script for the injection work makes the fault injection more robust, and ensures that the injection of faults is part of checkpoints and allow for reverse execution.  Any Simics user can create such modules themselves, using the Simics Extension Builder product.

If we program the fault injector to replace some output characters with other characters, we get something like this:

Note how the output is suddenly corrupted with a bit of L337-style text.  This is achieved by changing the values written to the serial port transmission register, on their way from the processor to the device. The processor is not affected, only the data in flight.

A second fault injection pattern hooks into the back end of the serial device. The serial device has a front end that handles the registers facing the processor, and a back end that handles the transmission and reception of characters (transactions) on the serial line.  The front end and back end are tied together by the core device functionality.

The back end implementation has the serial device modeling talking to a model of a serial console (or other hardware attached to a serial port) using the Simics "serial interface".  Note that the serial device does not interact directly with a user, but rather it expects some other model in the simulation to provide this service.  In this example, it is a serial console, but it could just as well be another virtual machine. It can also be a fault injection module that puts itself into the path of the data, as shown below:

Unlike the previous case, here we put the fault injection module on the actual data path.  The fault injection module actually receives all data and passes it on to the other side of the connection. It is not selective in the same way as an intercept module. Just like with the intercept module, making this an explicit part of the simulation enables checkpointing and reverse execution to work during fault injection.

In the code, each output from the serial device calls a function in the serial interface of the fault injection module, which in turn calls the same function in the serial interface on the serial console. If data is just forwarded, neither the serial device nor the serial console sees anything different.  However, the fault injector can do anything with the data as part of passing it on, including changing data and dropping data. 

It is also possible to introduce spurious data transmissions, simulating random noise on the line that looks like actual data transmissions to the devices in the system.  Note that since Simics does not actually simulate the analog encoding on the serial line, a model of random works by generating entire characters - essentially simulating a case where noise is interpreted as valid data by the receiver circuitry.  The intercept module used in the previous example could also create transactions, but would have to make use of additional interfaces to do it (it would essentially perform its own independent memory accesses).

 

For additional information from Wind River, visit http://www.facebook.com/WindRiverSystems.  

Emulating old computer gaming systems is a popular consumer-market application of simulation technology similar to Wind River Simics. There are a large number of emulators out there, emulating everything from old arcade games to the home computers of the 1980's to gaming consoles.  These emulators might seem quite similar to Simics on the surface, but in practice, the simulation technology employed is quite different.  You will never see Simics simulate an old Super NES running Super Mario Bros. 

It all comes down to the level of timing accuracy required.

To accurately run games written to run well on the very limited hardware of the 1980s, you need a fairly detailed simulation of the hardware circuits and their timing.  Game programming on 1980's and early 1990's was a matter of writing code that was very tailored to the hardware and intimately depended on the hardware timing.  A typical trick applied on many system was to change the color palette, display mode settings, or even the contents of the screen buffer during the time that it was being redisplayed on the TV. Since every unit shipped was identical, this was a viable strategy.  I wrote code like this myself on my old ZX Spectrum, achieving increased vertical color resolution, by changing the
color data in the graphics display while the graphics "system" was sending display data to the attached TV.

ArsTechnica ran a very interesting article in August of 2011, about the most accurate emulation yet of the Super Nintendo Entertainment System (SNES) (or Super Famicon as it was known in Japan).  The article describes how successive generations of SNES emulators have got closer and closer to the actual hardware timing.  The timing that needs to be simulated can be very detailed indeed, such as the interleaving of bus accesses between different hardware units.  At the same time, hardware requirements for running the emulator has increased hundredfold from the first emulators from 1997 until today. Precision costs compute power, and this is essentially about using 100 times more computation cycles on the host to run the target system just as fast.

As described in the article, the latest "BSNES" emulator is about 10 times more resource-intense than the previous "ZSNES" simulator, for an increase in accuracy of 10.  That is actually very impressive, usually precision tends to be exponential in cost.  However, the really interesting statistic is that this only gives you a few more playable games.  ZSNES manages to run about 95% of all games, for a much lower cost in computation and implementation. Only in a few cases is the additional accuracy of BSNES absolutely needded in order to correctly run games or render graphics effects correctly.

This is all very interesting, but in which way is the emulation of old video games relevant for what we do today with Simics and virtual platforms in the embeddded systems field? It offers a neat illustration of the tradeoff between speed of execution and level of detail (and accuracy) of the virtual platform model.  Simics is not trying to be BSNES, it is much more like ZSNES or its predecessors. 

Simics is designed to emphasize speed of execution over detail of model. This is a deliberate choice, and it has proven critical in enabling the usage that Simics is seeing today.  By being fast enough to real-world software workloads in a reasonable time, Simics is acceptable to software developers as a daily tool. Interestingly, this was echoed in a DAC 2011 panel on ESL - without sufficient speed, software developers will not use a virtual platform. It is a crucial enabler.

The trade-off is pretty drastic: adding just a little detail to a simulator can quickly cut your speed down by a factor of ten. The diagram below has been shown in any number of variants in any number of venues, and the crucial fact that the curve is not straight. It drops very quickly initially as you go from left to right.

The reason that building a fast simulator in the Simics style works is that current software does not rely on the physical system timing in the same way that old console games do. The hardware today is very variable in its timing (processor pipelines, multiple levels of cache, branch prediction, multiple processors competing for shared resources, etc., all conspire to make execution-time variable and unpredictable). With the proliferation of levels of software abstraction, code that is tightly coupled to the hardware is almost impossible to write in general.  Instead, hardware-software interfaces have become event-driven, using interrupts to signal completion, or using poll loops on status bits for handshakes.  Code is also supposed to live across several generations of hardware, discouraging coding that is too tightly dependent on the characteristics of any particular hardware platform.

This is much more simulation-friendly since there is no need to know exactly how long an operation would take on hardware, as long as the simulator latency is of the right order of magnitude.  For most use cases, adding more detail than that just makes the simulation run slower, limiting its usefulness for the majority of software developers. It also makes the development of the models more expensive, since building a very accurate and detailed model is many times more costly than building a good enough model.

There is certainly software that does depend on the precise timing of the hardware. In real-time-critical software like signal processing and low-level firmware, there is hardware that does have very predictable timing, and software that depends on this timing.  Still,
such software is but a small part of the overall body of software that we run on today's systems. When needed, you can often build small workarounds into a simulator to make the software run right - just like the many game-specific patches in ZSNES discussed in the ArsTechnica article.  This is overall much more efficient than always simulating at a high level of detail just to make the really picky software happy.

In our experience, it is more valuable to run all of the software at a sufficient level of detail, than to run a small part of the software with a very high level of detail.  And just like we see with the ZSNES simulator, you can cover a very large base of software without going into too much detail on the hardware side. 

I hope you now understand why Simics is not likely going to run Super Mario at the level of fidelity of BSNES - and why doing so really does not make sense for most uses of a virtual platform. But I still applaud and admire the effort and dedication that went into building BSNES, and the way in which this is preserving an important part of our digital heritage. 

 

While looking into our old SPARC server targets for our new SPARC legacy offering, I  opened up a Simics demo checkpoint that we have had around for a long time (at least eight years). This demo is an UltraSPARC II running Solaris 9, and it shows how web browsers looked around 2001.  I used real-network to connect the machine to the Internet, and tried the default webbrowser - Netscape 4.78, built in June of 2001.

Since the Javascript implementation in the browser is pretty poor, and its CSS support terrible, the main Wind River home page would not basically not render in any useful form. Our blog page at http://blogs.windriver.com/wind_river_blog/simics/ does render - but maybe not in the expected way:

As a friend of mine commented on Facebook - is the problem that there are too many Jakobs there?

For all its shortcomings, we also have to admit just how revolutionary Netscape was. I still recall using version 1 to look at the few pages of the nascent world-wide-web when I was still a university undergraduate... And I am writing this blog using Firefox 6, which essentially is the latest in the line of development leading up from Netscape 4.

While it makes for some laughs to visit the past in this way, it can also be very useful.  If you have some old applications still living on an old machine like a Solaris 9 SPARC, what do you do?  Sometimes you can port the applications to more recent systems, but there are cases where it is just simpler, cheaper, and less risky to use a virtual machine to keep it running.  For example, if the source code is lost, the application is just a binary, the application depends on some device driver (which we can run on the virtual platform, since we run the complete OS stack), or even when there is source it has annoying word-length and endianness problems.

Note that if we would have liked to see a web site like it was back in 2002, we could have added a server to the virtual network to serve the pages like they looked back then. Preferably, running a contemporary web server on a contemporary OS to make it more "authentic" .

 

A while after I woke up my laptop today, I realized that the fan was pretty noisy. That typically indicates that the processor is working busy on something, and then I spotted Simics in the task bar. Checking in on that Simics showed that it was indeed running, and pausing it did lower the load on machine to the point that the fans calmed down. Turned out that while I was doing other things, Simics had been busily simulating away on one of my cores. Indeed, it had simulated what was almost a complete year.

As you can see on the right, the time indicator in the Simics control window shows that the target had executed for 7988 hours. That is close to 333 days, which is pretty close to a year.

This was achieved in about 4 hours and 20 minutes of host time. Simics was moving time forwards about 1800 times faster than real time. Just how can that be?

Simple - idle optimization, also known as hypersimulation. This old Power Architecture model was simply sitting at prompt, doing nothing. When a target does nothing, Simics can simply jump ahead in time to the next interesting event (a device interrupt or timer interrupt, typically) it knows about. The key to making this work is that all Simics devices that perform input or output talk to the outside world through Simics-provided virtual IO systems and never talk directly to the host. This gives the Simics simulation kernel the information it needs to perform good hypersimulation.

Having a target doing nothing might seem like a rare case, but in practice it is pretty common. For example, if you have a network or rack of target machines, only a few are typically active unless you manage to inject a very heavy load. On a multicore device, many programs only make use of one or a few cores. In both these cases, hypersimulation virtually makes the other machines and cores go away in terms of performance - almost all simulation time can be spent on the active units of the simulation, increasing performance many times compared to slogging through each and every cycle on each and every target processor.

There are also systems that are mostly idle per design. Sensor networks is one such example and long-running space missions another. Often, such systems wake up for an instant to check on the world, and then sleep for 99% or so of their time. In this case, hypersimulation enables you to speed up time 1000-fold and run through scenarios that just cannot be tested on physical hardware since it would take just as much real time - and who would sit around for five years watching a hardware mockup?

My blog posts about Simics tend to focus on how virtual platforms are used and what you can do with them. However, underlying all Simics use is the issue of getting a virtual platform in place. You need some kind of platform to run your software on in order to get the full benefit of Simics. Thus, creating virtual platforms is an important part of practical Simics use for many of our users.

We have been working with Simics for more than a decade, and we have learnt a lot about modeling along the way. The approach we have ended up with is a top-down methodology, where we start by investigating the target system as a whole from the perspective of the intended use cases, and then explore the details of each subsystem in order to determine how best to model it.  We then switch to bottom-up, applying an iterative modeling flow that delivers something useful as quickly as possible. During development, the goal is to always keep a useful (albeit probably incomplete) platform available at all times.

It is easy to get distracted by the details when creating a virtual platform.  If you start with a list of all the components of a system without considering the system context, you are likely to assume you need to model everything, as well as every function of everything. This is usually an overwhelming task, resulting in a protracted development project without clear focus or any chance of delivering something early. So, we need a better way. 

The method we typically use with Simics is to start by investigating how the system is going to be used, and which hardware units are involved in these use cases. In this way, we can often exclude parts of the target system from consideration. Certain subsystems or subsystem functions might not be needed at all, or needed only at a later stage of the project.  Even when modeling an SoC with the eventual goal of a complete model, it is possible to create a priority list and staging sequence that quickly gets something into the hands of users that applies to their initial use cases.

Once this analysis is complete, the existing library of Simics models might well cover a large part of what is needed. There could also be "near misses", where a small adjustment to an existing model is all that is needed. In this way, the ever-growing library of Simics models provides significant short-cuts towards a final system model.

Next, it is time to start building the new models needed to complete the target system. This process is run in an iterative, agile way. We strive to define a minimal subset of the target system that we start with, so that software can be run on the target as soon as possible. The outer loop of the modeling effort is the extension of the overall system model and the testing of software on it.  The inner loop is the creation and changing of individual device models, using both model-level unit tests and software tests to drive the development.

The net result is an flexible project where the virtual platform can be updated in reaction to changes to the target system design. Often, modeling projects are begun before the details of the eventual target hardware are completely known. As shown in the picture below, the result is a series of incremental model deliveries that represent the current design state of the hardware.

This modeling method applies regardless of whether the modeling being performed is pre-hardware (for a new chip that might not even be announced yet or a new board built on existing components), post-hardware for a board or system that exists right now, or post-obsolence for an old system that still needs to be maintenaned.

Modeling can be done by Wind River, Simics users themselves, or third-party services firms. All Simics users are equal in what they can model and their access to the Simics modeling tools and APIs.

To learn more about system modeling with Simics, please read our new white paper on creating virtual platforms with Simics.

Happy modeling! And remember - resistance is futile, you will be simulated!

Developing software is an adventure where you explore the unknown, one line of code at a time.  At some point, the program reaches the stage where you can actually run it and try it - and that's where the real fun begins.  The software will sometimes behave as expected, but more often than not it will not. It will do something else, or crash, or generally do the wrong thing.  It is very much like a toddler - you can rely on it to some extent, but you never know when it will decide to do something unexpected, funny, or just throw a fit over something that would have seemed inconsequential.

I have spent quite a bit of time recently with toddler stage of Simics 4.6 (prereleases and betas).  The end result is really good and qualifies as an adult specimen (as discussed previously on this blog), but there was a lot of entertaining (and frustrating) things that went by as Simics 4.6 was a toddler. All software developers have stories like these to tell, but I just wanted to share a few of the funnier ones and give a glimpse of just how much work it is to get something like Simics running.

My favorite issue was the strange memory allocation happening in Eclipse on 64-bit Windows host.  When I opened the Simics timeline view, the Eclipse process grew by exactly 4 GB plus 4 bytes.  When I closed it, the memory was deallocated, all nice and tidy.  It worked perfectly well on all other hosts.  It turns out that the reason was that our magnification slider (see picture) used a Windows widget that when running in 64-bit mode allocated exactly 4 bytes for each step of
the slider.  Since we wanted a slider that could handle a very large data set as the worst case, we had set it to allow 1 billion as maximum value.  Reducing this to a few hundred thousand allows
for sufficient resolution and causes no memory issues.  One still wonders why Microsoft made the widget behave that way.  I guess it just wasn't tested for anything but a volume control with 11 steps or something.  Once again, users violate the assumptions of software designers.

In another case, the debugger got a bit confused about endianness, and managed to display some variables as BE and others as LE. In the same view, for the same target program, at the same time.  Gave me a good laugh seeing it and diagnosing why some variables with the value "1" were shown as 16777216. I am a nerd at heart, I admit.  A bit like a child writing letters backwards as they learn to write.

The Simics toddler also was confused about directions sometimes. Pressing the "reverse" button instead made it go forward.  Pressing "stop" when it was stopped also made it run forward, which was a bit more surprising. All was easily sorted by wiring the right buttons to the right functions.

All software developers have many stories like these to tell, and it underscores the continued importance of creative and intensive testing.  Many issues do not show up until you start to put all the pieces together, and that aspect of integration testing is one of the main uses of Simics itself.

Simics 4.6 is now out, and I would like to share some of my initial impressions of the new Simics version.

The most visible new feature is probably the Simics source code debugger in Eclipse.  Source code debug might seem a yawn, but the way it has been integrated with Simics provides some new and unique abilities that at least I have never seen before in a debugger.  Most importantly, the debugger works on the system level with a single debug connection. 

An example is shown in the screenshot below, where we have two machines in the same Simics simulation - "Tuna" is a dual-core Power Architecture P2020 running VxWorks, and "Herring" is a dual-core Intel Architecture Pentium 4 running Wind River Linux. 

 

OS awareness shows the active processes, tasks, and threads, and you can see which threads are active on which processors. This is very convenient for multicore multithread debugging.  You can debug both from the hardware view and the software view of the system.  

Obviously, reverse execution is fully supported. We even highlight the last few instructions so the direction of execution is clear:

The debugger also allows the Simics backend to take control over the execution. This means that you can set breakpoints and write and run arbitrary scripts on the Simics command-line, and then have the debugger update its view whenever and wherever Simics stops the execution. All the existing advanced breakpoint features in Simics can be used, and the source code debugger just makes it easier to understand what is going on in the target software.  This gives us a debugger that combines the power of a virtual platform command-line debugger with the convienience and overview of an Eclipse visual debugger.

There is another small but very useful improvement to the Simics user interface. When you use Simics checkpoints (see for example blog and blog), you can now add text annotations to the checkpoint. This makes it so much easier for a user to understand what a checkpoint contains and why it was created. The checkpoint is also automatically annotated with who created it, when, and on which machine. Some of this metadata has been present in Simics checkpoints for the past few versions, but not exposed to users until now. In the screenshot below, you can see an example from one my demonstrations. It shows a checkpoint that captures an intermittent bug, and which was created a few years ago on an old machine I no longer use, and in Simics 4.2. It opens nicely in 4.6, and the bug behavior is exactly the same.

Without checkpoint annotations, you would have to provide this information in some README file or in an email, and it could easily lose its association with the checkpoint. With checkpoint annotations, Simics checkpoints become much easier to communicate between people and between teams. Checkpoints can also be handled by workflow system and checked into source code repositories with the metadata integrated and available for inspection.

Yet another user interface addition is the new system panel build kit, which lets you build models of typical board front panels as part of your Simics hardware models. It can also be used to build arbitrary dashboards to display information about a target, based on custom scripts and trace modules. The screenshot below shows a simple panel for a board with 10 LEDs, a configuration switch, and a reset button. We also see the software animating the LEDs running in the debug view, as the tLedAnimate task in VxWorks.

 
Apart from these highly visible additions, just like with every release of Simics, Simics 4.6 contains many small fixes and improvements. We are constantly trying to remove little annoyances and make the tool easier and more logical to use.

One example that I like particularly is the introduction of a general way to describe OS awareness nodes in Simics scripts. You can now specify that you want to track a certain thread by its OS process ID or by the name of the binary. You can ask OS awareness to return all tracked software units in a system that match a certain criteria (like "all threads inside the process called rule30 inside an OS called vxworks"). Previously, users had to manually scan the process lists to find what they needed, now they can essentially use Simics to do "find" over the entire simulation. For setups involving a hypervisor, multiple guest operating systems, and hundreds or thousands of active processes, this is a huge usability benefit. It also makes automating things with OS awareness much easier.

There are many other new and improved things in Simics 4.6, but I have to stop somewhere. However, there is one final thing that I have to include.  Simics now supports Windows 64-bit hosts. This is very beneficial to Simics users on Windows, as the better register model on 64-bit x86 provides an immediate speedup for the simulator. It also makes it practical to run much larger simulations on Windows host, thanks to removing the 32-bit addressing limit.