Mark Hermeling

Paper: Multicore and Multi-OS

2009-11-14T22:17:51-08:00

Travel has it's disadvantages (30 hours door to door Ottawa to Singapore in an old United 747 for example). But it also has it's advantages: here I find myself at the airport with free wireless and a couple of hours to kill. I should really go through my inbox, but scanning the web for interesting papers and blog articles is a much more fun way to spend this time :).

Which is how I ran into a paper talking about Multicore and Multi-OS as a pattern in embedded systems that is gaining a lot of attention and is clearly increasing in importance. (Registration required)

The paper is from my Dan Driscoll and Stephen Olsen from Mentor Graphics and there is a lot of good stuff in this. They talk about SMP and AMP from the hardware perspective (SMP when you have multiple cores of the same architecture, AMP when the core-architecture is different), Wind River typically talks about SMP and AMP from an OS perspective as we have more of a software focus. Different backgrounds, but something to keep in mind when you read up about these topics.

The article is well written and it talks about how expensive it can be to set up a multi-OS environment. I refer to this as AMP, with AMP from the software sense. You need expertise from an OS as well as from an processor architecture perspective. I spend a lot of time talking about this as well to customers and my opinion is that this time could be much better spent developing differentiating content.

They also talk about a hypervisor, but only from the perspective of core virtualization: running multiple different virtual machines on a single core. This certainly is one of the important usages of a hypervisor. However, the technology behind the hypervisor can make the multi-OS configuration a lot easier as well.

I usually talk about the fact that there are three distinct resources to virtualize: Cores, Memory and Devices. A Hypervisor typically virtualizes all three.

The Wind River Hypervisor can be configured to only virtualize Memory and Devices, to distinguish this we refer to this configuration of the virtualization layer as 'the Supervisor'. The Supervisor is very useful to make the configuration of multiple OSes on a single multicore chip easy with native performance. You can see this an Supervised AMP configuration, with AMP in the software sense (once more).

Now, while we are talking about the complexity of setting up the AMP configuration and how the hypervisor can assist in making that easier; a requirement for this to be true is that the hypervisor is available for the particular processor architecture and that it is well integrated with the operating systems of choice. My choice in operating systems differ from the authors of the article naturally, but that is besides the point.

The problem with not having the operating systems pre-integrated is that you simply shift the problem. Instead of integrating the OS to the hardware, you are now integrating the OS to the hypervisor or debug tools.

This is one of the reasons that Wind River is so well positioned in the embedded world, as we have the embedded hypervisor, we have the operating systems and the development tools and excellent relationships with all the silicon vendors.

Have a look at the article and let me know what you think.

Making Multicore CPUs Work in Embedded Communications Designs

2009-11-10T12:49:12-08:00

A good article by Jarrod Siket highlighting the need for a good heterogenous multicore design to be able to meet the need of embedded communications systems. I couldn't agree more. Jarrod has a number of good suggestions as to which engineering challenges need to be addressed. The article is quite timely as I just finished a webinar with Kontron on Multicore and virtualization focussing on ATCA.

Jarrod also makes the statement that 'no other processor architecture is more widely adopted or better suited than x86'. I'll certainly agree that x86 is widely adopted and that it has been doing virtualization longer than the other architectures out there due to it's IT roots.

Nonetheless, Freescale's P4080 is popular in the embedded communications market for good reason: great performance per Watt. The SoC design is focussed on telecom, more so than the generic approach taken by Intel.

It is going to be interesting over the next little while to see how these two architectures will compare in fielded systems.

Primary Multicore Software Configurations

2009-11-09T15:15:15-08:00

Many people ask the question as to what the best approach would be for them to go to multicore and/or virtualization. This is a great question to start a discussion as there is not a single silver bullet. I meant to post a quick diagram on the different multicore configurations before, but life has been busy since we announced the Wind River Hypervisor earlier this year. Busy in this case is certainly a good thing.

I frequently use the graphic below to hold the discussion around. An easy graphic to draw on a white board, or to put on a projector and a sure fire way to get the discussion going. I also posted this earlier article on <a href="http://www.edn.com/blog/1200000320/post/1350046135.html">EDN on multicore configurations</a>.

The "Traditional" option needs little further discussion, it is how people have been developing systems. Choose an OS to go with the processor and you are done.

SMP and unsupervised AMP are the 'usual suspects' to low-core-count multicore processors. Symmetric Multiprocessing (SMP) gives all cores to a single kernel. This works well in low core count environments, but especially when the core count goes up and lots of small work packets have to be handled (interrupts, IP packets), then the impact of the single scheduler becomes evident. This can be countered partially by core affinity or reservation. SMP also runs a single OS, if it crashes for some reason, then the entire system goes down.

Unsupervised AMP for Asynchronous Multiprocessing runs multiple operating systems, one per core. This has the advantage of being able to run say VxWorks and Linux and will also keep the system 'up' if one crashes. However, establishing the separation of multiple operating systems on top of a single processor with shared memory, shared devices and such is not trivial. Ask anyone who has done this before, it requires detailed expertise on how the processor works as well as how the operating system uses the processor. The partitioning provided by AMP can solve the scalability problem of SMP, but only if the algorithm is well parallellizable (which is a different problem altogether).

Virtualization provides Supervised AMP, which provides the benefits of AMP, but without the technical hassles and with the benefit of flexibility. The virtualization layer creates the partitions (Virtual Boards) to run the operating systems within and then gets out of the way. The virtualization layer in this case is referred to as a 'supervisor', as there is very little run-tim impact of virtualization.

Supervised AMP is a great way to provide consolidation of an existing system that utilizes multiple single core processors into a single multicore processor (as the parallallization in that case has already been established in the existing device).

The last configuration in this list is the virtualized scenario in the upper right hand corner. This scenario uses core-virtualization, that is, it schedules multiple different virtual boards on top of a single physical processing core. This of course has impact on the amount of processing power that each virtual board gets. Luckily this scenario still provides determinism, that is, say you run VxWorks and Linux on top of a single core, VxWorks can still have real-time characteristics that you expect from an RTOS, while Linux can provide the services of a typical General Purpose Operating System.

Now, the diagram only shows the primary configurations, these configurations can be combined, for example an SMP virtual board within a sAMP system, or core virtualization on one core with sAMP on the others.

Again, which configuration is best for you... That depends, where do you come from, where are you going towards, what processor architecture do you have an how many cores. This quick overview is certainly not meant as an answer, but it may prove useful to start the discussion. Post questions or discussions below and I will get back to you as soon as I can.

You can also follow me as @markhermeling on Twitter

Addressing Core Issues

2009-10-27T08:12:26-07:00

A good article from colleague Jens Wiegand on multicore in medical devices. Jens talks about consolidation and innovation, two driving factors in both medical and industrial devices. However, he also points out the flip side of the coin: certification. Virtualization can help provide strong separation on multicore, which makes certification manageable (the article goes into more depth).

Jens also touches on tooling for multicore development, an often-overlooked and under appreciated aspect. A single development environment that can be used to develop the entire device (real-time, UI, kernels, userland) as well as drive testing and debugging is a must to create highly efficient development teams.

Three Flavours of Virtualization

2009-09-25T09:29:04-07:00

Here is a great article on the Intel Tools and Software Blog on the three flavours of virtualization: Binary translation, para virtualization and hardware assist. Sharon covers all three flavours briefly, but well balanced.

The message boils down to: you decide, which is consistent with many conversations I have every day with customers. It is Wind River's tasks to provide you with solutions and to help you understand how these solutions benefit you. There is no silver bullet, it all depends on your business needs, the market opportunity and the time window that you have to capitalize on that market opportunity.

Multi-core Needs Virtualization

2009-09-01T14:07:56-07:00

Here is a good blog posting by Gernot Heiser on some of the reasons why virtualization is required for effective resource management on multi-corechips. The post has a strong focus on wireless devices. While there may not be a lot of multi-core¹ chips in cell phones currently, this will certainly increase in the upcoming years. OK Labs is now a member of the multi-core association as well, welcome to the party guys!

¹ Not counting SoCs with heterogeneous cores

The reasons that Gernot mentions in his posting hold in other areas such as printers, storage devices, networking infrastructure nodes, industrial control, networking, aerospace, automotive and so forth as well. Resource management is important for virtually (pun intended) every device nowadays. Besides resource management, there are significant other reasons that customers are using virtualization on top of multi-core for:

1) Re-use and innovation. Software for embedded devices is typically not written from scratch. There are large mountains of code that need to be re-used, preferably on their existing OS. Virtualization enables multiple different (versions of) operating systems to run in different partitions on the same chip, the developer (more typically the system architect) can decide the mapping based on amount of horsepower needed by each partition.

2) Separation (and this is an important one). How does one ensure that one partition does not impact another partition with regards to safety or security? The answer: a layer underneath the operating systems that provides the separation that is small and secure and that manages the multi-core chip. The virtualization layer keeps the partitions apart and allows one partition to be restarted without impacting the rest of the system.

3) Ease of migration. In many of these devices people want to truly decouple their software layer from the physical layer. That is, decouple their software from their hardware. This will help them migrate to newer devices, more cores, different peripherals without having to make sweeping changes through their software architecture.

4) Agility.Virtualization also allows the development team to adjust how the partitions are mapped to the physical cores in a particular design. With the complexity of todays hardware and software, it is difficult, if not impossible to get your design correct the first time. Virtualization provides an additional layer to make adjustments in without touching the software loads of the system.

All in all, virtualization for embedded is here to stay, it's importance will grow as the usage of multi-core in embedded grows.

Multi-core No Silver Bullet?

2009-08-24T08:46:09-07:00

Love this post on 'Downer at the Multi-core Party' by Matthew Dublin. Beside the fact that the language in the first paragraph is quite hilarious.

The article talks, like many, about parallelizing algorithms for HPC applications and I agree that parallelization is an important topic. Developing parallel applications is not easy and any language and toolkit to help is a must have. I don't quite agree with the last statement 'the discussion is happening, ..., everybody just needs to lighten up'. Most of the work that I am seeing is academic or limited in scope of algorithms that they are applicable to. I do agree with the statement though that 'it hardly means that multi-core has failed to deliver'.

My biggest comment on these type of articles is that they largely overlook scenarios where a customer is looking to use multiple cores to solve different algorithms, which is common in the embedded world. Say for example, a multi-functional printer, earlier versions of the printer may have VxWorks as the operating system. Adding Linux would open up new capabilities like advanced web-driven interfaces for example. This can very well be solved by a single core, but a dual core processor has just that much more computing power.

Other examples are wireless infrastructure antenna applications. Handling the control of an antenna covering 360 degrees of separating is typically done in zones. The zones are independent and can nicely be handled by the different cores on a multi-core CPU.

In short: Yes, we do need more support for parallel languages and programming techniques. Even without them though, multi-core delivers significant benefit in the embedded world.

Wind River Hypervisor Heritage

2009-08-21T09:47:52-07:00

Most people I talk to are very interested in virtualization technology, especially when it comes to managing the power of multi-core processors. However, many people are wary of the feasibility and stability of the technology and are wondering how much of this has been proven in real systems.

Well, I don't think we have anything to worry there.

Wind River has been delivering virtualization solutions since 2002 with VxWorks 653, a virtualization solution aimed at safety critical Integrated Modular Avionics. VxWorks 653 has been proven on the world's most challenging aircraft: Airbus MRTT and A400, Boeing C-130 AMP, Boeing KC-767 Tanker, Boeing 787 Dreamliner and Boeing P-8A Multi-Mission Maritime Aircraf t among others.

Aerospace products are some of the most challenging systems ever built and the fact that virtualization has been common-place in that environment for a number of years should be sufficient proof that the technology is ready for prime time.

Wind River has used the experience from the VxWorks 653 platform to build its VxWorks MILS Platform and the Wind River Hypervisor. This experience in virtualization layers, combined with the strength of VxWorks and Wind River Linux allows Wind River to provide customers with a complete solution to use virtualization in your embedded devices.

IT vs Embedded Hypervisor (continued)

2009-07-31T07:11:51-07:00

This is a discussion that will never die out I think. I found this post through my Google alerts this morning and I think it is typical for how people look at IT virtualization.

Bottom line: performance is not the main deciding factor when choosing a virtualization layer for an IT system. 'Best effort' performance is often sufficient, which means that added latency, jitter and throughput is all acceptable (within limits naturally). The main goal of IT virtualization is to get better usage out of your processors.

Embedded virtualization is different, 'real-time' performance is a must, which means interrupt latency in the low microsecond range (basically little-to-no overhead compared to native), determinism, jitter and throughput are very important. The reasons of why people are considering embedded virtualization vary quite a bit. Some projects are using virtualization to facilitate the adoption of multi-core, some to consolidate and reduce their bill-of-materials, some use it to innovate.

Moral of the story: The Wind River Hypervisor runs circles around IT virtualization layers where performance is concerned.

Determinism and Virtualization

2009-07-27T11:00:37-07:00

I ran into this article on determinism by Kim Hartman while googling for something else. The point of the article is simple: determinism is important for embedded systems, faster processors don't change this. And while this is not news, Kim mentions virtualization (the use of Virtual Machines) on a single or multicore machine as a good way of adding real-time behavior to a non-realtime OS.

Virtualization to combine a real-time OS with a non-real-time OS is certainly one of the use cases that come up on a regular basis while talking to customers.

A hypervisor can provide virtualization and keep multiple different virtual machines with different operating systems separate. Either each running on it's own core of a multicore processor, or sharing a core (typically with the embedded RTOS at a higher priority).

One thing to consider here is device access. In a typical IT setting the hypervisor owns the devices and emulates a device for the guest. There is an additional layer of indirection between the actual hardware and the user in the guest. This layer of course can add additional delays and can harm determinism.

The Wind River Hypervisor solves this by giving the virtual machine direct access to the device. The Hypervisor does not 'own' the device, it merely creates an access window through which the virtual machine can access the device. This provides optimal performance while maintaining determinism.