Mark Hermeling

Virtualization and operating systems

2012-12-20T07:30:55-08:00

Virtualization in embedded remains a hot topic. Many customers have a need to do more with their multi-core processors. They typically have a need for a real-time operating system, but they also want to run other operating systems next to it, be it MS Windows, Linux, Android or a 3rd party operating systems on processors from Freescale, ARM or Intel.

The thing with embedded is that you really need to consider the entire system when you make architectural decisions with regards to which operating systems you run, how you run them, how you work with devices and how you partition or virtualize the hardware. This makes it that you can not just go off and select a any virtualization layer and any operating system and throw them together to build a system. This approach works in IT (grab a virtualization product from VMWare, grab an OS from Windows, or maybe Fedora Linux and off you go), however, it does not work in embedded.

In embedded you need a tight collaboration between your virtualization layer and your operating systems (VxWorks, Linux, ...) and that is exactly what Wind River provides with it's embedded solutions. A great example of the use of Wind River's virtualization portfolio is presented in this press release.

With 'tight collaboration' we typically talk about attributes like memory sizes (run a hypervisor in kilobytes or single digit megabytes), interrupt response time (sub microsecond, or single digit microseconds), inter-core communication bandwidth (gigabits or tens of gigabits per second depending on the architecture) and deterministic worst-case execution times. Customers describe which of these attributes are important to them and then design their system around it.

Some of the primary reasons why customers are relying on Wind River's embedded virtualization portfolio are that 1) Wind River has optimized the integration between the operating systems and the virtualization layer for these embedded scenarios; 2) the customer-facing teams at Wind River understand embedded and virtualization and assist the customers in designing their systems; and 3) Wind River takes ownership of the system, not just the virtualization layer.

All these facts put together provides the customer with a best of breed operating environment, the assistance to design the best system and the collaborative relationship with a stable vendor to bring this system to market.

Embedded Virtualization Opens New Possibilities

2011-01-06T06:23:18-08:00

One of the things that stands out in many customer visits that I do is that embedded virtualization often enables possibilities that the customer never thought possible. This of course is a good thing, it may enable them to put a new product to market that the customer had never thought of. In a few cases I have seen customers use virtualization to shift their business models from shipping custom hardware to shipping software only using COTS boards. Virtualization in this case is used to provide the customization that they formerly needed hardware for.

I went on a customer call earlier this year to a customer that was looking to go to multi-core. The customer was building a networking product and was using VxWorks on a single core Intel processor and wanted the additional processing power that a dual or quad-core could provide. They wanted to go to multi-core and came to Wind River to see what their options were. We discussed their business issues, came up with a set of requirements and decided that VxWorks SMP was their best bet.

The Symmetric MultiProcessing capability in VxWorks allows a customer to quickly utilize the additional cores, either by letting the operating system decide which process runs where, or by manually optimizing the configuration for whatever performance aspect is important to you (throughput, determinism, real-time response).

The reason that we settled on VxWorks SMP was that the customer had already split their application into tasks, each tasks was very independent in processing and memory usage and there was minimal locking between the tasks. On top of this the customer did not need strong separation between the tasks. VxWorks SMP was the easiest and fastest migration path with minimal risk for them.

However, as part of the discovery process we also found out that this VxWorks based product (let's call it product A) works with another product B that uses Linux and there is an additional management console C that uses Windows. Also as part of the discovery process we discussed embedded virtualization of course and how it could be used (for example) to combine products A, B and C onto a single quad core processor. This raised some eyebrows and there was some chatter between the participants, however at this point the company was selling each of them as independent products and they were okay with that. The immediate problem they needed to solve was to get their product A to multi-core.

Fast forward a couple of months. The company has been making great progress migrating their product A to a dual core processor. However, it turned out that the discussion on embedded virtualization had reached other levels of management level inside the company and they wanted to explore the possibility of introducing a new product combining A, B and C into a single box.

It turned out that the reason for the separation was scalability (as it often is). One management box C can be combined with several boxes running A and 2 boxes running B to build a particular configuration. The main market for these products was huge networking installations. However, for the bottom of the market this was complicated and expensive, a single box with a pre-integrated combination of A, B and C would open a new market segment of much smaller networking installations, something that without virtualization is either not feasible, or at the very least expensive to do.

Long story short, product ABC is now under development using a mixture of VxWorks SMP, Wind River Linux, Wind River Hypervisor and Microsoft Windows XP. The immediate target is a quad core processor, however, this can easily be extended to a higher core count product that includes multiple partitions running product A, a few B and a single C.

The funny thing (to me at least) is that we have a number of examples that are very similar in many other markets. The examples are the same in that the initial call with the customer is about migrating to multi-core, and how this can be easiest achieved and then morph into a combination of multi-core and virtualization to reach new markets to deliver new products.

It is a great proof point that indeed embedded virtualization opens up new possibilities and your Wind River representative is happy to talk to you about this to see how you can benefit from this exciting technology.

Virtualization on Handsets or not?

2010-12-29T05:57:33-08:00

Here is an interesting article, continuing a discussion thats has been going on for a long time on mobile virtualization. It argues the case that a single mobile phone can run multiple operating systems using virtualization. A bit of googling will show that this is indeed already the case. However, there is a big caveat typically that you do not see exposed.

Mobile phones have two big constraints: processing power and battery power. Typically one operating systems is the 'phone operating system' and the other OS is more general purpose. With 'phone OS' I mean that the OS has to handle a lot of the time critical tasks, sometimes even to the level of passing data off to/from the signal processing side for communication with the antenna and the wireless network.

The signal processing side is very time sensitive and hence it is important that the time constraints are met and data is delivered/picked up on the right time. This makes it that typically the virtualization layer will have to be aware of this and it makes the solution not a general virtualization solution, but a highly device customized solution.

The other solution (mostly used with Windows or Linux/Android type devices) is to just run everything on one OS. This of course works as well, but has limitations in the area of security, stability, upgradability and more. The biggest issue is that the phone side and the general purpose side run on the same kernel and in the same OS address space, this makes security much more difficult to guarantee. Also applying fixes to the OS to correct phone glitches may impact the general purpose side. It is all one big ball of spaghetti, difficult to understand and untangle when needed.

My vote is on the virtualization solution, the concepts mentioned above hold for not just mobile phones, but virtually (pun intended :) any device in the market today. Whether it is a multifunctional printer, a network switch, industrial control, automotive, aerospace and defence or another application, virtualization can help put multiple 'personalities' on your device for the various tasks that you may have to do (from real-time and safe&secure to general purpose).

Virtualization can help here, but it will have to be embedded virtualization, with that we mean that the virtualization layer can be optimized for a particular device and that the device-builder has all the tools to fine-tune the virtualization layer to the device, OS and application he is building. With fine-tune we typically mean the fine-tuning of sharing and/or partitioning of processing cores, memory and devices. Embedded virtualization provides the flexibility and the tools to perform this fine-tuning, for an example, have a look at the Wind River Hypervisor.

Migrating to Multi-core using Embedded Virtualization

2010-11-22T17:52:28-08:00

So, let's assume that you are currently on a single-core board and you would like to explore the possibilities of delivering a next generation device using multi-core technologies. There are a lot of different ways to approach this. The most elaborate involve doing an architecture study into ways in which you could evolve your current device architecture. This study would need to be approached from both a technical, as well as a business level.

The funny thing with software is that 'virtually' anything is possible, but the trick is to find approaches that are not only technically possible, but that provide business benefits, usually in a short time-frame. Part of the equation is the amount of work required to build this next generation of device.

In the 'olden days' people would spend a lot of time deciding about which hardware to use for a project at the beginning of that project. Once the hardware is decided, the software is next.

Well, embedded virtualization is one of the leading technologies to support the migration to multi-core and embedded virtualization is a bit like hardware and a bit like software. There are three variables to decide on when planning to use embedded virtualization: how to partition or virtualize cores, memory and devices. And that is immediately where the complexity lies, there are many degrees of freedom. Doing an architecture study at the beginning of the project helps you define which parameters you are interested in such that you can then explore the proper degrees of freedom with a proof of concept for example.

So let's assume that the study has been done and we know what direct we want to go into. Now it is time to explore and do a proof of concept. Well, let's also assume that you are targeting an architecture that has hardware assist for virtualization. Examples of this are Intel processors with VT-x, or Freescale processors with the e500mc core, or the upcoming ARM Eagle. Well in these cases life is relatively easy. One can run unmodified guests, meaning unmodified operating system kernels, and run them directly on the new multi-core hardware on top of a hypervisor in a virtual board (also known as a virtual machine). In this case one would directly assign all devices directly to this one virtual board and the software will run as in a native environment, but now run on a virtual machine.

There may still be some changes required to the BSP, for one, the devices on the new platform may be different for example. However, this is by far the easiest and straight-forward way to migrate to a new multi-core based platform. The next step is to use the additional processing cores, memory and devices to create new content.

It is easy to add another virtual board with a new operating system, maybe add Linux to an already existing real-time operating systems. At this point, maybe one or more of the devices need to be shared or virtualized and maybe the new operating system needs to be connected to the initial virtual board using some form of IPC possibly.

Once this is done it is time to start building new content and to experiment with the new, multi-core environment based on the parameters laid down in the original study.

Now, let's take a step back and assume that we are not targeting a processor with hardware assist for virtualization. In this case, we can still do the exact same things explained above, but with the difference that we can not use unmodified operating systems inside the virtual boards, instead we need paravirtualized the operating systems we intend to use. Any operating system can be paravirtualized (the source needs to be available to make the changes though), and the Wind River operating systems (VxWorks and Wind River Linux) are already paravirtualized to run on top of the Wind River Hypervisor.

The rest of the workflow remain the same, we have to be a bit more careful with regards to device drivers. The operating system, the device drivers and the hypervisor have to collaborate to deliver the functionality of the system. There is a bit more work and coordination involved, but getting a basic software architecture up and running is still fairly straight forward.

All in all, there are well described paths to migrate a single core system to a multi-core, or combining multiple single core systems onto a multi-core. The true challenge comes when resources (such as devices, cores and memory) need to be shared, but I will leave that discussion to a later blog post.

Following these paths provides a quick way to evaluate how existing designs would run on top of an embedded virtualization layer in a proof of concept. Going through the process of making a proof of concept not only validates the findings from the architecture study, but it is also a learning experience for your software development teams in preparing them for the new world of multi-core.

Embedded Virtualization in Gaming

2010-11-01T06:13:09-07:00

A few weeks ago I found myself in the Hard Rock Hotel in Las Vegas. My reason for being there was a seminar that Wind River was organizing together with Intel and AVNET specifically focussed on companies that build gaming. The main topics were graphics, manageability and of course, virtualization.

I am always eager to fold real-life examples into my presentations, hence I decided to try my luck at some of the slot machines in the Hard Rock Hotel under the disguise of 'domain study'. And I have to say, the state of slot machines has progressed significantly since the last time that I was in Vegas many years ago. The top of the line machines had multiple 30" inch displays with either static or moving imagery, they had player tracking, surround sound, some of them even had imagery over the rolls themselves.

Since this was a domain study, I started thinking what type of hardware would be running in these slot machines. Off the top of my head I could come up with the following sub-systems:

Money collection: The part where you insert your dollar bills
Money return: much less used, but important nonetheless
The game: This is the part that is the most fun
Machine management: The part that reports on whether the machine is doing okay, that connects money collection, return and the game to the central casino management
Player tracking: where you slide your card and it keeps track of your preferences, and feeds your gambling habits back to the casino for perks. This is also where (in a more advanced scenario) you could order drinks, make reservations for restaurants and stuff.

So that is 5 separate subsystems and I expect that almost all of them run on their own dedicated hardware. That is 5 processors, memory, power supplies and so forth in that big box. Now, the slot machines are fairly big, so I don't think that space is an issue, but heat dissipation could be and maintenance certainly is. I was talking to another customer and they were using a number of 6% as the yearly failure rate of a single board. The higher the board count, the higher the potential of failure of course.

Technically I could integrate all of these 5 different subsystems in a single multicore chip using embedded virtualization. Each subsystem would run it's own OS inside of a virtual machine. The OS may be a real-time OS, or a general purpose or graphics OS such as Windows. The game would be the biggest compute power user, so it would need a dedicated core, possibly 2. The other parts are much smaller and can probably all share (time slice) a single core. From a device assignment, most of the devices would all be partitioned between the different virtual machines, the one exception possible be graphics. The virtual machines would also need to communicate between each-other which is very easy to set up within the multicore environment.

One thing to keep in mind is that these virtual machines will run at different levels of certification. The money collection machine for example may need to interact with credit card verification back ends, the game of course needs to be certified by the gaming commission and such. To this extent the embedded virtualization layer needs to be certifiable to at least the most stringent certification level of each of the subsystems.

Virtualization of the display is a fun topic. In principle it is not difficult, however, in this case there is a bit of added complexity with regards to the certification. It needs to be very clear to the user when he is gambling, versus when he is looking at player tracking, venue reservation or generic web-browsing and the data entered should never been able to be leaked into the other application. For instance, when the user needs to input credit card data for venue reservation, that should not be usable by the game or by the generic web browser, or very bad situations can develop.

The 'domain study' resulted of course in the casino making some money off me, but I had sufficient content to talk about in the seminar the next day.

Wind River Hypervisor 1.2, which was officially announced last week, provides support for 'unmodified guests', meaning that one can run Microsoft Windows, a 3rd party operating system, or an in-house operating system unmodified on processors that support hardware assist for virtualization (such as the Freescale P4080, or Intel processors with VT-x). This is a main enabler for consolidating multiple boards on a single multi-core processor, which is of course not limited to the gaming industry highlighted in this post, but also much desired in industries such as Networking, Industrial Automation, Medical, Automotive. Aerospace and Defence and so forth.

Embedded Virtualization and Multi-Core

2010-10-25T06:00:00-07:00

VDC has released another report showing that multi-core is making fast inroads in embedded systems and we are seeing that at Wind River as well. There are exciting new possibilities with these chips, we have never had this much processing power in such small packages.

Take for example Freescale's new P4080 chip, 8 cores, 2 memory busses and a wealth of peripherals that are designed for multicore usage (such as the data path acceleration architecture (DPAA)). There are many things to consider with this new chip, I have blogged before about the many primary multi-core configurations available from SMP to uAMP, to sAMP to core virtualized. You have to consider partitioning, or virtualizing cores, memory and devices.

This may seem straight forward, but there are many things to consider. As mentioned, the P4080 has two memory busses, all cores have access to these busses at the same cost, however, if you run all cores with memory-heavy workloads on the same bus, you may impact performance, both from a throughput perspective as well as from a real-time and determinism perspective.

Many customers are wondering why, when and how to make the jump from multi-core. They are looking for information, recommendations and experience. Luckily there is help around.

First of all, there is a webinar this Wednesday, Oct 27 at 2pm ET with Curtiss Wright, Freescale and Wind River in which the three parties provide an overview of the P4080 and its possibilities on Curtiss Wright boards.

Secondly, Wind River has today announced the immediate availability of a new version of the Wind River Hypervisor with support for the P4080. Support for a many-core chip like the P4080 in the embedded space requires a hypervisor custom built for scalability and with embedded virtualization in mind. You can’t target this architecture by using an IT virtualization layer or by retrofitting an RTOS to do virtualization. Wind River’s support for the P4080 of course does not stop at the embedded virtualization layer, it extends to VxWorks, Wind River Linux, Wind River Simics and Wind River Workbench. The entire arsenal of products are available to get software our as fast and correct as possible.

Lastly, Wind River has released the Wind River Multi-Core Adoption Package, where multi-core experts from Wind River can work with your team to discover the business case for multi-core with your team and build a list of options as to how you can step into the multi-core era. The goal of this package is to firstly provide you with a list of options within the parameters that were discussed with your team on how you can best utilize multi-core for your business. If you decide that one of these options is the best choice for you, then Wind River is happy to assist you with to assist you with the implementation as well as a second step.

The multi-core disruption is an exciting time to start reconsidering how to architect your next generation of embedded devices.

Devices! Devices! Devices!

2010-09-17T11:05:32-07:00

That is what embedded systems is all about: handling devices. An embedded software system has input devices and output devices and some amount of software running on operating systems, running on hardware that turn input into output. The trick is to handle the input in a fast an deterministic way and generate output in a fast and deterministic way. Determinism is important all the way through the system, from the device, to the hardware, to the operating system to the software running on the operating systemAdd virtualization to the picture and quickly understand that not all virtualization layers are created equal. Especially in the area of IT-virtualization the focus has been on abstraction of the physical environment, which includes devices. This means that often the software does not have access to the physical device, it has access to a device-driver inside the hypervisor that 'owns' the device.

And abstraction quickly impacts determinism and real-time aspects such as latency. A true embedded Hypervisor gives the system designer full control over how he wants to handle his devices such that the designer can minimize the impact in the areas where determinism is crucial for the embedded device.

A true embedded hypervisor (such as the Wind River Hypervisor) provides a device management infrastructure that provides multiple options as to the handling of devices:

The first and most simple option available is to partition. That is, given a set of devices, partition them between the virtual boards running on the processor. Say your hardware has 2 PCI busses and 2 virtual boards, give one PCI bus directly to each virtual board and assign the devices on the bus to the virtual board. Now you have a nicely partitioned system and the hypervisor does not need to intervene, hence there is absolutely no impact to the determinism in your system.

The second option is to share, assign the device directly to one of the virtual boards and the operating system in that virtual board owns the device. The software in that virtual board has direct access to the device and hence no impact to the determinism. If another virtual board needs access to the device you can export it through a inter-board messaging protocol. The second virtual board has access, but it is slower and non-deterministic. If this is okay, then we have an easy solution.

The third option is to virtualize the driver into the hypervisor. This is where we are starting to add a layer of delegation into the device handling path, so we need to make sure we write the driver carefully, consider core-interaction in the design (driver running on core 1, virtual board that uses it on core 2, hence more inter-core communication) and make sure that we keep our worst-case execution times in check. We also have to consider multiple users (which is why we are using virtualization, if there was only a single user we could use partitioning) and how it impacts our performance.

Give your Wind River representative a call if you are planning your next generation of embedded system and he can help you design a way to include virtualization into your system without impacting the determinism you require.

Consolidate, Consolidate, Consolidate

2010-08-27T06:57:42-07:00

Many telecom applications are actually built up from multiple smaller sub-applications, often running on their own server in a rack, ATCA or otherwise. These servers run on multi-core processors, depending on the age of the last refresh this could be a dual, quad core or more. This is of course nothing new, what's new is how virtualization can improve server utilization. First of all, todays multi-core processors are upwards of 8 cores, such as a Freescale p4080, or 24 logical cores with a dual Intel Westmere configuration. However, each individual sub-application can not use this much horse power and hence the new, shiny multi-core processor may go partially un-used. Un-used, unless you consolidate multiple of these sub-applications onto these shiny processors.

And that is exactly what virtualization through a hypervisor gives you. If the original configuration was using 4 sub-application on 4 dual core servers, then now we can try to fit these 4 applications on one high core-count multi-core server. And the good news is that the effort to perform this consolidation is often very minor. The applications and operating systems will migrate effortlessly, the one important question to answer is how each application gets access to devices, usually ethernet ports. And there are many technical ways to resolve that.

The added benefit of this consolidation is that the application configuration becomes less tied to the hardware, so much more flexible business models are possible, where the hardware is truly COTS and the telecom applications are configured onto this hardware through a configurable virtualization (Hypervisor) layer. With configurable here I mean that the designer (or the customer) can decide how many cores each application gets, so the final software load is very tuneable to changing network conditions.

The IT-people under us may be wondering why this is new, they have been consolidating for years. What is new is that telecom applications can often not accept the penalty incurred by full virtualization like in an IT hypervisor. They need low latency, determinism (low jitter) and full device performance. An embedded hypervisor maintains these real-time characteristics, without sacrificing flexibility.

Type 1 versus Type 2 Hypervisors

2010-08-26T09:26:00-07:00

I have always found the difference between Type 1 vs Type 2 hypervisors rather uninteresting. In short, a type-1 hypervisor is a hypervisor that has direct access to the hardware, where a type-2 hypervisor executes inside an operating system. Most hypervisors are type-1 hypervisors, including IT hypervisors such as VMWare, Xen, KVM and such. Type-2 hypervisors are applications like VMWare Workstation (or Fusion), Parallels, .... The distinction between type-1 versus type-2 is really not as useful as most people think. There is a good blog article by Anthony Liguori that describes this as well, complete with references, the article is a bit old, but still correct. What is actually much more interesting are the properties of the hypervisor, the following is my short list of important properties for an embedded hypervisor:

real-time performance, related to interrupt latency and context switch times. For all intents and purposes interrupt latency needs to be as close to native as possible. This is usually measured in low microseconds (or sub-microsecond).

determinism, related to things like jitter and the worst case latency time of interrupts, again measured in low microseconds or below

footprint, how large is the hypervisor in RAM, typically measured in kilobytes, with a reasonable range around the 50-100k, though for some really small systems you may need to be able to go lower than 50k

device access, can the virtual board have direct access to devices, without the hypervisor getting in between for best real-time behavior, while at the same time not limiting true virtualized devices (shared ethernet for example)

configurability, is it possible for the embedded developer to configure exactly how the hypervisor is configured with regards to mapping of devices, cores and memory

certification, becoming more and more important in many critical systems: a path to safety certification

The above properties designate an embedded hypervisor and it makes it quite evident where the embedded hypervisors differ from IT level hypervisors.

Freescale on Multi-core and Virtualization

2010-08-19T05:15:48-07:00

Two good blog posts from some of my colleagues at Freescale. One on Heading Into Hyperspace: Hypervisor and Multi-core design by Jim Trudeau and one by Rob Oshana on the Top 3 Keys to Multi-core software development. Jim talks about the multi-core aspects mostly from Freescale's P4080 perspective, an elegant, powerful and very popular processor in the networking space. The P4080 has not only many processors, it is truly designed for multicore with multiple peripherals (multiple PCI hosts) and multiple memory controllers for example to reduce contention.

The P4080 is hardly the only multi-core SoC in Freescale's arsenal, some other popular processors are the P1022, the P2020 and all their derivatives. The difference between the P4080 and the others is that the P4080 is based on the e500mc core, which has hardware assist for virtualization. However, that doesn't stop us (us being Wind River in this case) to offer a Hypervisor on the other QorIQ processors as well (or even pre-QorIQ processors such as an PPC8572 or even PPC8548).

The points made by Jim and Rob are good for sure, to paraphrase and summarize:

A powerful processor
A flexible Hypervisor
Inter-OS messaging
Flexible tooling to configure the Hypervisor and the OSes
Hypervisor aware On-Chip (JTAG) debug to deal with driver debugging

Wind River of course provides all these. However, there is one, important, element that I would like to add: You need to run operating systems on the Hypervisor, typically a real-time operating system such as VxWorks, but general purpose operating systems such as Linux are also popular. Ideally, what you would like is to have all this pre-integrated on the silicon. That is, a single vendor that can provide you with a complete package of Hypervisor, operating systems and tooling to enable you to start developing for this powerful Freescale design.

And that is exactly the philosophy behind the Wind River Multi-core Software offering, an integrated package from one vendor to reduce your risk and accelerate your time to market.

Contact me directly if you would like to have a more in-depth discussion on how to use Freescales single and multi-core processors with virtualization.