According to the National Science Foundation, about 25% of students worldwide earned undergraduate degrees in science and engineering fields. However in Asia almost 50% of undergrads receive engineering degrees compared to only 5% in the US.

Although the employment outlook here in the States has improved recently, employers are still choosy about where they seek engineering talent. On-campus recruiting has been particularly affected. While hiring is still vibrant at top engineering schools, it’s dropping at others. According to a July article from IEEE Spectrum, recruiting for electrical and computer engineers at Ohio State has dropped 13%.

When only the top grads from the top schools are sure to receive job offers, how do we incentivize students to develop the core skills needed so that we are assured a rich talent pool locally as well as globally? Engineers still have high earning potential, but increasingly starting salaries are lower than in years past. This is partly due to prior layoffs in the industry. According to BLS data, both the number of EEs without jobs and with EE jobs fell slightly from the third quarter of 2009 to the fourth quarter of 2009.

As companies begin to hire once more, students are now competing with more experienced candidates, and the cost gap between novice and veteran is less than it has been in prior years.Unfortunately, this isn’t good news for the inexperienced employee. As hiring managers and entrepreneurs, we need to look forward with vision and not focus solely on the short-term gains. Remember that long-term investments do make returns.

[click image to enlarge]

1. Apple – Jobs for Student and New Grads
2. AMD – Co-op / Intern Program
3. ARM – Student Internships and Engineering Placements
4. Broadcom – Internship / Co-op Program
5. Centaur Technology – Internships
6. Cisco Systems – Graduate Careers & Internships / Co-ops
7. Freescale Semiconductor – Student Intern / Co-op Program
8. Fujitsu Labs America – Internship and Job Opportunities
9. IBM – Intern / Co-op Opportunities
10. Intel – Internships
11. LSI – College Recruiting and Internships
12. Marvell – College Recruiting Programs
13. NEC Labs – Internships
14. NVIDIA – Intern and Co-op Programs
15. Oracle – Internships at Sun Microsystems Labs
16. Qualcomm – Internship Programs
17. Samsung – Internships
18. ST Ericsson – Student Opportunities
19. Texas Instruments – Opportunities for Students & New College Grads
20. Xilinx – Internship and Co-op Programs

The scholarships will be made available to graduate students performing research in the field of processor verification and to recent graduates with less than three years experience.

“We believe MTV is an important event that defines Austin as a worldwide hub for processor verification”, said Eric Hennenhoefer, CEO of Obsidian Software. “Obsidian is proud to sponsor and collaborate on this world-class event.”  2010 will mark the fifth year of sponsorship for Obsidian.

The deadline for scholarship applications is October 1, 2010. Potential applicants are encouraged to apply online at: http://www.obsidiansoft.com/community/mtvcon-scholarship/

About MTVCon

The purpose of MTV is to bring together researchers and practitioners to exchange innovative ideas and to develop new methodologies for solving difficult challenges facing the processor design community.

About Obsidian Software

Obsidian Software, a privately held company, has been providing processor verification products, verification consulting and training services to processor designers and semiconductor fabs since 1997. Obsidian’s RAVEN software has been used to successfully verify dozens of processor implementations by many of the world’s leading semiconductor companies. Obsidian has been recognized as part of the INC500, Austin Heavy Hitters, and Austin Fast 50.

Contacts:

###



Obsidian Software, Inc., a growing processor verification company, and EDAcon Partners Ltd., a sales and marketing outsourcing company, announced today the expansion of Obsidian’s sales internationally. Both companies stand to benefit from the rising strength in the semiconductor industry, which has outperformed the broader technology sector thus far this year.

“EDA firms like Obsidian Software are positioned to grow in this economy and can take advantage of our global network of representatives, knowing that an experienced manager is driving their sales in the specified territories,” said Coby Hanoch, President and CEO of EDAcon Partners. Hanoch founded EDAcon Partners in 2007. Prior to this, Hanoch established sales offices in North America, Europe and Asia while serving as VP of Worldwide Sales for Verisity Design, Inc. and VP of Verification Sales at Cadence Design Systems, Inc.

Obsidian’s engineering team specializes in processor design testing through creation of functional test suites and random test generators. Austin and Silicon Valley have always been profitable for Obsidian, but with chip sales up worldwide, Obsidian is positioned to grow internationally.

EDAcon offers a sales outsourcing model for EDA vendors, saving companies like Obsidian significant resources by providing a worldwide network of experienced sales representatives. EDAcon sales representatives are local to each country and have a proven track record of selling EDA products.

“Through this investment in sales, Obsidian will continue to expand its reach in the semiconductor industry and advance our reputation as a leader in processor verification. We have demonstrated a strong track record across a variety of architectures such as ARM, MIPS, X86, Power ISAs, and proprietary RISC/CISC implementations for SoC core designs,” said Mark Glenewinkel, COO of Obsidian Software.

About Obsidian Software

Obsidian Software, a privately held company, has been providing processor verification products, verification consulting and training services to processor designers and semiconductor fabs since 1997. Obsidian’s RAVEN software has been used to successfully verify dozens of processor implementations by many of the world’s leading semiconductor companies. Obsidian has been recognized as part of the INC500, Austin Heavy Hitters, and Austin Fast 50.

About EDAcon Partners

EDAcon Partners enables EDA and IP vendors to outsource their sales activities, including definition of sales strategy and development of marketing materials. With its worldwide network of representatives, EDAcon Partners provides an instant sales channel with proven abilities, saving EDA and IP vendors the enormous investment in time and money needed to find and ramp up representatives in every country.

Contacts:

Obsidian Software
Saturday Schroder, Marketing
(512) 330-9818 x 113
saturday@obsidiansoft.com

http://www.obsidiansoft.com

EDAcon Partners
Coby Hanoch
+972-545-421-321
coby@edacon-partners.com

http://www.edacon-partners.com

For those of you unfamiliar with DVClub, membership is free and is open to all non-service provider semiconductor professionals. Most members work in verification, but there are also plenty of entrepreneurs, professors, students, managers, investors, and even design engineers who attend. If you’re interested and would like to learn more, why not join the club?

Chuck Alley, IBM
Using PSL and FoCs for Functional Coverage Verification

Bob Colwell, Intel (Retired)
The Validation Attitude

Raj Dayal, Qualcomm
Managing Deployment of SVAs in Your Project

Ish Kumar Dham, Texas Instruments
Design Verification to Application Validation of a Multiprocessor SoC

Sanjay Gupta, IBM
Cell Verification Metrics

Narasimha Karunakar, AMD
Low-Power Verification Challenges

Mark A Firstenberg, IBM
Experience with Formal Methods, Especially Sequential Equivalence Checking

Jai Kumar, Sun
Leveraging Low-Cost FPGA Prototyping for Validation of Highly Threaded Server-on-Chip

John Ludden, IBM
Mainline Functional Verification of IBM’s POWER7 Processor Core

Milind Padhye, Freescale
Wireless Low Power and Verification Challenges

Somdipta Basu Roy, Texas Instruments
OMAP Verification

Scott Runner, Qualcomm
Verification of Wireless SoCs: No Longer in the Dark Ages

Sakar Jain, Freescale
Verification of the QorIQ Communication Platform’s CoreNet Fabric with SystemVerilog

Shahram Salamian, Intel
Intel Atom Processor Pre-Silicon Verification Experience
CPU Verification Metrics

Jason Stinson, Intel
Pre-Si Verification for Post-Si Validation

Paul Tobin, AMD
Verification in a Global Design Community

Durgam Vahia, Sun
Mapping Server-Class Multi-Threaded OpenSPARC T1 Processor Core on FPGAs

David Williamson, ARM
Verification Metrics

Paul Zehr, Intel
Intel Xeon Pre-Silicon Validation

Complexity

In his recent presentation discussing verification of the Power7 processor, John Ludden of IBM opened with a quote from an IBM exec more than a decade ago. “it’s not rocket science”- a perception held by some members of the management and design communities at that time.

However, designs have become a whole lot more complex over time. The Power7 processor at 45nm has 1.2B transistors on a 567 sq. mm die, supporting 8 cores with 4 threads each, an on-chip eDRAM, 3 levels of caches and 2 DDR memory controllers. Yet as verification complexity multiplies in this multi-threaded design, it’s very helpful to have some of the more advanced tools and methodology at your disposal.

Tools and Methodology

Fortunately for Ludden and the Power7 team, IBM has invested in verification technology for years (in spite the quote from the exec). The company continues to develop and rely on in-house tools for many of the advanced verification technologies for processor-specific testing. These include the test-bench, multi-thread test generators, hardware accelerators, formal and semi-formal tools, micro-architecture checkers (API based), cache coherency checkers and coverage tools. Exercisers
originally developed for post-silicon validation were used to exploit the hardware acceleration platform. Forty-five thousand coverage points were organized to assist with big picture and were used to re-direct the test generator and exercisers for accelerators.

To support corner case testing for events that occur rarely, especially in multi-threaded scenarios, software irritator threads were used. These irritators are capable of creating the worst possible contentions. Through their application, twenty-three high quality bugs were revealed hiding in the corners.

A methodical application of these tools and technology clearly captured and advanced the industry best practices.

Designing for Verification

Designing for Verification was an important element in managing the overall risk to verification time line. IBM minimized the risks by maintaining a tight interaction between the specification and verification teams during the design phase and allowing the verification team to maintain architectural changes. “Chicken switches” were placed in silicon that allowed verification team to back-off an area considered risky or possible of otherwise compromising the verification effort. These switches provide workarounds, with some small impact on performance but no functional change, for accessing difficult to verify micro architectural features. Hardware irritators were also used to enable stress testing of corner cases in both pre-silicon and post-silicon testing.

Conclusion

The Power7 draws many architectural features from the Power5 and 6 designs, although it is a much more complex and powerful processor with a much shorter verification cycle. Ludden and the Power7 team accomplished this remarkable feat with a lot of foresight in planning, metrics collection and careful execution. Tight interlocking between metrics collected and verification plan was key part of tracking mechanism and functional closure. This project should serve as an example of how to plan for and manage risks in a complex verification project.

Kudos to John and the IBM team. His full presentation can be downloaded here.

As we rely on our mobile devices more as mini computers and less as simple phones, processing power becomes increasingly vital for companies like Apple. Nearly 90% of US households have a cell phone, yet voice minutes use has flat lined while more households increasingly give up their landlines. We’re still using our phones, but using them more and more for data and less for talking.

When Steve Jobs introduced the iPad, he referred to the A4 as the best and most complicated chip Apple ever designed. Intrisity worked closely with a division of Samsung Austin Semiconductor, said to be the largest chip manufacturing plant in North America that builds the A4 chips for Apple. Intrinsity CEO, Bob Russo, credited the partnership with Samsung for bringing visibility to the relatively small, 100-employee company based in Austin, Texas.

What does this mean for Austin?

This acquisition brings focus once again to Austin, Texas, a city increasingly important in the microprocessor field. Before this acquisition, Intrinsity was a David competing with Goliaths such as Texas Instruments and Freescale Semiconductor, both based in Austin. They were also going head to head with Intel Corp.’s Austin-designed Atom processor. Clearly when it comes to fast chips, there’s quite a lot going on in Austin, Texas.

Naturally, Austin is also becoming a hub for processor verification. Game-changing advances in chip performance call for equally nimble and innovative advances in processor verification. Obsidian Software and Intrinsity share a common passion for excellence and innovation, they were both founded in 1997 in Austin, and Mark McDermott, former VP of Engineering for Intrinsity sits on Obsidian’s Advisory Board.

Has the processor business turned?

The Intrinsity deal is the second chip acquisition for Apple in two years. Recall Apple’s purchase of PA Semi in 2008. Companies who design cutting edge mobile devices are increasingly choosing to do processor design and to some extent verification. Are these purchases by Apple part of a defensive strategy to stay ahead of emerging technologies funded by increasingly scarce venture capital? Or will we see a new generation of innovation come from the Intrinsity team that is now a part of Apple?

Companies such as Qualcomm, NVIDIA, and Marvell build their own unique ARM chips for their devices. Perhaps the line between device makers and chip makers will become increasingly blurred as both camps work to gain market advantage through increased vertical integration.

Further Reading:

IEEE Spectrum – Forecasting Apple’s Intrinsity Acquisition

Mac Daily News – Apple’s Intrinsity Deal is a Snapdragon Slayer

Austin Business Journal – Apple Inc. Acquires Intrinsity

Anandtech – Apple’s Intrinsity Acquisition: Winners and Losers

NY Times – Intel and Qualcomm Eye Each Other’s Terrain

The idea of an intelligent testbench has long been of interest in functional processor verification, although it has always seemed to fall short of expectations when it comes down to just what degree of “intelligence” is really involved. Throughout this document, we will present the argument that a sophisticated, well-evolved, dynamic random test generator, when used as part of a complete verification plan, can be of more value than marketing-driven intelligent verification products.

Defining Intelligent Testbench

An accurate definition of “intelligent testbench” is difficult to find, so let’s begin with that offered by Wikipedia. The intelligent testbench is described as something that “uses information derived from the design and existing test description to automatically update the test description to target design functionality not verified, or covered by the existing tests”^[1]. This implies that a feedback loop exists which is capable of creating new test sequences based on what has and has not yet been tested. Other than this closed loop system, the concept is very similar to that of a random test generator.

Not all Test Generators are Created Equally

I remember at one time, we were speaking with a potential client who said something along the lines of “I don’t know why you people want so much money for this RAVEN thing. I can just get a co-op to write one in a week.” He wasn’t wrong in his assumption that a relatively unskilled engineer could conceivably write a generator in a short amount of time. However, the old adage remains that you get what you pay for. A generator of this sort would be incapable of effectively verifying a design of any complexity whatsoever. This is analogous to running every instruction followed by every other instruction and calling verification complete. There are no standard methodologies for constructing test generators, and each one will have different methods for achieving randomness, different capabilities in pipeline exploration, varying abilities in multi-processor testing, etc.

Functional Coverage

Many intelligent testbench products claim to automatically create test sequences based on pre-designated coverage points. However, the belief that hitting every coverage point means that your design is verified is a big mistake. By the very fact that coverage points are singular points in a vast space, they cannot cover the entire design. Engineers can work hundreds of hours writing more coverage points, but it will never be enough to completely verify the design. Because of this, our test generator uses templates (created by engineers) that automatically create sequences to hit not only the coverage point, but also other behaviors around that point.

Figure 1. Abbreviated Flow of Randomness

Random Stimulus Compared to Feedback Looping

RAVEN is very good at what it does; it is designed to hit both simple and complex behaviors randomly with little direction from human users. For instance, if we’re looking at instruction A followed by instruction B with operands X, Y and Z, then we’re going to hit that randomly with ease. Constrained-random templates can replace 95-98% of all directed test sequences. It’s only a matter of time and simulation power applied before we randomly hit all of these simple scenarios and many of the complex ones as well.

The whole point here is that coverage points that are easy to direct (i.e. via feedback), will have already been covered by virtue of random testing. Highly complex behaviors and difficult-to-reach corner cases require a significant degree of architectural knowledge, and they are too difficult and too architecturally dependent to effectively be covered by a piece of software. If an effective feedback loop could be done with good logic or programming skills, we would have already done it.

“At a recent ASIC verification panel discussion at DesignCon, a question was asked about intelligent testbenches — something promised for a long time but not really delivered by the EDA companies. One of the panel members from a design company responded, and said that if you ever tell his engineers that his testbenches are not intelligent then he would be very upset. I am sorry but I have to break the news to him. Testbenches are dumb!” – Brian Bailey ^[2]

The Promise of Eliminating Redundant Testing

Eliminating “redundant testing” via software is a dangerous thing to do. Suppose that we have two similar sequences of 20 instructions, but the second test has one instruction that is different. Are those tests redundant? They seem like it, and they have a lot in common. But depending on what those instructions are, what registers they use, what the pipeline looks like and whether they took exceptions, that one instruction can be the one that makes the difference. So it’s dangerous for a piece of software to make the supposition that this test is redundant. It could very well be that this next instruction could be the one that reveals an error.

When I was working at a major processor company several years ago, we found a case in silicon where the processor would hang for seemingly no reason. What we found was that an illegal access to a register was causing the error approximately 1000 instructions before the hang would occur. This taught me that sometimes even the designer is not aware of the conditions that can lead to a bug. Designers may know their own block, but the interactions between the blocks can be very complex, and oftentimes this can be confusing even for experienced engineers. So I think that it’s really dangerous to assume that you can get rid of redundant tests in this manner.

But this is not to say that ineffective tests should be continually simulated. Test templates should remain in the suite only as long as they continue to uncover errors. When it no longer seems like it’s finding bugs, that template should be archived and replaced by another. But having a piece of software decide that for you is not a good thing.

Whether Intel can pull the US market out of the red has yet to be seen however. The company recently announced that their manufacturing deal with TSMC is now on hiatus due to a lack of demand among consumers for more Atom based devices. In addition, Intel seems to be favoring overseas locations to US based sites for new fab construction.

Intel is set to begin production of chipsets in a new China based facility later this year, and it seems likely that Haifa will be selected as the site of the next 22nm fab considering their gains in 2009 and the closing of the Ireland facility. Two of Intel’s six US based facilities are scheduled to be closed this year according to Semi’s map of closing/closed frontend fabs, shown below. Other major players include Toshiba, who is currently contemplating the creation of a new $8.9B fab in Japan and giant TSMC, who is holding strong despite recent challenges with wafer costs and 11 straight quarterly losses.

Although it is unclear if Intel intends to make additional investments into the remaining US fabs, hopefully the current race among NAND flash manufacturers can bolster the US market. Electronics manufacturer Samsung is pushing hard to open a new Austin facility in 2010 that will be the largest chip fab in North America. Additionally, Texas Instruments is set to begin production at its new analog fab in Richardson, TX later this year, and GlobalFoundries is planning the creation of a new facility in Saratoga, NY.

Map of Closed and Closing Fabs

View Closed/Will be Closed Frontend Semiconductor Fabs in a larger map

Don’t Begin Multi-Processor Testing Before You’re Ready

One of the core principles of Obsidian’s verification methodology is to catch every possible bug in the simplest possible configuration. That stands to reason that the single processor configuration needs to be thoroughly verified with a high degree of confidence before MP verification begins. The reason for this is simple – if the same bugs can be found in a simpler processor configuration, much less time will go to waste. Multiplied hundreds of times over, this savings in time can result in a time-to-market improvement of weeks or even months.

Beginning Multi-Processor Verification with Non-Sharing Tests

The first stage of MP testing that we implement is non-sharing. The goal at this stage is to find and correct errors that occur with the least amount of sharing as possible to simplify debugging and save time troubleshooting complex scenarios.

In the initial stages of MP verification, it’s best to divide the memory up into large partitions that are each allocated to a single processor core. Some sharing will always exist between the CPUs in spite of restrictions. This is due to the way that multiple CPUs behave upon reset (i.e. beginning the execution stream at the same reset address) and because complex MP systems typically require each CPU to be aware of their own CPU number (i.e. CPU0, CPU1, etc.).

Ideally, sharing should be minimized as much as possible at this stage of verification, although it is impossible to eliminate it altogether. If the architecture permits, it’s also beneficial to start with paging disabled to simplify debugging. This eliminates the possibility of errors where one physical address is mapped to multiple pages by more than one CPU.

With these restrictions imposed, large numbers of random test sequences are generated and debugged as needed. Once errors become extremely sparse in this mode of testing, the memory is divided up into cache lines which are each assigned to allow accesses by only a particular CPU. A more complex environment is then created in which more complex bugs can be found, such as paging errors. As before, copious amounts of random test sequences are generated and more errors will be found.

Some verification teams will completely forgo the large partition phase of false-sharing and start by immediately dividing the memory into cache lines. While this does work, it also tends to complicate debugging and forces the verification team being to prematurely deal with paging issues while debugging simple errors.

False-Sharing Multi-Processor Tests

The false-sharing arrangement is similar to the second stage of non-sharing in that the memory is divided up into cache lines. In this mode, multiple CPUs are allowed to hit the same cache lines, but not the exact same physical addresses. What this does is allow the cache accesses to come very close to each other without hitting the same spots. Several cache problems can be found and corrected using this method.

Debugging caches in this way greatly simplifies the process of testing actual shared memory and better limits the occurrence of simple errors in complex scenarios. At every stage of the process, it is important to perform cacheable and non-cacheable testing as there may be bugs lurking in both of these areas.

True-Sharing Multi-Processor Tests

True-sharing tests are more complex as they allow multiple CPUs to access the same physical memory addresses. In both deterministic and in non-deterministic modes, tests generated by RAVEN are guaranteed to execute correctly regardless of timing issues including fast or slow memory or insertion of external interrupts.

Deterministic True-Sharing

As a simplified example of deterministic true-sharing, let us assume that we are verifying a dual CPU configuration. We begin by dividing up the execution stream using semaphores. Each divided zone may contain different numbers of instructions for each CPU with varying amounts of memory accesses, exceptions, etc. Because of this, some CPUs may complete their zones more quickly than others. During the same semaphore, the CPUs are unaware of values written by the other. Not until the semaphore has been completed will the CPUs sync up again and be aware of the other’s transactions. Once this happens, CPU0 can determine that it is legal to write to a line that CPU1 completed in the last semaphore. Upon completion of all zones, final values are compared, checking for consistency and errors. Although our rules are much more complicated than this, the example should elaborate on the kind of errors that can occur in MP verification. Because our tool implements complex rules, the test generated can determine if memory location will be deterministic or non-deterministic based on execution divided by semaphores. This is how we do deterministic sharing and actual reads and writes to the same physical memory addresses. This can find a lot of bugs.

Non-Deterministic True-Sharing

Non-deterministic true-sharing is the most complex mode of sharing and should always be conducted as the final stage of MP verification. In this method, we don’t use semaphores to divide up the execution stream, but instead allow CPUs to read addresses that are being written by other CPUs. If two CPUs write to the same address, then a read of this address will result in an unknown value. If this value was intended for use in address generation or arithmetic calculation, then the result of these tasks will be unknown, and verification becomes much more difficult. Non-deterministic registers then cannot be used as sources, but only destinations until they take on deterministic values.

If non-deterministic transactions set flags, then the flag register must immediately be re-written. This is especially important in ARM processors where every instruction is conditionally executed. If something is done that results in a flag, the next instruction must be set to always execute regardless of flag values and perform the additional step of re-writing the flags. We have settings and rules that determine what percentage of registers are allowed to become non-deterministic based on their type (i.e FPR, GPR, etc.). So we do have some pretty complicated algorithms for handling determinism and non-determinism and determining legality of actions.

Non-deterministic tests become much more difficult to debug because you will have a list of possible values for each access and the errors can be timing dependent. These areas are important to cover, but they should be done last as the difficulty in debugging errors of this type is extremely high.

Verifying the Snooping Mechanism

One of the most critical areas in MP verification is cache coherence. Errors of this type can be detrimental to practical operation of the design. For example, it is possible to have errors in which data integrity is compromised due to multiple CPUs accessing shared memory.

To further illustrate an error of this type, let us assume that we are verifying two CPUs that both share a single L1 cache. At some point the L1 cache will have an image of data that resides in the main memory. If both CPUs write to the same cache line of the L1, then it is necessary to verify that the correct values get written back to main memory.

Adding an L2 memory to a situation such as this makes matters even more complicated. In this case, cache lines in the L1 will be an image taken the L2, which itself contains portions of data in the main memory. However, there is now the additional problem of L1 data being written back to main memory without crossing the L2. This creates a corresponding line of L2 cache that must now be invalidated to avoid errors.

Further conflicts can occur if a CPU flushes a cache line in the L1 that another CPU has written to, was going to write to, or is in the process of writing to. Fox example, suppose that we have a four CPU situation with two L1 caches, each being shared by a pair of CPUs and an L2 that is shared by all four CPUs. This is a common paradigm in MP designs. In this situation, it is physically possible for CPUs 0 /1 to write to the same line of memory as CPUs 2/3, creating another situation that has to be resolved. Each cluster of CPUs must then ask the other if for permission to read/write to main memory to avoid conflicts.

Verification of Shared Registers

One of the great things about having a random test generator is that it can effectively deal with obscure cases and propagate them across multiple CPUs. In the case of shared registers, we can divide them up into an arrangement similar to our true sharing example or allow only one CPU to handle all the shared registers. We can also use semaphores to do time/execution division of accesses and do non-deterministic testing, although this depends on the type of register. Although shared registers present some unique challenges, they also allow for some different opportunities to take advantage of.

These are all complicated issues which may be further compounded by timing issues. Errors such as these are what MP verification is really about and where the bulk of the errors can be found.

The early stages of functional verification present several unique challenges that must be overcome to achieve first silicon on schedule. This article identifies several common problems that Obsidian recently encountered in the verification of a new ARM based processor design and discusses how the RAVEN random test generator was used in overcoming these obstacles.

Eliminating Bottlenecks

Many organization want to begin verification as soon as the first logical unit of RTL becomes available. However, the tool-chain (i.e. assembler, linker, compiler, etc.) often lags behind RTL development and architectural changes, creating a bottleneck in verification. RAVEN allows verification teams to completely bypass this problem and begin identifying functional errors before the tool-chain is implemented. Customers using RAVEN can have reported large productivity gains over directed testing, especially in the beginning and intermediate stages of verification – hitting 95-98% of all required coverage points using random tests.

Verifying Intermediate RTL

Let’s assume that the ALU unit of your design is ready for testing at an early stage, but the MMU, floating-point and load/store units are not. Some teams might begin by testing each instruction in order, varying only one or two operands for the sake of expedience. However, this creates a problem. Following this methodology, only a small portion of the RTL will be tested and the entire design will be exposed to errors that should have been corrected early on. With the same investment in time, RAVEN can produce significantly better results with only slight modifications to the default template.

Start Test
Random Instruction (ALU)………………..min=10     max=20
End Test

RAVEN template files are used to designate which behaviors should be tested. At the most basic level, these templates provide the ability to interchange instruction groups, instruction quantity, and selection of operands. More advanced configurations allow for the biasing of certain exceptions and addressing/operand modes. This ability is important in the sense that it allows additional behaviors to be added incrementally as bugs are discovered and new areas of the RTL become available in emerging designs.

In the case of our ALU example, the Random Instruction control can be directed to use only instructions in the ALU group and can even disable the selection of individual instructions within that group. Available instructions can then be tested with an almost infinite number of parameter configurations. As further instruction groups and processor features become available (e.g. integer divide, MMU, load/store) RAVEN templates can be automatically updated to include these new behaviors. Simply re-generating the same templates then allows for the creation of thousands of new tests.

Planning for Bugs in Your Testing Methodology

As you’re creating your first tests, it’s important to think about how long they’ll take to simulate and debug. Smart design teams take into account where they are in the development cycle and the amount of processing power available for simulation when creating tests. This way tests can be run and completed within a targeted amount of time.

When a generated test hits a bug, it’s also much easier to isolate and identify the error if it is detected in a test with ten instructions rather than one with a thousand instructions. Even if you hit a bug in only one out of every thousand tests of ten instructions, this is still beneficial when compared to the time required to debug large tests.

Once you’re able to generate 10-20K short tests without finding errors, then it’s time to move up to tests of a hundred instructions and eventually on to tests of a thousand instructions. The reasoning here is that the more rare your bugs then become, the more dependent they will be on processor state, and longer tests will be more useful in this respect. Changing the quantity of instructions in a RAVEN template is as easy as adding extra zeros and regenerating.

Once bugs are found, RAVEN can be configured to continue testing around the error by disabling access to problematic behaviors in the RTL. Most internally developed generators don’t have this fine-grained level of control. It’s far easier for RAVEN to hit new behaviors while avoiding others that aren’t yet ready to be tested than for most other EDA tools on the market.

Sure, self-checking code can be used with directed tests. But it’s time consuming, cumbersome and there’s no randomization. RAVEN has several features that make it work really well in a silicon validation environment for creating self-checking tests.

Inherent Self-Checking

Taking them one by one. If you’re leaving RAVEN undirected, which is usually what you want to do, then what gets intermixed are lots of jumps that depend on the previously generated values. Since tests generated by RAVEN can go anywhere and do anything, it’s not uncommon for complex tests to end with a jump off of some value, leading to another instruction sequence. So, inherently RAVEN tests are self checking.

Inside of a random test, you’ll usually have lots of jumps. If any of the calculations leading to those jumps change, then you’ll jump off into an area that you don’t want to be fairly quickly because some calculated value went wrong. Depending on your architecture, what some companies will do is to preload the memory area so that undefined instructions will cause traps resulting in a fail. Now you have to go into your silicon and figure out how you got to this point, but at least you know that you have a failure. Directed tests won’t have the same results as RAVEN because the engineers writing them won’t jump off of their results into strange places. It’s too hard for humans to use the results of calculations as jump targets, but for RAVEN it’s actually quite easy.

Configurable Self-Checking Features

The second thing that users can do is to turn on RAVEN’s self-checking feature, which includes a number of options for how you want to do self-checking. This feature tells RAVEN to insert self-checking code, much like a directed programmer would do to validate something, like a series registers. Since RAVEN knows when registers are updated, we can tell it to check all registers to make sure that their information is valid. Alternatively, we can check only the registers that were written or read. We can do this check after a preset number of instructions, at the end of the sequence, or it can be randomized to occur between a certain number of instructions.

Adapting Self-Checking Tests to Fit the Hardware Environment

For many customers, the actual test or hardware that they are operating in is embedded in an SoC with some test mode that allows them to bring signals out. But their environment is very different from the environment of their simulation world. In simulation, they may have a large amount of memory to use for testing. In this new environment though, they may want to restrict the tests to use only the on-board device memory. This might be a only very small amount, let’s say somewhere between 256K and 2MB.

Because RAVEN has configuration files that describe the environment that the chip is in, you can move tests originally written for a 4GB address space into 1MB of memory. RAVEN can then re-generate all of the tasks from your templates, forcing them into that much more constrained memory area. Now you can take your whole suite, and probably with some exceptions, regenerate all of your tests toward your real hardware platform and even re-simulate them again in their original environment with slight modifications to mimic what the hardware will look like. If all of these tests can be successfully run at full speed, then there is a high degree of confidence that the model is accurately reflected in the design and that there won’t be hidden problems in the silicon.

Ability to Verify RTL and Instruction Set Simulator Agreement

Another, more comprehensive method of self-checking in the RTL environment is the intermediate state information provided by RAVEN. Our test output files contain information about all updates to registers and memory that occur as a result of instruction execution. The testbench can be instrumented so that there are checkers that watch registers and memory to make sure that they progress through values predicted by simulation. This allows the testbench to detect the discrepancy in the exact instruction that caused it or within just a few cycles of that instruction, greatly reducing the time required for a verification engineer to isolate the problem.

When a load/store instruction crosses a page boundary, it is difficult to create all possible combinations of exceptions for both halves of the instruction. For example, if 2 possible exceptions exist then there will be 16 possible combinations of exceptions for 2 halves of the access. Because of this, it can be very hard to reach these exceptions, even with directed testing.

Obsidian’s RAVEN technology addresses this issue with its random methodology. User configurable biases may be used to direct the generator into areas where data access instructions may cross page-boundaries. Difficult to reach errors such as these can be uncovered with minimal effort and without diverting your greatest resource, the time of your most experienced engineers.

Confirmed Panelists include:

Ty Garibay

• Ty Garibay is the Program Manager for ARM Processor development in Texas Instruments’ Wireless Terminals Business Unit and site manager for TI’s Austin office. Currently, he and his team are focused on the completion of the industry’s first 45nm implementation of ARM’s Cortex-A8 super-scalar processor core.
• Over the previous 20+ years, Ty has designed microprocessors at ARM, Alchemy Semiconductor, SGI/MIPS, Cyrix and Motorola, participating in all phases of design from circuits and layout through architecture and product definition.
• He holds over 30 patents in the areas of integrated circuit design, computer architecture and design methodology.

Brian Wong

• Mr. Wong was President and CEO of D2Audio Corporation since 2005, a leading audio semiconductor and software company. D2Audio was recently acquired by Intersil Corporation in August, 2008. He is currently running the D2Audio-Intersil business as Director of D2Audio.
• Previously, Mr. Wong was CEO at Primarion Inc, a company focused on data communications and Power Management ICs, which was acquired by Infineon.
• Prior to that, he was a senior manager at TRW and managed the Mixed Signal IC business, which included data converter, Clock/Data Recovery, PLL, optical datacom, and high speed digital ICs.
• Mr. Wong holds a BSEE from University of California, Los Angeles, a MSEE from University of Southern California, and has taken graduate management classes at UCLA Anderson School of Management. He has co-authored textbooks and papers on semiconductor technology and data converters. He is the Chairman of The Austin Technology Council, serves on the Advisory Board for the UC Davis ECE Department, and on the board of Integral Wave Technologies, a power management company.

Jim Reinhart

• Jim Reinhart is President and CEO of Luminary Micro and one of the company’s three cofounders. Jim has more than 25 years of experience in the creation and commercialization of computing technologies. A sixth-generation Texan with degrees from Rice and St. Edwards Universities, Jim will give a talk on the changes that are driving a massive shift in the semiconductor industry.
• This event will be hosted by Eric Hennenhoefer of Obsidian Software and moderated by Steven Schulz of Silicon Integration Initiative.

Registration

DVClub membership is free and is open to all non-service provider semiconductor professionals. Most members work in verification, but there are also plenty of entrepreneurs, professors, students, managers, investors, and even design engineers who attend. Participation by service providers (solicitors) is limited to event sponsors, who supply the funds for DVClub events.

Help us plan for a proper setup and RSVP here.

10th International Workshop on Microprocessor Test and Verification (MTV 2009)
December 7-8, 2009, Hyatt Regency On Town Lake, Austin, Texas, USA.

Website: http://mtv.ece.ucsb.edu/MTV/

This is the 10th edition of the MTV Workshop, a testament to its success in providing an ideal environment for cross- examination of test and verification experiences and innovative solutions. MTV has been held in Austin for the last 8 years, so please plan on participating in order to make this another successful forum.

Purpose

The purpose of this workshop is to bring researchers and practitioners from the fields of verification and test together to exchange innovative ideas and to develop new methodologies to solve the difficult challenges facing us today in various processor and SOC design environments. In the past few years, some work has been done on exploiting techniques from test to solve problems in verification and vice versa.

Topics

AREAS OF INTEREST include, but not limited to:

• Validation of microprocessors and SOCs
• Test/Verification of multimedia processors
• Performance testing
• High-level test generation for functional verification
• Emulation techniques
• Silicon debugging
• Formal techniques and their applications
• Verification coverage
• Test Generation at the transistor level
• Equivalence checking of custom circuits
• ESL Methodology
• Virtual Platforms
• Software verification
• Circuit level verification
• Switch-level circuit modeling
• Timing validation techniques
• Path analysis for verification or test
• Design error models
• Design error diagnosis
• Design for Testability or Verifiability
• Optimizing SAT procedures with applications to testing and formal verification

Important dates

Submission: Sept 1, 2009
Notification: Oct 1, 2009
Final version due: Nov 1, 2009

Design productivity is tied to the ability to quickly generate large numbers of tests. RAVEN can be implemented in nightly test suites to continually generate hundreds of thousands of tests. Processor designs are verified with less time and effort than with alternative verification methodologies.

Fewer Respins

Obsidian can help clients eliminate costly manufacturing respins due to functional errors. Using a dynamic random test generator ensures that more of the design will be tested with better coverage, specifically around difficult to reach corner-cases. With proper deployment and training of RAVEN, verification teams can save the equivalent of one full respin for each design project.

Time-to-Market Advantage

Fully, 55% of embedded processor designs do not meet their scheduled release dates. Being first-to-market among competitors can be extremely advantageous for processor designers as revenues are typically doubled during the extra time period.

Cutting Verification Costs

If Obsidian can cut verification time by a modest 10%, the end user can claim a 5% reduction in total design costs. Based on the $50M cost of a 45nm processor, this element alone would result in a savings of $2.5M for the design project. These numbers will be higher for more complex processors and with better time savings.

This article presents an overview of functional design verification using a coverage driven methodology while attempting to answer the question of how much testing is enough. The part being verified in this case will be a general purpose microprocessor, such as those found in mobile computing devices. Note that an approach of this magnitude is not always required. Designs with very limited instruction sets or highly restricted functionalities may be satisfied by simply writing directed assembly code tests to verify their intended functionalities.

Comparison of Simple and Complex Architectures

Figure 1 depicts a simple architecture as compared to a complex one. Note that the number of corner cases and unpredictability of the verification space increases as the architecture gains complexity. Thus, the complexity of the architecture determines how much testing will need to be accomplished to properly verify the component’s function.

Figure 1. Comparison of Verification Spaces

Measuring Verification Progress

Coverage metrics are the dominant method for measuring verification progress in the industry today. Coverage points are normally designated by the design engineers looking at the logic of their block and by verification or system engineers looking at the functional definition of the part. Both of these are critical insights into the required verification coverage of the design.

Coverage points, indicated by the red dots in Figure 2, are deliberately chosen with respect to placement and density according to design knowledge and risk assessment.

Figure 2. Distribution of Coverage Points

Directed Testing VS Random Methodology

Some organizations will use only directed tests to hit coverage points. Because directed tests are by their very nature highly targeted and relatively inflexible, this results in much of the design not being tested as is shown by the ratio of red to gray in Figure 2. In addition to the low overall coverage that results from this approach, creation of directed tests is time consuming, requiring approxomately 20-30 minutes per test, and requires highly skilled engineers. In this approach, testbench checkers that detect hits to coverage points are often overlooked with the assumptions that the engineers writing the tests know how to hit the required coverage points and that human errors will not a significant problem. As the design changes over the course of RTL development, directed tests may lose track of their targeted coverage points. Without coverage monitors, these types of errors will not be detected and the design will not be as thoroughly verified as it appears to be on paper.

Using a Random Test Generator to Close Coverage

As processor designs became more complex, the need to hit more coverage points became apparent. Once the grid has been established, large numbers of purely random tests may be incorporated to begin closing coverage. Some of these tests may hit points on the coverage grid while others will not.

Figure 3. Intersection of Coverage Grid and Pure Random

Hitting Corner Cases

Approaching the problem of hitting coverage points from a random test generator viewpoint, a single engineer begins by writing a few generator templates and then generates tests using those templates. The generated tests are then run on a testbench which incorporates coverage monitors. The coverage monitors report all coverage points that are hit by the tests. As long as tests generated from the templates continue to hit new coverage points, the templates are kept in the nightly suite. As the rate of hitting new coverage points declines, new generator templates are created to target coverage holes. This approach requires skilled engineers to write checkers for the testbench but less skilled engineers to run the test generator.

Directed-random templates are created around points not hit by the purely random templates. We now begin to see the coverage grid closing more tightly (around 95%), and the verification process comes closer to completion.

Figure 4. Coverage Grid, Directed Random and Pure Random

Hitting Corner Cases

Not all coverage points will be hit by fully random or directed random templates. Some coverage points require a long series of events before the targeted behavior takes place. In this case, there are two possible approaches: write directed tests and write directed templates. Directed tests can get to these most difficult coverage points more quickly but prove only one or a few cases around that point. Directed templates take more time to create but can be written to allow as much random behavior around the coverage point as possible.

Figure 5. Review Templates and Relax Restrictions

Finally, existing tests are reviewed, and as much directed behavior as possible is removed before the tests are run again. Coverage then reaches full closure, and these tests are run until the schedule no longer permits.

This blog post originally appeared at:

http://www.chipdesignmag.com/martins/2009/03/19/bailey-on-verification-at-the-club/

— March 19, 2009 @ 11:14 pm

Today I attended the latest meeting of the Silicon Valley branch of the DVClub. For those not familiar with the DVClub (DV = Design Verification), it was started by Eric Hennenhoefer in Austin a few years ago. It now has branches in Austin, Bangalore, Boston, Bristol, Dallas, RTP, San Diego and Silicon Valley. In Silicon Valley it meets about once a quarter for a talk on some aspect of verification. I first heard of this about 1.5 years ago when we were invited from Tensilica to give a talk about verifying our video subsystem. The club has all the right ingredients to attract a crowd of engineers:

1. a free lunch
2. interesting speakers
3. did I mention a free lunch?
4. a chance to meet new colleagues and old friends
5. and of course, a free lunch

(Sponsors such as Cadence, Doulos, Denali, Silicon Elite and Obsidian pick up the tab for the venue and lunch (updated after original post, on Friday 20 March 2009, to correct list of sponsors)).

Today’s speaker was Brian Bailey, a friend and co-author of mine, speaking on “Is it time to declare a verification war?” The place was packed out with about 130 people, filling the room to capacity (Eric said this was the largest Silicon Valley DVClub crowd to date).

Brian Bailey

Brian spoke about his philosophy of verification, drew some analogies to Sun-Tzu’s Art of War, and also spoke about three technologies that he felt had potential to change verification significantly:

1. Functional Qualification – as exemplified by Certess (now SpringSoft) Certitude
2. Raising abstraction – as exemplified by Calypto’s sequential equivalence checking
3. “Intelligent testbenches” – as exemplified by Jasper’s Behavioural Indexing

Brian’s slides are available here.

IF you live or work anywhere any of these branches of the DVClub, and have an interest in verification, I recommend that you check them out. Sign up for their newsletter and get notified of meetings in advance.

In many cases, a processor design team will write a simple test generator for the first phase of a project and gradually evolve it into a more advanced form as the architecture matures. This continual evolution of test generator technology stems from several causes:

• Earlier designs tend to be simpler with later revisions adding more features and complexity.

• Later designs may prove complex enough to require a new approach.

• The verification effort may initially be underestimated.

• Estimates become more realistic over time as they become based on knowledge gained in earlier revisions.

• Products that go through several revisions and enhancements are likely to be those that have proven successful in the market and these tend to have better funding for both design and verification.

Table Based Generators

Table based test generators are the simplest possible generators. Creation of such generators can be accomplished relatively quickly, and maintenance requirements are often low. These generators work by capturing ISA knowledge and storing it in a central table for later use. Because of their simplistic nature, table based generators may be used by less skilled personnel to create tests. There is a drawback to these generators however, as their implementation is generally restricted to simple architectures. Usage on more complex designs may result in an inability to reach corner-cases or create complex scenarios. Table based generators may also generate invalid tests at times.

Static Generators

Static generators are similar to table based generators with the exception that the majority of the instruction, operand and data selection reside in complex procedural code. Static generators are capable of producing more random behavior than table based generators, but still have trouble hitting many corner-cases. In addition, the skill level required to create and maintain such a tool rises sharply once this level of sophistication is reached.

Dynamic Generators

Dynamic generators incorporate significant knowledge about the architecture being tested. They enhance the ability of less-skilled users to generate complex tests that can hit hard-to-reach corner cases without stumbling on subtle programming pitfalls. This added knowledge, flexibility and ease-of-use is reflected in a more complex generator and consequently the cost of creating and maintaining the generator are greater than for table-based or static generators.

Obsidian Software’s RAVEN is a dynamic random test generator that has been developed and maintained by processor verification experts since 1997. During this time RAVEN has been used to verify dozens of processor implementations by design teams across the globe.