Alex Melikian: The next topic I want to cover is coordination between teams. We mentioned earlier about the involvement of software teams or other departments who have a stake in the product and its features. One procedure that is brought up in the planning stages involving coordination between teams is the “Deep Dive”. Can you describe that for those who are not familiar with this approach?

Jason Sprott: Well, there’s various techniques you can use to gel teams together and get everyone in the mindset of more joined-up thinking. One good technique is to have the product, marketing, software, various design, and verification teams work together to think about the end product. A “deep dive” is when you throw these guys together in a room for a while and let them tear through requirements and functionality. You give them some time to mull these points over in their head and hope they actually do it and don’t turn up to the meeting cold. The idea is to get as much input from the wider team as possible, and have access to the people who know both deep implementation and customer level requirements.

And start them with what someone thinks is the most important features of the design and then work it through the stuff, so that you have the people, the major stakeholders, for all the different components in the system.

You have them in the room, so there’s even guys in there who could say, “Hey, wait a minute. That feature isn’t important at all. Yeah, let’s take a note to speak to the client about that because the guy we’re going to sell this to might not care about that.” Another thing someone might say is “Now that could you go into the next chip”. So at one end, there’s people in the room who can look at priorities, who can look at features, who can look at the ways things are used. And at the other end of the spectrum, there are guys in the team who can assess an impact of a given feature.

So in this setup, someone might say, “Oh, yeah, we’ve got to have this feature in the design.” At the same time, someone from the verification team can respond with “Fair enough, that will take you only ten minutes to implement in a piece of RTL, but it’s going to take two weeks to verify that, is that what you want? Is it worth two weeks?” That’s the sort of interaction that takes place and you would like to happen.

The advantage of having the stakeholders for the different components of the product in one place, is that you can often get to the bottom of decisions made for specific requirements that may require a lot of effort. And of course, these decisions are being made, before the project is being carried out, hence all teams go in eyes wide open. It’s also a good opportunity to discuss the interaction and expectations between components. It all sounds so trivial, but many costly project “surprises” happen as a result of not taking these things into account. When these “surprises” happen, the best case scenario is additional effort will have to be spent, and the worst case is that they manifest as an issue in the final product.

AM: It’s strange: I think a lot of our readers when told it’s good to spend half, or even a full day of planning for a chip would, probably find that excessive. However, if we compare that amount of time to the overall time of a project, you realize the time spent for that full day is well worth it. It can really pay off over the course of the project, because not only is there more coordination and understanding across all the stakeholders, but you can also avoid these very costly “surprises” in the future.

JS: That’s right. You don’t want spend too much time though, because things change. You could approach this meeting by asking the participants to come up with the top five things that they care about in this design. That would be a good start, and then if you have time left, you’ll cover more. It must be emphasized to the participants to come in prepared, like with some of the top things they care about, and why you care about them. That will definitely drive some discussion.

Also the other thing you really need when you’re going in a deep dive situation is historical data. This can really help making more sensible and informed decisions. If you don’t have anything, meaning it’s all just made up on the day, and relies solely on the knowledge and skill of the people in the room, then in situations like this, decisions can be affected by anything from vague recollections, emotions, heated debate, bullying, or even hunger, like when participants ask “how close is it to lunch”.

AM: That’s funny, but true. I’m glad you mentioned ‘historical data’, as one of the ways that data can be generated or archived would be through postmortem meetings. So what are some of the things you think makes a postmortem very valuable to a team? And what is the specific knowledge a team should be retaining or archiving when they’re doing a postmortem?

JS: I like the postmortems, but in fact, what I prefer is to do both peri and postmortem reviews. Some things you can only know at the end of the project, but it’s also possible to take regular retrospectives along the way. These can be used to capture more detail and affect decisions within the scope of the current project, unlike a postmortem.

You can record many things, but it’s bad only to focus only on the negative. It’s interesting to record how and why things went well, not just things that went wrong.

It’s always good to get to the root cause of things. For example, if stuff goes wrong because the code wasn’t in good shape, the root cause might not be the code itself. The root cause might be that we didn’t do very good peer reviews, or we did the peer reviews too late. During in-project retrospectives, it’s sometimes possible to see patterns of things going wrong, also known as anti-patterns, and fix them within the scope of the project.

In terms of the statistics that you want to archive, you should look at things like bug rates, the type of bugs you had, where the bugs were found, how long it took to turn around the bug fix, simulation times, development times. You also want to record the project conditions that might affect those metrics, such as the amount of change to the design, time pressure, or skill level in the development team. For example, you might expect to get very different results from an experienced team than from a team that have never worked together, or is inexperienced. This is important data that can be used to weight the metrics recorded.

Since everyone always wants to know, “How long do you spend planning?” it’s always useful to record that accurately in project records.

Postmortems, should also always want to look at your reuse. What did you actually reuse from a previous project? And by that, I mean what did you use without modifying the code itself? Did you have to hack about with it?

And also, what did you output from a reuse point of view? What did you give back that other projects may be able to use? That’s very valuable information.

What I would say is at the end of the day, the one thing you should really care about is that you have a clear picture of what you keep for the next project and what you dump.

AM: I think you summarized it well. Postmortems, or peri-mortems, are very good ways of applying the old virtue of learning from your mistakes and experiences in our business.

Love to hear more but sadly that’s all the time that we have for this conversation. Thanks a lot Jason for your time and your input. I hope our readers have better appreciated the human side of best practices in verification. Look forward to the next time.

JS: Okay, thank you, Alex.

Alex Melikian: Lets move on to other facets coming into play in today’s modern day silicon product. We’re seeing things like power management, mixed signals, or analog mixed signals involved in making a product. How do you see these new areas affecting how a verification plan is put together?

Jason Sprott: Well, I guess now low power is a fairly well understood concept for verification, and we have power aware simulation and the like. However, what people sometimes fail to understand is that low power features in a design are a massive crosscutting concern. They can slice across all aspects of functionality. So instead of having a piece of functionality that’s exercised only in one mode, it could need to be exercised in many modes for different power scenarios. And that can explode into a massive state space. So I think this is another area of ruthless prioritization, where you can’t possibly verify everything in all different modes. Usually, that’s just too onerous.

So you have to look at what really matters for the low power modes you’ll be using. So I think you have to ask yourself is: “Well, in real life, how would the software control the power modes?” You often have to work hand-in-glove with the software department to keep a tight rein on the verification requirements that end up in the product.

And it has to be well understood that if someone changes the software to do something else, it could have an impact by pushing into areas that haven’t been verified. I think this is an area that really, really needs to be tightly tied down.

A completely different area of verification and much less understood is analog verification. That can be the functional border between the analog and digital domains, or true analog functional verification.

We have to consider what level of accuracy and performance we build the analogue models to. This is an area that will have an increasing effect on verification as we go forward. We haven’t really tied down the languages everybody should use for modeling and for doing the verification. And as much as we understand functional coverage in the digital domain, what does it mean for analog verification?

You have to really tie down the requirements of the crossing between the domains. Sometimes, analog isn’t that complicated in terms of the number of features, as compared to digital designs, but you have lots of discreet signals with independent timing. This can add up to a lot of combinations to verify. Not all combinations are valid, or possible. Understanding what, or if anything, can be cut back in this domain is essential to make prioritization decisions.

I think one of the biggest things to come out of analog functional verification, is a more considered approach to modeling. The accuracy, performance, and validation of the models against the circuits, are going to play a bigger part in verification in general. All of these things are being demanded now of the analog verification team, whereas in the past, they weren’t given much consideration. There may not have even been an analogue verification team. Chip level simulations often ended up with models of the wrong accuracy (typically too accurate for the context), or were buggy and not validated.

Chip teams up until recently haven’t been asking for proof that analog models have been verified against a circuit, or specified requirements for the models in terms of performance and accuracy. So yeah, this is a big area and it’s going to have quite an effect, I think, going forward.

AM: Interesting to see how ‘prioritization’ is applied in the context of analog modeling, that is to say how close to the real thing you would really want it to be and keeping in mind the implications on the schedule.

We’ve been talking about the many elements involved in the process of planning throughout this discussion. Let me bring up a question that would cover this theme from a different perspective. Would you agree that one approach that engineers have to take when they’re doing their planning is applying a risk assessment for each feature as opposed to thinking along the lines of step-by-step procedures?

JS: Risk assessment is certainly one aspect of planning. I think what you really have to aim for with risk assessment is better predictability of results. Risk and priority are two separate things. You can have something that’s a very high risk but very low priority and vice versa. So I think risk assessments are part of the planning, but how far you want to go depends on your product requirements. Some designs can tolerate more risk than others. For example, high reliability designs for automotive, or medical, typically require a much more detailed analysis. This is relevant to planning, as you have to ensure you don’t eliminate necessary work from the schedule. Sometimes this work is only highlighted by risk assessment.

I think you’ve got to decide how much risk assessment you do, and prioritize the risks, as well. But you do need to know about all the features in order to do these things. So you can’t just look at them in isolation and say, “Ah, yeah, we’re going to consider all the risks up front without knowing what the features are” because the risks are related to the features. So you need to do both.

In Part 1, Alex and Jason discuss the concept of “ruthless prioritization” and the differences in practices between FPGA and ASIC development.

Alex Melikian: Hi, Jason. Thanks for joining me on this conversation. Today, we’re going to be talking about the ‘human’ side of best practices in verification. I emphasize ‘human’ in the title because this topic of conversation will not focus so much on tools, technical issues nor coding details. Rather, we take a closer look in the way we do the day-to-day human activities related to verification. I’m talking about things like planning, team coordination and cooperation.

So Jason, you just got back from DVCON 2013, where you were one of the panelists on the discussion of “Best Practices in Verification Planning”. Let’s get started with this conversation with one of the points you greatly emphasized at the panel you called “ruthless prioritization”. Talk about this a little bit more. How should managers or engineers execute this approach of “ruthless prioritization”?

Jason Sprott: Well, I’m glad we’re talking about the human factors because I think they play a major part in team productivity. This “ruthless prioritization”, as I call it, is something that’s difficult to automate, maybe even impossible.

There’s a lot of spin-around things like prioritization, project planning etc. Ruthless prioritization, is when you make sure that as much work as possible goes towards the features and parts of the design that really matter: the ones that the end users are going to use. These are the things that you’re taping-out and people are going to notice if they’re broken. Whereas, what we tend to do in designs, not just verification environments, is to design in a lot of things that may never be required.

There are many reasons we focus on unimportant features, but at the end of the day they can be a major distraction and don’t necessarily matter to the end result. And the problem is we’ve got to verify all those things, or at least spend time considering them. So for me, part of the planning process and the human aspect to the planning process is to ruthlessly prioritize. Not just once at the beginning but continually through the project. The aim is to ensure all the things we’re working are towards the highest priority product goals at the end of the day: the things that really matter. That’s not an easy task, but it’s worth it.

When thinking about the priorities, you have to really consider them at all stages of the development. It’s not something you just do at the very beginning. You’re continually testing: “Is this something I should be working on now?”, or “does this affect something that will definitely make it to the final product?”

If the answer is “no”, you’ve got to ruthlessly throw it out. Otherwise you may be working on stuff that doesn’t matter and you’re just burning development cycles.

AM: I can relate to that: coming from more of an FPGA background myself, I see some of the parallels in the FPGA development process. Not so much, as you said, in ruthless prioritization, but rather setting goals on how much verification you want to exert for each feature. For FPGAs, the name of the game is ‘time to market’. So you can allow yourself to make mistakes before going to the bench in the lab. You don’t have to do 100 percent coverage and test everything down to a ‘t’. Similarly, you have to set priorities.

Of course, there are some parts that are critical, and testing for 100% coverage would be beneficial. However, there are other parts where you can take the risk and aim for 80% coverage in simulation, as long as you have a good bench available in the lab carrying out the exhaustive testing. It’s counterintuitive for us verification engineers to allow a design to go into the lab with the possibility of bugs. However, by carefully allocating some of the validation effort into the lab, I think more often than not, you will achieve complete coverage without running into a bug that requires additional time to debug in the lab. Therefore the overall time would be less, than if you spent the effort to verify everything in simulation with 100% coverage. You know what they say, 80% of effort will be spent chasing the last 20% of coverage. So some time saving can be done there when dealing with an FPGA.

This decision making process of how much verification should be done in simulation also involves continual planning. This means that in mid-project it can be decided that testing a certain feature can get pushed to the lab, or conversely, it becomes necessary to simulate it with 100% coverage.

So this management approach has a lot of parallels to the “ruthless prioritization” process. Do you have any thoughts about that?

JS: Really, it depends on how rigorously you want to verify your design and on the risk you want to take. Just because something can be eyeballed on the bench, doesn’t make it the right way to verify it . I think it’s very, very important in FPGAs to understand what you’re deferring to the bench testing rather than simulation.

The things that you want to push on the bench are things that can’t be done easily in simulation, or things that make more sense for performance, or practical reasons. On the bench, controlling the design to put it in states that exercise all user scenarios can be very difficult. It can often be difficult to exactly duplicate these conditions when regression testing.

It’s just as important to plan and prioritize which features will be verified in real hardware, as it is in simulation. It’s just as easy to waste time on features that are not important to the final product on the bench. In fact, if you factor in repeatability issues, hardware setup, and phony debugging, you can end up wasting a lot of time on unimportant features.

There are also things we can also do in FPGA designs that aren’t typically possible in ASIC, such as building verification modes that can operate in simulation and on the bench, e.g. short frames. The mode is only a mode ever used for verification, and creates something short enough or concise enough to simulate, but it can also be repeated on the bench. Although typically not a real mode of operation, it does exercise areas of functionality very completely, that we would otherwise not be able to simulate if we deferred the whole thing to the bench.

So, don’t just drop stuff onto the bench assuming that the simulations will be too long. Take a more pragmatic approach and analyze what’s actually required, understand the risks you’re taking and find ways to mitigate them.

AM: Definitely agree. Your points emphasize how a lot of coordination is needed between the verification plan and the laboratory validation plan, meaning the planning of what gets tested in the lab. This is absolutely key if the strategy of splitting the burden between simulation and lab testing is to be successful in saving verification cycles.

This presentation shows a novel, more complete approach to functional verification of sub-cycle timing using SystemVerilog assertions in an OVM verification environment. This approach found many bugs which otherwise were missed in OVM-only simulations that didn’t include assertions.

This functional sub-cycle timing behaviour includes maintaining fixed delays and phase relationships between inputs and outputs and ensuring there are no glitches on clocks or delayed signals.

SystemVerilog assertions are evaluated on successive occurrences of an event or timing expression. This presents a challenge for sub-cycle timing verification, where there is no obvious reference clock suitable for triggering the assertions. Assertions sample their expressions in the preponed region of the simulation timestep, but the requirements called for sampling both before and after each triggering point. Examples of assertions showing how to overcome this and many other issues will be shown along with recommendations on how to write assertions for functional timing verification.

This paper is complementary to the paper presented by Paul Marriott at DVCon 2013 entitled Run-time Configuration of a Verification Environment - A Novel Use of the OVM/UVM Analysis Pattern. The sub-cycle timing relationships were dynamically varied during simulation and the assertions used were required to check for correctness as the actual relationships varied during the simulation.

Mark Litterick won the best paper award for his paper entitled SVA Encapsulation in UVM - Enabling Phase and Configuration Aware Assertion

The slides for Mark’s presentation can be downloaded here: SVA Encapsulation Slides

Paul Marriott presented a poster entitled Run-time Configuration of a Verification Environment - A Novel Use of the OVM/UVM Analysis Pattern and the associated paper can be downloaded here: Run time configuration paper

Mark also presented a poster entitled Pragmatic Verification Reuse in a Vertical World and its associated paper can be downloaded here: Pragmatic reuse paper

As well as winning best paper and presenting a poster, Mark also gave his thoughts on OVM to UVM Migration - or There and Back Again, a Consultant’s Tale which formed part of the tutorial on Lessons from the Trenches: Migrating Legacy Verification Environments to UVM

Peggy Aycinena wrote an interesting piece on Jason Sprott’s participation in the Cadence-sponsored luncheon panel on Best Practices in Verification Planning.

Finally, check out Richard Goering’s writeup of JL Gray’s Industry Leaders Panel: “The Road to 1M Design Starts”

This presentation provides an overview of the Kanban technique and guidance on how you can start to use it in your team’s ASIC/FPGA development flow.

The key advantages of using Kanban are that it provides a visual overview of what your team is working on, clearly identifies current blocking issues and bottlenecks in your process, and creates opportunities for the team to proactively discuss their processes and resolve any blocking issues.

Alex Melikian: Changing gears a bit, and I know I risk dating you here, but you’ve been around for a while and seen a lot of languages come and go. And of course some of them have stuck around. The verification languages that we mentioned at the start of this conversation were not the only ones that have appeared. There have been some attempts by third party groups, some of who have constructed and publically released their languages for verification.

For those that didn’t catch on, what do you think are the reasons they failed to capture the interest of the verification community? Or, asking this from another angle, what elements to a verification language are absolutely necessary for it to be considered viable and worthy?

Jonathan Bromley: Well, beware of my personal bias here, obviously, because for one thing I’ve been heavily invested in SystemVerilog standardization for some time now. And for another thing, I’m personally a little conservative in my nature, so I would say that the two things that I would be looking for in any verification tool are completeness and standardization. Completeness is required because I don’t want to have to reinvent wheels for myself. I don’t mind writing code; that’s okay. But I do mind doing stuff that’s going to be superseded by somebody else’s efforts six months down the line. And standardization because I want my skills to be portable and I want my code to be portable as far as possible; I want to be confident that a range of different tool vendors are going to be supporting whatever code it is that I write.

So my personal take would be that there are some extremely worthy efforts out there that have done really great things. For example, one that springs to mind is the library from trusster.com, which is a C++ library of verification tools that you can hook into a Verilog simulator. It works. It’s based on a standard language. So how bad could that be? But it hasn’t really taken off in a very big way – it hasn’t had industry wide traction – and personally I think that’s overwhelmingly because it can’t be as complete as it needs to be. There are things that it has not got around to doing, and that’s no criticism of the people who put it together. As I say, it’s a great job, and it does really good things. But you need your tools to be complete, otherwise, you’re always fighting.

And so somehow, you get some critical mass behind some tools. SystemVerilog is the obvious example of that. Somehow or other, the industry decided it was a good thing and collectively threw its weight behind the development of SystemVerilog. And - lo and behold - it’s now looking pretty complete, looking in pretty good shape, whereas some other approaches aren’t.

And the other requirement, as I said, is standardization. Anything that looks like it’s locked into a single vendor tends to make users jumpy.

AM: True. I think in these days, business is moving so fast and market forces change so quickly that development teams have to be ready to move quickly and adapt. Being locked into one vendor or tool can make some people a bit nervous.

JB: Right. And oftentimes, the choice of vendor is made at a somewhat different level than the technical people on the ground writing the code. And if you spent the last five years investing huge amounts of technical effort in building a verification environment in language ‘Y’, and then suddenly, your management tells you we must for commercial reasons move to a vendor that doesn’t support language ‘Y’, then life is not great.

AM: If I can suggest another verification language that’s out there and is not exclusive to any of the big vendors, it would be PyHVL. It’s based in Python, so you have all the qualities of that language at your disposal, in addition to its standard and open-source libraries. PyHVL also provides a VPI library allowing you to hook into any simulator that supports it, which is pretty much a standard feature these days. Of course, PyHVL is not a fully developed commercial product, so it has its limitations, namely there is currently no support for directly interfacing with VHDL. Also, PyHVL has no native constructs for making constrained-randomization statements, nor anything for implementing functional coverage. However, someone can creatively use the rich set of functionality available in Python libraries to construct something that would be a subset of these features. It’s not perfect, but I think PyHVL can make an interesting candidate for someone who is limited to choosing a verification platform outside those provided from the major vendors.

So let’s go back on the topic of completeness in the context of a verification language. When we say ‘completeness’, we’re talking about constrained-randomization and good support for implementing functional coverage. Of course, let’s not forget the ability to produce both high-level and low-level modeling. Am I missing something?

JB: Well, if you knew what we were all missing, then I’m quite sure your future would be very bright, Alex! But that sounds like a pretty good hit list to me. One of the things that really makes a verification language stand out from other general purpose languages is the way they provide dedicated support for randomization and coverage. Those are two things that are really quite difficult to code up as libraries in a general purpose language. You can do it -of course you can because it’s only code. But it’s very hard to do it in a way that looks smooth and fluid and convenient to use when you’re actually putting a verification environment together. And having dedicated support in the language is just exactly what you need. So I would always, writing verification code, go for a language that has that support if I got the chance.

AM: Definitely something someone has to look out for should they choose not to go with the commercial route, and is either limited to or wants to take a third party route when choosing a suitable verification language.

Well, that’s about all the time that we have, thank you Jonathan for joining me in this conversation on verification languages!

JB: Okay. Thank you, Alex. Good speaking with you.

• Monday February 25th, Tutorial Lessons from the Trenches: Migrating Legacy Verification Environments to UVM™ Mark Litterick presenting
• Tuesday February 26th, Poster session Paul Marriott presenting “Run-Time Configuration of a Verification Environment – A Novel Use of the OVM/UVM Analysis Pattern” and Mark Litterick presenting “Pragmatic Verification Reuse in a Vertical World”
• Tuesday February 26th, Paul Marriott is chairing the REGULAR SESSION: Case Studies - I
• Wednesday February 27th “Session 11 - Hardcore UVM-II” Mark Litterick presenting “SVA Encapsulation in UVM - Enabling Phase and Configuration Aware Assertion”
• Wednesday February 27th Luncheon Best Practices in Verification” Jason Sprott, panelist
• Wednesday February 27th JL Gray moderates Industry Leaders Panel: The Road to 1M Design Starts

Verilab is also sponsoring the Best Paper and Poster Award.

Alex Melikian: You’ve been following the developments of SystemVerilog 2012 very closely. Can you tell us about some of the new language additions that we should be looking out for in this upcoming version of SystemVerilog?

Jonathan Bromley: Yes. I’ve been involved in that more than any normal, reasonable person should expect to be. I’ve been serving as a member of the IEEE committee that works on the testbench features of SystemVerilog for the past 7 years.  I think there’s some very exciting stuff coming up in SystemVerilog 2012. It was deliberately set up as a relatively fast track project. Normally, the revision cycle for IEEE standards is five years, but SystemVerilog 2012 comes only two and a half years after the 2009 standard. So it’s really fast tracked. And it was very carefully focused on a small number of new features. So there’s not a huge list of big ticket items. But there are a couple of things in the verification world that I think are really important.

The first one is a big extension to flexibility of the coverage definition system. You can now define your cover points and your cross cover points in a much more sophisticated, much more algorithmic way than was possible before. There’s a big bunch of stuff that came out there, which looks really exciting. And I get the impression that the vendors are going to rally behind these new items very quickly.

There’s also an interesting new feature in the object oriented programming world that System Verilog calls interface classes. People who come from a Java background might know it as interfaces. People who come from a C++ background might call it multiple inheritance, but we’ve got to be a bit careful there because it isn’t multiple inheritance; it’s interface inheritance, which is slightly different. That’s the other big ticket item, and I guess we haven’t got the time to talk about it in detail here. But it’s certainly well worth following up, and I think it may well have a big impact on the major verification methodologies over the next handful of years. But I can’t comment on when vendors will implement it.

AM: But you do feel that it is imminent that these changes you’re talking about will eventually be incorporated into the SystemVerilog language?

JB: Absolutely right. It’s inevitable. I guess in principle, if it’s in the standards document, then powerful users can lobby and make reasonable demands for the tools to support it. It has to be said that there’s a handful of small things in the SystemVerilog standard that have been there since the very beginning but have still not been implemented by any vendor because they’re actually quite badly designed features. And the vendors have correctly put their foot down and said, “we are not going to do that.” So the fact that a feature is in the standard doesn’t guarantee its implementation. But I think in the case of the new SystemVerilog-2012 features, the big vendors are heavily enough represented on the standards committee that everybody understands that these can and will be done.

Alex: So what are some of the ‘smell’ tests, or litmus tests if you prefer, that you apply to new features that come out in a language to see if they are properly supported and have practical value?

JB: I guess it goes a bit beyond ‘are they supported?’ The first thing you do is when you’re looking at these things, and you see a new feature, the first thing you ask yourself is “can I imagine myself using that?” And we’re geeks, right? We enjoy doing this stuff. So the answer is usually “yes, hey, I’m going to try that. It’s cool. It’s a new feature. Let’s try it.” But the reality is that you then start thinking… you start to try to work out what would I do with this feature? How would this feature make my verification activities better and more productive? And sometimes, it doesn’t make a big difference.

Interestingly, I initially was very excited about the interface inheritance feature that’s coming into SystemVerilog soon. It seemed like just the kind of thing that I really wanted to be able to use. Thinking about it more carefully and trying to map it onto what I expect to be doing in verification over the next few years, it’s not so clear to me now that interface inheritance really helps solve the problems that I’m going to have to solve. I guess we’ll have to wait and see.

But then, of course, it comes back to the other thing you were asking. Is a feature ready for prime time use yet? Dare I try using it myself? Well, to a certain extent, I’m in a privileged position, right? I work for a small consulting company. We can try all kinds of stuff, and that is exactly what we should be doing: to experiment with new features and find out when they’re ready for use. Great. But then, we also have clients. And those clients are likely to be a little bit more conservative. They have real work to do. They have products to get out the door. And they can’t risk that by sticking their necks on the block trying out new features that might not work.

AM: So besides being sort of the crash test dummies for our clients for the new language features and constructs, what are some of the things that we do to convince them that these new elements are ready for use and can be beneficial for them?

JB: Well, I think we can point to our experience, to places where we’ve been able to show a client: hey, look, you ought to be using this language feature because it’s just going to make your life better, and here’s how. So that’s absolutely something we should be doing and can do. But of course, you can’t do that with confidence until - we come back to your previous question - until we know that the vendors have implemented the feature reliably, and I can write a piece of code that uses that feature, and it works reliably across all the major vendor simulators. There’s no point in promoting a feature that works in simulator X but doesn’t work in simulator Y, because we’ve got to be ready to work with clients using any of the major verification tools.

So there’s a series of maturation processes you have to go through. And maybe, you have to use your own skill and judgment about the individual client, too. Some clients are very gung-ho about these things and really want to try stuff out, and are ready to take a risk, and have a fallback position if the risk doesn’t pay out. But other clients are much more conservative and need to be absolutely sure.

AM: That sounds familiar. We’ve recently seen some interesting debate about how programmers can be ‘conservative’ or ‘liberal’ in their approach. It’s understandable that some stick to the devil that they know. There are good reasons for that too, as change always involves some level of risk. As we know and accept, some are more risk averse than others.

JB: Right. And sometimes, changes are just a simple point feature. For example, in SystemVerilog 2012, there’s a new constraint allowing you to specify that a bunch of values should all have unique values and there are no common values between them. It’s a useful new randomization constraint. And that’s a snap, a no brainer. As soon as it’s available for use, we’ll use it.

But there are other bigger, more structural things where you really have to think about whether the people that are going to be using this are really comfortable with what they’re doing and are actually productive with it, because throwing a new piece of technology of any kind, even if it’s as simple as a new language feature, throwing that at people who aren’t ready to use it is just counterproductive. It doesn’t help them get their work done. They may well be more productive using techniques they’ve used in the past.

AM: That’s a good point, something that is new doesn’t necessarily mean it suits your needs better.

JB: Absolutely. New doesn’t necessarily mean better. And even if it really is better in some absolute sense, it’s not necessarily better for any specific person. They have to be ready for it, enthusiastic about it, and prepared for it by training or reading or experimentation or whatever is their chosen way of ramping up.

Alex: I agree. I believe engineering teams should allow themselves time and accept the potential, maybe even likely outcome of failure whenever they are experimenting with something new. One of my pet peeves with the way engineering teams are sometimes managed is that they are given absolutely no room for experimentaton. I think this is a tactical mistake in management. Experimentation should be a separate process, done independently from the processes of delivering a bug-free product. As you mentioned before, something new always involves risk. So, unless you try and explore, you never know. I believe the right attitude is to allocate time and resources for the team to experiment with new things, but without any expectations for the outcome. Even if the conclusion of the experiment turns out to be a complete bust, it’s never really a complete waste of time or resources, because the team gains the factual knowledge and experience of dealing with what is new, as well as the chance to properly evaluate it.

In Part 1, Alex and Jonathan review the different verification languages available today, their histories and differences.

Alex Melikian: Hello, Jonathan, thanks for joining me on this edition of “Thoughts on Verification”. So the theme of this conversation is going to be about verification languages, the ones that exist today and what they’re going to be like tomorrow. So to get started, for the readers out there who are not too familiar with verification languages, maybe you can run through a few of them and describe what exists and what is available.

Jonathan Bromley: Well, whatever I say, I’m sure it will be incomplete. But if you go back maybe 15 years, people who were doing verification of any digital designs were likely using the same languages that they were using to do the design itself. And I guess there’s a good historical reason for that because those languages typically were actually designed for verification. They were designed to model electronic systems rather than to create them. And it was only at the beginning of the 1990’s that logic synthesis became popular as a way of taking a subset of those languages and turning it into physical hardware. So it makes good historical sense that those traditional languages, typically VHDL and Verilog, would have been used for doing verification.

But it wasn’t too long before people began to realize those languages were running out of steam and weren’t flexible enough. They weren’t dynamic enough. They weren’t good enough at coping with the kind of software-like constructs like strings, for example, that you expect to be able to use. So people moved on, and we now see people doing verification with languages that may or may not look quite a lot like those earlier ones.

Nevertheless there is still a huge bunch of people doing verification using VHDL; there’s still a huge bunch of people doing verification with what we all know as vanilla Verilog (I mean, there’s no such thing as vanilla Verilog anymore; the standard got absorbed in to SystemVerilog a few years back, but people are still doing that stuff). But overwhelmingly, I think, if people are doing serious verification of their digital designs, they’re using dedicated languages for it.

So what have we got? Well, the obvious big hitter is SystemVerilog. That grew out of Verilog itself, and also some other origins we can talk about later. It began to come into the public consciousness around about 2003, began to get useful around 2005, and has become very useful recently with the availability of UVM. (I guess we’ll talk about that some more later). So SystemVerilog is here to stay. It allows you to take what looks quite a lot like traditional Verilog, and then suddenly, you glom onto it a whole load of software stuff like object oriented programming and strings and interfacing to C language constructs and a ton of stuff like that. It might be perhaps a little frightening for IC [integrated circuit] designers, but it’s cool for software people.

Then there are other people who are doing verification using much more traditional software-like languages. You may well have come across SystemC, which is an object oriented library totally written in C++. There’s nothing magic about it; it’s a class library written in C++ but carefully designed to be useful for modeling and verifying digital systems. It turns out that that’s most useful, I think, not for the IC designers but for the systems integrators and the system concept engineers who want to work out what the architecture of the system should look like, and model it, long before it’s designed. But nevertheless, SystemC is out there and doing good stuff for people in the verification world too.

Next, there are many, many people doing their verification largely based on C or C++ code integrated with their Verilog device-under-test using features of the simulator that let you do that, the PLI or VPI features. And of course, once you’re in a language like C or C++, you can do basically what you like. There have been a few libraries developed by various third parties. I guess, again, we can talk about those later, too.

AM: I certainly think that developing with a language that streamlines integration such that you can have the common code going to the target device or used in the simulator is quite beneficial. So it makes SystemC and C++ base environments still very attractive.

JB: Right, absolutely. Although it has to be said that it’s quite a big step from writing code that will execute on the target system, and writing the kind of C or C++ that will be used to do verification. You can do both those things, for sure, but there’s a lot of machinery in between those two things.

AM: Sure. Of course, it’s always easier said than done.

JB: Right. But as you say, there are a load of people doing that stuff and making really good use of it and modeling the behavior of their electronic systems in precisely the way they’re going to be used by the software that’s running on the application processor. And that’s a great thing to be able to do.

AM: What about the languages that were specifically made for verification like Specman/e and Vera?

JB: Okay… They’re kind of in between those two areas, I guess. Between SystemVerilog, which is absolutely the outgrowth of a hardware design language, and C++ which is absolutely from the software world. Somewhere in the middle, there’s a bunch of languages that were designed as dedicated verification languages. And I guess the two big hitters there are Vera, which is a Synopsys product dating from – I’m trying to remember now exactly when it was first introduced, but it was something in the order of 15 years ago – and e, the language that goes with the Specman product that is now marketed by Cadence.

So yeah, there are those interesting, dedicated verification languages. Vera, in fact, provided a lot of the impetus for SystemVerilog’s verification extensions, so there is a lot of commonality between the way Vera works and the way that SystemVerilog works. The e language that goes with Specman is quite a different beast, very interesting, fascinating from a software and language design point of view, and has its own strong army of followers for its own unusual and very interesting and very productive features. But there have been some other attempts at dedicated verification languages, too.

Going in another direction, there have been various different attempts to do verification using other standard languages hooked to a Verilog or VHDL simulator by specialized libraries. There are people using Python, for example, as their primary verification language. But I’m not aware of any commercially available products that make that possible with the kind of level of confidence for ordinary users that you would get from buying a SystemVerilog simulator, or Specman, or another of the commercial tools that we all know and love.

AM: We’ll get into that a bit later in the conversation. So you mentioned some of the languages that have come through from a background of design modeling and some from software modeling. Coming back to Specman or Vera, as we mentioned they have evolved from about 15 years ago. However, we don’t see so much evolution anymore in those languages. Is it because they’ve gone through their maturity process and plateaued? Do you see any more room for improvement in some of those languages? Especially, if we compare it to, say, SystemVerilog, which seems to be still in its process of evolution.

JB: Okay… I think it would probably be better to mute the microphone right now because anything I say at this point is likely to lead me into the law courts (laughs). But we have to accept the fact that Vera and e, the Specman language, are both proprietary in the sense that they’re absolutely owned by one specific corporation. And that ownership brings its own benefits and its own drawbacks for users. It means that that corporation can do precisely what they want with the language. If they see an opportunity to do something exciting and new, they can do it, as they have no standardizing body to answer to. But on the other hand, it means that a user is basically locked into a single vendor. [Editor's note: The e language in fact has an IEEE standard (1647), but the vendor's implementation has consistently been a superset of that standard.]

So that’s clearly a real compromise for users. They will see new, cutting edge ideas getting into the language faster and perhaps more effectively than they might expect with a more standardized language. But they’ve got the fear of being locked into a single vendor. What do you do about that? I can’t comment on that. That’s a decision that individual users have to make for themselves.

I kind of take issue with you, about you suggesting there was some stagnation going on there. I’m not sure it’s true because if you look at what’s happened to Specman, for example, over the last handful of years, it has had a process of steady development, cleaning up of raw edges, and general improvements. And it’s still heavily used by a significant user base. So is it stagnant? I’m not sure.

AM: Well, I agree that there has been a good, steady progression the past couple of years, but these changes have not been as ‘ground-breaking’ as the ones we were seeing about 10 or 15 years ago, when the language was still emerging in the industry. However, I think it’s natural for a new language to evolve very quickly initially and less quickly afterwards. I feel those languages have kind of matured. There’s always room for improvement, but I personally feel that they’ve reached a level of maturity whereas maybe the other languages are playing catch-up. It’s important that one must not confuse maturity of a language with the negative connotations of stagnation in its capabilities.

JB: Still, there’s a potential leading question going on there, isn’t there? Are you saying to me that Vera and e are now mature languages that are ready for prime-time use and SystemVerilog is not? Clearly, that’s not quite the situation. But equally, you’re right that there is a huge effort continuing in SystemVerilog to improve it and modify it and change it. And the standards committee is still active. It’s on the point right now of coming out with SystemVerilog – I was going to say 2012 but I’m not sure, it might end up being called SystemVerilog 2013 depending on precisely when it’s formally released. But anyway, there’s a new revision of the SystemVerilog standard coming out any day now. And then the standards committees will continue to work on the enhancement of SystemVerilog after that.