2023 was marked by the generative AI boom, representing a new era for how artificial intelligence can be used across industries.

The year’s top videos from the NVIDIA YouTube channel reflect this focus, with popular videos highlighting the technology powering large language models, new platforms for building generative AI applications and how accelerated computing and AI can advance climate science.

And don’t miss replays of NVIDIA founder and CEO Jensen Huang’s event appearances — his GTC keynote in March has garnered 22 million views, making it by far the most-viewed video on the channel.

Tune in to NVIDIA’s top five videos of the year:

Predicting Extreme Weather Risk — Weeks in Advance

Explore in colorful detail how running FourCastNet — an AI framework developed by researchers at NVIDIA, Caltech and Lawrence Berkeley Lab — on NVIDIA GPUs enables quicker, more accurate extreme weather predictions.

Accelerating Carbon Capture and Storage

Buckle up — learn how reservoir engineers are using NVIDIA Omniverse, NVIDIA Modulus and accelerated computing to optimize carbon capture, ensuring long-term storage and safer operations.

Visualizing Global-Scale Climate Data

Seeing is achieving with this stunning demo of the NVIDIA Earth-2 platform, which offers high-resolution climate visualizations for scientists, as well as breathtakingly detailed urban airflow information for architects and city planners.

A Tour of the NVIDIA DGX H100

Presenting the engine behind the large language model breakthrough — the NVIDIA DGX H100. Hear from Huang on why DGX is “the essential instrument of AI.”

Fine-Tuning Generative AI With NVIDIA AI Workbench 

Check out this demo — featuring a multitude of Toy Jensens — to learn how NVIDIA AI Workbench streamlines selecting foundation models, building project environments and fine-tuning models with domain-specific data.

Explore generative AI sessions and experiences at NVIDIA GTC, the global conference on AI and accelerated computing, running March 18-21 in San Jose, Calif., and online.

Companies across industries are looking to use interactive avatars to enhance digital experiences. But creating them is a complex, time-consuming process requiring state-of-the-art AI models that can see, hear, understand and communicate with end users.

To ease this process, NVIDIA is providing creators and developers with real-time AI solutions through Omniverse Avatar Cloud Engine (ACE), a suite of cloud-native microservices for end-to-end development of interactive avatars. In collaboration with early-access partners, NVIDIA is delivering improvements that will provide users with the tools they need to easily design and deploy various kinds of avatars, from interactive chatbots to intelligent digital humans.

AT&T and Quantiphi are among the first to experience how Omniverse ACE can help increase employee productivity and enhance customer service experiences.

Omniverse ACE users can now seamlessly integrate NVIDIA AI into their applications, including Riva for speech AI, NeMo service for natural language understanding, and Omniverse Audio2Face or Live Portrait for AI-powered 2D and 3D character animation.

With the latest improvements to Omniverse ACE, teams can also deploy advanced avatars across web conferencing and customer service use cases by integrating domain-specific NVIDIA AI workflows like Tokkio and Maxine.

Early Partners and Customers Develop AI-Driven Digital Humans

AT&T is planning to use Omniverse ACE and the Tokkio AI avatar workflow to build, customize and deploy virtual assistants for customer service and its employee help desk. Working with Quantiphi, one of NVIDIA’s service delivery partners, AT&T is developing interactive avatars that can provide 24/7 support in local languages across regions. This is helping the company reduce costs while providing a better experience for its employees worldwide.

In addition to customer service, AT&T is planning to build and develop digital humans for various use cases across the company.

“Quantiphi and NVIDIA have been collaborating to make customer experience more immersive by combining the power of large language models, graphics and recommender systems,” said Siddharth Kotwal, global head of NVIDIA Practice at Quantiphi. “NVIDIA’s Tokkio framework has made it easier to build, deploy and personalize AI-powered digital assistants or avatars for our enterprise customers. The process of seamlessly integrating automatic speech recognition, conversational agents and information retrieval systems with real-time animation has been simplified.”

Leading professional-services company Deloitte is also working with NVIDIA to help enterprises deploy transformative applications. Deloitte’s latest hybrid-cloud offerings — which consist of NVIDIA AI and Omniverse services and platforms, including Omniverse ACE — will be added to the Deloitte Center for AI Computing.

An Advanced, Streamlined Solution for Deploying Avatars

Omniverse ACE provides all the necessary tools so users can streamline the development process for realistic, intelligent avatars. Teams can also customize pre-built AI avatar workflows to suit their needs with applications like NVIDIA Tokkio. Additionally, Omniverse ACE is bringing new improvements to existing microservices.

Learn more about NVIDIA Omniverse ACE and register to join the early-access program, available now for developers.

Dive into the art of AI avatars at GTC, a global conference for the era of AI and the metaverse. Join sessions with NVIDIA and industry experts, and watch the GTC keynote below:

Powerful AI technologies are making a massive impact in 3D content creation and game development. Whether creating realistic characters that show emotion or turning simple texts into imagery, AI tools are becoming fundamental to developer workflows — and this is just the start.

At NVIDIA GTC and the Game Developers Conference (GDC), learn how the NVIDIA Omniverse platform for creating and operating metaverse applications is expanding with new Connectors and generative AI services for game developers.

Part of the excitement around generative AI is because of its ability to capture the creator’s intent. The technology learns the underlying patterns and structures of data, and uses that to generate new content, such as images, audio, code, text, 3D models and more.

Announced today, the NVIDIA AI Foundations cloud services enable users to build, refine and operate custom large language models (LLMs) and generative AI trained with their proprietary data for their domain-specific tasks.

And through NVIDIA Omniverse, developers can get their first taste of using generative AI technology to enhance game creation and accelerate development pipelines with the Omniverse Audio2Face app.

Accelerating 3D Content With Generative AI

Specialized generative AI tools can boost creator productivity, even for users who don’t have extensive technical skills. Anyone can use generative AI to bring their creative ideas to life, producing high-quality, highly iterative experiences — all in a fraction of the time and cost of traditional game development.

For example, NVIDIA Omniverse Avatar Cloud Engine (ACE) offers the fastest, most versatile solution for bringing interactive avatars to life at scale. Game developers could leverage ACE to seamlessly integrate NVIDIA AI into their applications, including NVIDIA Riva for creating expressive character voices using speech and translation AI, or Omniverse Audio2Face and Live Portrait for AI-powered 2D and 3D character animation.

Today, game developers are already taking advantage of Audio2Face, where artists are more efficiently animating characters without a tedious manual process. The app’s latest release brings major quality, usability and performance updates, including headless mode and a REST API — enabling developers to run the app and process numerous audio files from multiple users in the data center.

Mandarin Chinese language support can now be previewed in Audio2Face, along with improved lip-sync quality, more robust multi-language support and a new pretrained female model. The world’s first fully real-time, ray-traced subsurface scattering shader is also demonstrated in the demo with Diana, a new digital human model.

GSC Game World, one of Europe’s leading game developers, is adopting Omniverse Audio2Face in its upcoming game, S.T.A.L.K.E.R. 2 Head of Chernobyl. Join the NVIDIA and GCS session at GDC to learn how developers are implementing generative AI technology in Omniverse.

Fallen Leaf, an indie game developer, is also using Omniverse Audio2Face for character facial animation in Fort Solis, a third-person sci-fi thriller game that takes place on Mars.

New generative AI services such as NVIDIA Picasso, announced at GTC, preview the future of building and deploying assets for game production pipelines. Omniverse is opening portals to enrich workflows with generative AI tools powered by NVIDIA and its partners, and the momentum around unifying the game asset pipeline is growing.

Unifying Game Asset Pipelines With Universal Scene Description

Based on the Universal Scene Description (USD) framework, NVIDIA Omniverse is the connecting fabric that helps creators and developers build interoperability between their favorite tools — like Autodesk Maya, Autodesk 3ds Max and Adobe Substance 3D Painter — or make their own custom applications.

And with USD — an open, extensible framework and ecosystem for composing, simulating and collaborating within 3D worlds — developers can achieve non-destructive, collaborative workflows when creating scenes, as well as simplify asset aggregation so content creation teams can iterate faster.

Tencent Games is adopting USD workflows based on Omniverse to better streamline content creation pipelines. To create vast worlds in every level of a game, the artists at Tencent use design tools such as Autodesk Maya, SideFX Houdini and Unreal Engine to produce up to millions of trees, buildings and other properties to enrich their scenes. The technical artists often look to optimize their content creation pipelines to speed up this process, so they developed a proprietary Unreal Engine workflow powered by OmniObjects.

With USD, Tencent Games’ teams saw the opportunity to easily streamline and seamlessly connect their workflows. Building on Omniverse as the platform for developing USD workflows, the artists at Tencent no longer need to install plug-ins for each software they use. Using just one USD plug-in enables interoperability across all their favorite software tools. Learn more about Tencent Games by joining this session at GDC.

New and updated Omniverse Connectors for game engines are also now available.

The open-beta Omniverse Connector for Unity workflows helps users of Omniverse and Unity collaborate on projects. Developed by NVIDIA, the Connector delivers USD support alongside Unity workflows, enabling Unity users to take advantage of interoperable workflows. It offers Omniverse Nucleus connection and browsing, USD geometry export, lights, cameras, Material Definition Language and preview for USD materials. Early features also include physics export, USD import and unidirectional live sync.

And with the Unreal Engine Connector’s latest release, Omniverse users can now use Unreal Engine’s USD import utilities to add skeletal mesh blend shape importing, and Python USD bindings to access stages on Omniverse Nucleus. The latest release also delivers improvements in import, export and live workflows, as well as updated software development kits.

Learn more about these latest technologies by joining NVIDIA at GDC.

And catch up on all the groundbreaking announcements in generative AI and the metaverse by watching the NVIDIA GTC keynote.

Follow NVIDIA Omniverse on Instagram, Medium, Twitter and YouTube for additional resources and inspiration. Check out the Omniverse forums, and join our Discord server and Twitch channel to chat with the community.

Promising more timely deliveries for consumers around the globe, NVIDIA’s cuOpt real-time route optimization software has set records on a key route optimization benchmark.

NVIDIA cuOpt set three new records on the widely followed Li & Lim pickup and delivery benchmark.

Last-mile delivery is the most expensive part of the logistics industry, representing over 40% of overall supply chain cost and carbon footprint, according to Gartner. Nearly 150 billion parcels are shipped every year, according to Pitney Bowes.

AT&T is using cuOpt to optimize routes for 30,000 technicians. “With cuOpt, AT&T can find a solution 100x faster and update their dispatch in real time,” said NVIDIA CEO Jensen Huang, during his keynote at NVIDIA’s GTC technology conference Tuesday.

AT&T is also testing digital assistants built with NVIDIA Omniverse Avatar Cloud Engine to enhance the customer service experience and improve its employee help desk. Additionally, the company is accelerating its data-processing workflow using NVIDIA RAPIDS Accelerator for Apache Spark, a suite of libraries that enable GPU acceleration of data-science pipelines.

“AT&T has adopted a full suite of NVIDIA AI libraries,” Huang said.

Better solutions to the pickup and delivery problems result in lower costs for manufacturers moving goods and services across the globe, quicker disaster relief and hotter, fresher pizza, among other benefits.

Introduced in 2021, NVIDIA cuOpt offers enterprises the ability to adapt to real-time data to optimize delivery routes by analyzing billions of feasible moves per second.

cuOpt is now at the center of a thriving partner ecosystem of system integrators and service providers, logistics and transportation software vendors, optimization software specialists and location service providers.

A Global Benchmark

Route optimization is one of the most critical industrial computing problems of our time.

While the problems involved in route optimization are simple to understand — what’s the most efficient way to visit the most places — the computation required to determine the most efficient routes stacks up, fast, as the number of delivery vehicles, customers and delivery destinations increases.

That’s made the benchmarks for pickup and delivery problems set forth in 2001 by Hiabing Li at the New Jersey Institute of Technology and Andrew Lim at the Hong Kong University of Science and Technology — who introduced a collection of 300 datasets by which a route’s efficiency can be measured — a widely watched global standard.

Researchers have been proposing best route plans for these benchmarks for more than two decades, inventing algorithms that set and reset the world’s best-known solutions, with past winners focusing on making small tweaks to previous routes.

The route cuOpt created, by contrast, looks unlike the routes created by previous winners. It was able to find an entirely new approach to the problem, delivering three world-record solutions in the largest instances of the Li & Lim benchmark suite. They include 1,000 pickup and delivery locations.

The benchmark’s top objective is to minimize the fleet size first, and the total distance traveled next. Nevertheless, cuOpt was able to cut the distance traveled by as much as 0.8% to 1,000 pickup and delivery locations.

By relying on the parallel computing capabilities of NVIDIA A100 Tensor Core GPUs, cuOpt is able to search for routes more deeply and more broadly.

The result: cuOpt delivered an improvement 7.2x higher than the improvement over the previous record on the benchmark and 26.6x higher than the improvement gained by the record-setting effort before that.

cuOpt for production environments is available with NVIDIA AI Enterprise, the software layer of the NVIDIA AI platform. With enterprise support for over 50 production-ready frameworks, pretrained models and development tools included, NVIDIA AI Enterprise is designed to accelerate enterprises to the bleeding edge of AI, while also simplifying AI to make it accessible to every enterprise.

Enterprises can also work with NVIDIA service delivery partners including Deloitte and Quantiphi to integrate cuOpt solutions into their business.

Learn More

Explore what cuOpt can do for your business, stay up to date on the latest news for cuOpt and join us at GTC.

Resources to learn more:

• cuOpt Product Page
• cuOpt Cloud Service Early Access Program
• Route Optimization AI Workflow (powered by cuOpt)
• cuOpt News Sign Up Form
• cuOpt GTC sessions
  • Advances in Operations Optimization
  • Accelerated AI Logistics and Route Optimization 101

New cloud services to support AI workflows and the launch of a new generation of GeForce RTX GPUs featured today in NVIDIA CEO Jensen Huang’s GTC keynote, which was packed with new systems, silicon, and software.

“Computing is advancing at incredible speeds, the engine propelling this rocket is accelerated computing, and its fuel is AI,” Huang said during a virtual presentation as he kicked off NVIDIA GTC.

Again and again, Huang connected new technologies to new products to new opportunities – from harnessing AI to delight gamers with never-before-seen graphics to building virtual proving grounds where the world’s biggest companies can refine their products.

Driving the deluge of new ideas, new products and new applications: a singular vision of accelerated computing unlocking advances in AI, which, in turn will touch industries around the world.

Gamers and creators will get the first GPUs based on the new NVIDIA Ada Lovelace architecture.

Enterprises will get powerful new tools for high-performance computing applications with systems based on the Grace CPU and Grace Hopper Superchip. Those building the 3D internet will get new OVX servers powered by Ada Lovelace L40 data center GPUs. Researchers and computer scientists get new large language model capabilities with NVIDIA LLMs NeMo Service. And the auto industry gets Thor, a new brain with an astonishing 2,000 teraflops of performance.

Huang highlighted how NVIDIA’s technologies are being put to work by a sweep of major partners and customers across a breadth of industries.

To speed adoption, he announced Deloitte, the world’s largest professional services firm, is bringing new services built on NVIDIA AI and NVIDIA Omniverse to the world’s enterprises.

And he shared customer stories from telecoms giant Charter, as well as General Motors in the automotive industry, the German railway system’s Deutsche Bahn in transportation, The Broad Institute in medical research, and Lowe’s in retail.

NVIDIA GTC, which kicked off this week, has become one of the world’s most important AI gatherings, with 200+ speakers from companies such as Boeing, Deutsche Bank, Lowe’s, Polestar, Johnson & Johnson, Kroger, Mercedes-Benz, Siemens AG, T-Mobile and US Bank. More than 200,000 people have registered for the conference.

A ‘Quantum Leap’: GeForce RTX 40 Series GPUs

First out of the blocks at the keynote was the launch of next-generation GeForce RTX 40 Series GPUs powered by Ada, which Huang called a “quantum leap” that paves the way for creators of fully simulated worlds.

Huang gave his audience a taste of what that makes possible by offering up a look at Racer RTX, a fully interactive simulation that’s entirely ray traced, with all the action physically modeled.

Ada’s advancements include a new Streaming Multiprocessor, a new RT Core with twice the ray-triangle intersection throughput, and a new Tensor Core with the Hopper FP8 Transformer Engine and 1.4 petaflops of Tensor processor power.

Ada also introduces the latest version of NVIDIA DLSS technology, DLSS 3, which uses AI to generate new frames by comparing new frames with prior frames to understand how a scene is changing. The result: boosting game performance by up to 4x over brute force rendering.

DLSS 3 has received support from many of the world’s leading game developers, with more than 35 games and applications announcing support. “DLSS 3 is one of our greatest neural rendering inventions,” Huang said.

Together, Huang said, these innovations help deliver 4x more processing throughput with the new GeForce RTX 4090 versus its forerunner, the RTX 3090 Ti. “The new heavyweight champ” starts at $1,599 and will be available Oct. 12.

Additionally, the new GeForce RTX 4080 is launching in November with two configurations.

The GeForce RTX 4080 16GB, priced at $1,199, has 9,728 CUDA cores and 16GB of high-speed Micron GDDR6X memory. With DLSS 3, it’s twice as fast in today’s games as the GeForce RTX 3080 Ti,  and more powerful than the GeForce RTX 3090 Ti at lower power.

The GeForce RTX 4080 12GB has 7,680 CUDA cores and 12GB of Micron GDDR6X memory, and with DLSS 3 is faster than the RTX 3090 Ti, the previous-generation flagship GPU. It’s priced at $899.

Huang also announced that NVIDIA Lightspeed Studios used Omniverse to reimagine Portal, one of the most celebrated games in history. With NVIDIA RTX Remix, an AI-assisted toolset, users can mod their favorite games, enabling them to up-res textures and assets, and give materials physically accurate properties.

Powering AI Advances, H100 GPU in Full Production

Once more tying systems and software to broad technology trends, Huang explained that large language models, or LLMs, and recommender systems are the two most important AI models today.

Recommenders “run the digital economy,” powering everything from e-commerce to entertainment to advertising, he said. “They’re the engines behind social media, digital advertising, e-commerce and search.”

And large language models based on the Transformer deep learning model first introduced in 2017 are now among the most vibrant areas for research in AI, and able to learn to understand human language without supervision or labeled datasets.

“A single pre-trained model can perform multiple tasks, like question answering, document summarization, text generation, translation and even software programming,” Huang said.

Delivering the computing muscle needed to power these enormous models, Huang said the NVIDIA H100 Tensor Core GPU, with Hopper’s next-generation Transformer Engine, is in full production, with systems shipping in the coming weeks.

“Hopper is in full production and coming soon to power the world’s AI factories,” Huang said.

Partners building systems include Atos, Cisco, Dell Technologies, Fujitsu, GIGABYTE, Hewlett Packard Enterprise, Lenovo and Supermicro. And Amazon Web Services, Google Cloud, Microsoft Azure and Oracle Cloud Infrastructure will be among the first to deploy H100-based instances in the cloud starting next year.

And Grace Hopper, which combines NVIDIA’s Arm-based Grace data center CPU with Hopper GPUs, with its 7x increase in fast-memory capacity, will deliver a “giant leap” for recommender systems, Huang said. Systems incorporating Grace Hopper will be available in the first half of 2023.

Weaving Together the Metaverse, L40 Data Center GPUs in Full Production

The next evolution of the internet, called the metaverse, will be extended with 3D, Huang explained. Omniverse is NVIDIA’s platform for building and running metaverse applications.

Here, too, Huang explained how connecting and simulating these worlds will require powerful, flexible new computers. And NVIDIA OVX servers are built for scaling out metaverse applications.

NVIDIA’s 2nd-generation OVX systems will be powered by Ada Lovelace L40 data center GPUs, which are now in full production, Huang announced.

Thor for Autonomous Vehicles, Robotics, Medical Instruments and More

In today’s vehicles, active safety, parking, driver monitoring, camera mirrors, cluster and infotainment are driven by different computers. In the future, they’ll be delivered by software that improves over time, running on a centralized computer, Huang said.

To power this, Huang introduced DRIVE Thor, which combines the transformer engine of Hopper, the GPU of Ada, and the amazing CPU of Grace.

The new Thor superchip delivers 2,000 teraflops of performance, replacing Atlan on the DRIVE roadmap, and providing a seamless transition from DRIVE Orin, which has 254 TOPS of performance and is currently in production vehicles. Thor will be the processor for robotics, medical instruments, industrial automation and edge AI systems, Huang said.

3.5 Million Developers, 3,000 Accelerated Applications

Bringing NVIDIA’s systems and silicon, and the benefits of accelerated computing, to industries around the world, is a software ecosystem with more than 3.5 million developers creating some 3,000 accelerated apps using NVIDIA’s 550 software development kits, or SDKs, and AI models, Huang announced.

And it’s growing fast. Over the past 12 months, NVIDIA has updated more than 100 SDKs and introduced 25 new ones.

“New SDKs increase the capability and performance of systems our customers already own, while opening new markets for accelerated computing,” Huang said.

New Services for AI, Virtual Worlds

Large language models “are the most important AI models today,” Huang said. Based on the transformer architecture, these giant models can learn to understand meanings and languages without supervision or labeled datasets, unlocking remarkable new capabilities.

To make it easier for researchers to apply this “incredible” technology to their work, Huang announced the Nemo LLM Service, an NVIDIA-managed cloud service to adapt pretrained LLMs to perform specific tasks.

To accelerate the work of drug and bioscience researchers, Huang also announced BioNeMo LLM, a service to create LLMs that understand chemicals, proteins, DNA and RNA sequences.

Huang announced that NVIDIA is working with The Broad Institute, the world’s largest producer of human genomic information, to make NVIDIA Clara libraries, such as NVIDIA Parabricks, the Genome Analysis Toolkit, and BioNeMo, available on Broad’s Terra Cloud Platform.

Huang also detailed NVIDIA Omniverse Cloud, an infrastructure-as-a-service that connects Omniverse applications running in the cloud, on premises or on a device.

New Omniverse containers – Replicator for synthetic data generation, Farm for scaling render farms, and Isaac Sim for building and training AI robots – are now available for cloud deployment, Huang announced.

Omniverse is seeing wide adoption, and Huang shared several customer stories and demos:

• Lowe’s, which has nearly 2,000 retail outlets, is using Omniverse to design, build and operate digital twins of their stores;
• Charter, a $50 billion dollar telecoms provider, and interactive data analytics provider HeavyAI, are using Omniverse to create digital twins of Charter’s 4G and 5G networks;
• GM is creating a digital twin of its Michigan Design Studio in Omniverse where designers, engineers and marketers can collaborate.

New Jetson Orin Nano for Robotics

Shifting from virtual worlds to machines that will move through their world, robotic computers “are the newest types of computers,” Huang said, describing NVIDIA’s second-generation processor for robotics, Orin, as a homerun.

To bring Orin to more markets, he announced the Jetson Orin Nano, a tiny robotics computer that is 80x faster than the previous super-popular Jetson Nano.

Jetson Orin Nano runs the NVIDIA Isaac robotics stack and features the ROS 2 GPU-accelerated framework, and NVIDIA Iaaac Sim, a robotics simulation platform, is available on the cloud.

And for robotics developers using AWS RoboMaker, Huang announced that containers for the NVIDIA Isaac platform for robotics development are in the AWS marketplace.

New Tools for Video, Image Services

Most of the world’s internet traffic is video, and user-generated video streams will be increasingly augmented by AI special effects and computer graphics, Huang explained.

“Avatars will do computer vision, speech AI, language understanding and computer graphics in real time and at cloud scale,” Huang said.

To enable new innovations at the intersection of real-time graphics, AI and communications possible, Huang announced NVIDIA has been building acceleration libraries like CV-CUDA, a cloud runtime engine called UCF Unified Computing Framework, Omniverse ACE Avatar Cloud Engine, and a sample application called Tokkio for customer service avatars.

Deloitte to Bring AI, Omniverse Services to Enterprises

And to speed the adoption of all these technologies to the world’s enterprises, Deloitte, the world’s largest professional services firm, is bringing new services built on NVIDIA AI and NVIDIA Omniverse to the world’s enterprises, Huang announced.

He said that Deloitte’s professionals will help the world’s enterprises use NVIDIA application frameworks to build modern multi-cloud applications for customer service, cybersecurity, industrial automation, warehouse and retail automation and more.

Just Getting Started

Huang ended his keynote by recapping a talk that moved from outlining new technologies to product announcements and back — uniting scores of different parts into a singular vision.

“Today, we announced new chips, new advances to our platforms, and, for the very first time, new cloud services,” Huang said as he wrapped up. “These platforms propel new breakthroughs in AI, new applications of AI, and the next wave of AI for science and industry.”

Promising to help process images faster and more efficiently at a vast scale, NVIDIA introduced CV-CUDA, an open-source library for building accelerated end-to-end computer vision and image processing pipelines.

The majority of internet traffic is video. Increasingly, this video will be augmented by AI special effects and computer graphics.

To add to this complexity, fast-growing social media and video-sharing services are experiencing growing cloud computing costs and bottlenecks in their AI-based imaging processing and computer vision pipelines.

CV-CUDA accelerates AI special effects such as relighting, reposing, blurring backgrounds and super resolution.

NVIDIA GPUs already accelerate the inference portion of AI computer vision pipelines. But pre- and post-processing using traditional computer vision tools gobble up time and computing power.

CV-CUDA gives developers more than 50 high-performance computer vision algorithms, a development framework that makes it easy to implement custom kernels and zero-copy interfaces to remove bottlenecks in the AI pipeline.

The result is higher throughput and lower cloud-computing costs. CV-CUDA can process 10x as many streams on a single GPU.

All this helps developers move much faster when tackling video content creation, 3D worlds, image-based recommender systems, image recognition and video conferencing.

Video content creation platforms must process, enhance and moderate millions of video streams daily and ensure mobile-based users have the best experience running their apps on any phone.

• For those building 3D worlds or metaverse applications, CV-CUDA is anticipated to enable tasks to help build or extend 3D worlds and their components.
• In image understanding and recognition, CV-CUDA can significantly speed up the pipelines running at hyperscale, allowing mobile users to enjoy sophisticated and responsive image recognition applications.
• And in video conferencing, CV-CUDA can support sophisticated augmented reality-based features. These features could involve complex AI pipelines requiring numerous pre- and post-processing steps.

CV-CUDA accelerates pre- and post-processing pipelines through hand-optimized CUDA kernels and natively integrates into C/C++, Python and common deep learning frameworks, such as PyTorch.

CV-CUDA will be one of the core technologies that can accelerate AI workflows in NVIDIA Omniverse, a virtual world simulation and collaboration platform for 3D workflows.

Developers can get early access to code in December, with a beta release set for March.

For more, visit the early access interest page.

Featured image credit: Factory42/BBC Studios

How do I start a career as a deep learning engineer? What are some of the key tools and frameworks used in AI? How do I learn more about ethics in AI?

Everyone has questions, but the most common questions in AI always return to this: how do I get involved?

Cutting through the hype to share fundamental principles for building a career in AI, a group of AI professionals gathered at NVIDIA’s GTC conference in the spring offered what may be the best place to start.

Each panelist, in a conversation with NVIDIA’s Louis Stewart, head of strategic initiatives for the developer ecosystem, came to the industry from very different places.

Watch the session on demand.

But the speakers — Katie Kallot, NVIDIA’s former head of global developer relations and emerging areas; David Ajoku, founder of startup aware.ai; Sheila Beladinejad, CEO of Canada Tech; and Teemu Roos, professor at the University of Helsinki  — returned again and again to four basic principles.

1) Start With Networking and Mentorship

The best way to start, Ajoku explained, is to find people who are where you want to be in five years.

And don’t just look for them online — on Twitter and LinkedIn. Look for opportunities to connect with others in your community and at professional events who are going where you want to be.

“You want to find people you admire, find people who walk the path you want to be on over the next five years,” Ajoku said. “It doesn’t just come to you; you have to go get it.”

At the same time, be generous about sharing what you know with others. “You want to find people who will teach, and in teaching, you will learn,” he added.

But the best place to start is knowing that reaching out is okay.

“When I started my career in computer science, I didn’t even know I should be seeking a mentor,” Beladinejad said, echoing remarks from the other panelists.

“I learned not to be shy, to ask for support and seek help whenever you get stuck on something — always have the confidence to approach your professors and classmates,” she added.

2) Get Experience

Kallot explained that the best way to learn is by doing.

She got a degree in political science and learned about technology — including how to code — while working in the industry.

She started out as a sales and marketing analyst, then leaped to a product manager role.

“I had to learn everything about AI in three months, and at the same time I had to learn to use the product, I had to learn to code,” she said.

The best experience, explained Roos, is to surround yourself with people on the same learning journey, whether they’re learning online or in person.

“Don’t do it alone. If you can, grab your friends, grab your colleagues, maybe start a study group and create a curriculum,” he said. “Meet once a week, twice a week — it’s much more fun that way.”

3) Develop Soft Skills

You’ll also need the communications skills to explain what you’re learning, and doing, in AI as you progress.

“Practice talking about technical topics to non-technical audiences,” Stewart said.

Ajoku recommended learning and practicing public speaking.

Ajoku took an acting class at Carnegie Mellon University. Similarly, Roos took an improv comedy class.

Others on the panel learned to perform, publicly, through dance and sports.

“The more you’re cross-trained, the more comfortable you’re going to be and the better you’re going to be able to express yourself in any environment,” Stewart said.

4) Define Your Why 

The most important element, however, comes from within, the panelists said.

They urged listeners to find a reason, something that drives them to stay motivated on their journey.

For some, it’s environmental issues. Others are driven by a desire to make technology more accessible. Or to help make the industry more inclusive, panelists said.

“It’s helpful for anyone if you have a topic that you’re passionate about,” Beladinejad said. “That would help keep you going, keep your motivation up.”

Whatever you do, “do it with passion,” Stewart said. “Do it with purpose.”

Burning Questions

Throughout the conversation, thousands of virtual attendees submitted more than 350 questions about how to get started in their AI careers.

Among them:

What’s the best way to learn about deep learning? 

The NVIDIA Deep Learning Institute offers a huge variety of hands-on courses.

Even more resources for new and experienced developers alike are available through the NVIDIA Developer program, which includes resources for those pursuing higher education and research.

Massive open online courses — or MOOCs — have made learning about technical subjects more accessible than ever. One panelist suggested looking for classes taught by Stanford Professor Andrew Ng on Coursera.

“There are many MOOC courses out there, YouTube videos and books — I highly recommend finding a study buddy as well,” another wrote.

“Join technical and professional networks … get some experience through volunteering, participating in a Kaggle competition, etc.”

What are some of the most prevalent tools and frameworks used in machine learning and AI in industry? Which ones are crucial to landing a first job or internship in the field?

The best way to figure out which technologies you want to start with, one panelist suggested, is to think about what you want to do.

Another suggested, however, that learning Python isn’t a bad place to begin.

“A lot of today’s AI tools are based on Python,” they wrote. “You can’t go wrong by mastering Python.”

“The technology is evolving rapidly, so many of today’s AI developers are constantly learning new things. Having software fundamentals like data structures and common languages like Python and C++ will help set you up to ‘learn on the job,’” another added.

What’s the best way to start getting experience in the field? Do personal projects count as experience? 

Student clubs, online developer communities, volunteering and personal projects are all a great way to gain hands-on experience, panelists wrote.

And definitely include personal projects on your resume, another added.

Is there an age limit for getting involved in AI? 

Age isn’t at all a barrier, whether you’re just starting out or transitioning from another field, panelists wrote.

Build a portfolio for yourself so you can better demonstrate your skills and abilities — that’s what should count.

Employers should be able to easily realize your potential and skills.

I want to build a tech startup with some form of AI as the engine driving the solution to solve an as-yet-to-be-determined problem. What pointers do you have for entrepreneurs? 

Entrepreneurs should apply to be a part of NVIDIA Inception.

The program provides free benefits, such as technical support, go-to-market support, preferred pricing on hardware and access to its VC alliance for funding.

Which programming language is best for AI?

Python is widely used in deep learning, machine learning and data science. The programming language is at the center of a thriving ecosystem of deep learning frameworks and developer tools. It’s predominantly used for training complex models and for real-time inference for web-based services.

C/C++ is a popular programming language for self-driving cars which is used for deploying models for real-time inference.

Those getting started, though, will want to make sure they’re familiar with a broad array of tools, not just Python.

The NVIDIA Deep Learning Institute’s beginner self-paced courses can be one of the best ways to get oriented.

Learn More From GTC Sessions

At NVIDIA GTC, a global conference on AI and the metaverse, professionals spoke about how they got started in their careers.

Watch the on-demand GTC sessions How to Be a Deep Learning Engineer and 5 Paths to a Career in AI.

Learn the AI essentials from NVIDIA fast: check out the “getting started” resources to explore the fundamentals of today’s hottest technologies on our learning series page.

Promising to transform trillion-dollar industries and address the “grand challenges” of our time, NVIDIA founder and CEO Jensen Huang Tuesday shared a vision of an era where intelligence is created on an industrial scale and woven into real and virtual worlds.

Kicking off NVIDIA’s GTC conference, Huang introduced new silicon — including the new Hopper GPU architecture and new H100 GPU, new AI and accelerated computing software and powerful new data-center-scale systems.

”Companies are processing, refining their data, making AI software, becoming intelligence manufacturers,” Huang said, speaking from a virtual environment in the NVIDIA Omniverse real-time 3D collaboration and simulation platform as he described how AI is “racing in every direction.”

And all of it will be brought together by Omniverse to speed collaboration between people and AIs, better model and understand the real world, and serve as a proving ground for new kinds of robots, “the next wave of AI.”

Huang shared his vision with a gathering that has become one of the world’s most important AI conferences, bringing together leading developers, scientists and researchers.

The conference features more 1,600 speakers including from companies such as American Express, DoorDash, LinkedIn, Pinterest, Salesforce, ServiceNow, Snap and Visa, as well as 200,000 registered attendees.

Huang’s presentation began with a spectacular flythrough of NVIDIA’s new campus, rendered in Omniverse, including buzzing labs working on advanced robotics projects.

He shared how the company’s work with the broader ecosystem is saving lives by advancing healthcare and drug discovery, and even helping save our planet.

“Scientists predict that a supercomputer a billion times larger than today’s is needed to effectively simulate regional climate change,” Huang said.

“NVIDIA is going to tackle this grand challenge with our Earth-2, the world’s first AI digital twin supercomputer, and invent new AI and computing technologies to give us a billion-X before it’s too late,” he said.

New Silicon — NVIDIA H100: A “New Engine of the World’s AI Infrastructure”

To power these ambitious efforts, Huang introduced the NVIDIA H100 built on the Hopper architecture, as the “new engine of the world’s AI infrastructures.”

AI applications like speech, conversation, customer service and recommenders are driving fundamental changes in data center design, he said.

“AI data centers process mountains of continuous data to train and refine AI models,” Huang said. “Raw data comes in, is refined, and intelligence goes out — companies are manufacturing intelligence and operating giant AI factories.”

The factory operation is 24/7 and intense, Huang said. Minor improvements in quality drive a significant increase in customer engagement and company profits, Huang explained.

H100 will help these factories move faster. The “massive” 80 billion transistor chip uses TSMC’s 4N process.

“Hopper H100 is the biggest generational leap ever — 9x at-scale training performance over A100 and 30x large-language-model inference throughput,” Huang said.

Hopper is packed with technical breakthroughs, including a new Transformer Engine to speed up these networks 6x without losing accuracy.

“Transformer model training can be reduced from weeks to days” Huang said.

H100 is in production, with availability starting in Q3, Huang announced.

Huang also announced the Grace CPU Superchip, NVIDIA’s first discrete data center CPU for high-performance computing.

It comprises two CPU chips connected over a 900 gigabytes per second NVLink chip-to-chip interconnect to make a 144-core CPU with 1 terabyte per second of memory bandwidth, Huang explained.

“Grace is the ideal CPU for the world’s AI infrastructures,” Huang said.

Huang also announced new Hopper GPU-based AI supercomputers — DGX H100, H100 DGX POD and DGX SuperPOD.

NVIDIA will be first to build a DGX SuperPOD with the groundbreaking new AI architecture to power the work of NVIDIA researchers advancing climate science, digital biology and the future of AI.

Its “Eos” supercomputer is expected to be the world’s fastest AI system after it begins operations later this year, featuring a total of 576 DGX H100 systems with 4,608 DGX H100 GPUs.

To connect it all, NVIDIA’s new NVLink high-speed interconnect technology will be coming to all future NVIDIA chips — CPUs, GPUs, DPUs and SOCs, Huang said.

He also announced NVIDIA will make NVLink available to customers and partners to build companion chips.

“NVLink opens a new world of opportunities for customers to build semi-custom chips and systems that leverage NVIDIA’s platforms and ecosystems,” Huang said.

New Software — AI Has “Fundamentally Changed” Software

Thanks to acceleration unleashed by accelerated computing, the progress of AI is “stunning,” Huang declared.

“AI has fundamentally changed what software can make and how you make software,” Huang said.

Transformers, Huang explained, have opened self-supervised learning and unblocked the need for human-labeled data. As a result, Transformers are being unleashed in a growing array of fields.

“Transformers made self-supervised learning possible, and AI jumped to warp speed,” Huang said.

Google BERT for language understanding, NVIDIA MegaMolBART for drug discovery, and DeepMind AlphaFold2 are all breakthroughs traced to Transformers, Huang said.

Huang walked through new deep learning models for natural language understanding, physics, creative design, character animation and even — with NVCell — chip layout.

“AI is racing in every direction — new architectures, new learning strategies, larger and more robust models, new science, new applications, new industries — all at the same time,” Huang said.

NVIDIA is “all hands on deck” to speed new breakthroughs in AI and speed the adoption of AI and machine learning to every industry, Huang said.

The NVIDIA AI platform is getting major updates, Huang said, including Triton Inference Server, the NeMo Megatron 0.9 framework for training large language models, and the Maxine framework for audio and video quality enhancement.

The platform includes NVIDIA AI Enterprise 2.0, an end-to-end, cloud-native suite of AI and data analytics tools and frameworks, optimized and certified by NVIDIA and now supported across every major data center and cloud platform.

“We updated 60 SDKs at this GTC,” Huang said. “For our 3 million developers, scientists and AI researchers, and tens of thousands of startups and enterprises, the same NVIDIA systems you run just got faster.”

NVIDIA AI software and accelerated computing SDKs are now relied on by some of the world’s largest companies.

• Microsoft Translator accelerates global communications with real-time translation capabilities powered by NVIDIA Triton.
• AT&T accelerates their data science teams with NVIDIA RAPIDS software that makes it easier to process trillions of records.

“NVIDIA SDKs serve healthcare, energy, transportation, retail, finance, media and entertainment — a combined $100 trillion of industries,” Huang said.

‘The Next Evolution’: Omniverse for Virtual Worlds

Half a century ago, the Apollo 13 lunar mission ran into trouble. To save the crew, Huang said, NASA engineers created a model of the crew capsule back on Earth to “work the problem.”

“Extended to vast scales, a digital twin is a virtual world that’s connected to the physical world,” Huang said. “And in the context of the internet, it is the next evolution.”

NVIDIA Omniverse software for building digital twins, and new data-center-scale NVIDIA OVX systems, will be integral for “action-oriented AI.”

“Omniverse is central to our robotics platforms,” Huang said, announcing new releases and updates for Omniverse. “And like NASA and Amazon, we and our customers in robotics and industrial automation realize the importance of digital twins and Omniverse.”

OVX will run Omniverse digital twins for large-scale simulations with multiple autonomous systems operating in the same space-time, Huang explained.

The backbone of OVX is its networking fabric, Huang said, announcing the NVIDIA Spectrum-4 high-performance data networking infrastructure platform.

The world’s first 400Gbps end-to-end networking platform, NVIDIA Spectrum-4 consists of the Spectrum-4 switch family, NVIDIA ConnectX-7 SmartNIC, NVIDIA BlueField-3 DPU and NVIDIA DOCA data center infrastructure software.

And to make Omniverse accessible to even more users, Huang announced Omniverse Cloud. Now, with just a few clicks, collaborators can connect through Omniverse on the cloud.

Huang showed how this works with a demo of four designers, one an AI, collaborating to build a virtual world.

He also showed how Amazon uses Omniverse Enterprise “to design and optimize their incredible fulfillment center operations.”

“Modern fulfillment centers are evolving into technical marvels — facilities operated by humans and robots working together,” Huang said.

The ‘Next Wave of AI’: Robots and Autonomous Vehicles

New silicon, new software and new simulation capabilities will unleash “the next wave of AI,” Huang said, robots able to “devise, plan and act.”

NVIDIA Avatar, DRIVE, Metropolis, Isaac and Holoscan are robotics platforms built end to end and full stack around “four pillars”: ground-truth data generation, AI model training, the robotics stack and Omniverse digital twins, Huang explained.

The NVIDIA DRIVE autonomous vehicle system is essentially an “AI chauffeur,” Huang said.

And Hyperion 8 — NVIDIA’s hardware architecture for self-driving cars on which NVIDIA DRIVE is built — can achieve full self-driving with a 360-degree camera, radar, lidar and ultrasonic sensor suite.

Hyperion 8 will ship in Mercedes-Benz cars starting in 2024, followed by Jaguar Land Rover in 2025, Huang said.

Huang announced that NVIDIA Orin, a centralized AV and AI computer that acts as the engine of new-generation EVs, robotaxis, shuttles, and trucks started shipping this month.

And Huang announced Hyperion 9, featuring the coming DRIVE Atlan SoC for double the performance of the current DRIVE Orin-based architecture, which will ship starting in 2026.

BYD, the second-largest EV maker globally, will adopt the DRIVE Orin computer for cars starting production in the first half of 2023, Huang announced.

And Lucid Motors revealed that its DreamDrive Pro advanced driver-assistance system is built on NVIDIA DRIVE.

Overall, NVIDIA’s automotive pipeline has increased to over $11 billion over the next six years.

Clara Holoscan puts much of the real-time computing muscle used in DRIVE to work supporting medical instruments and real-time sensors, such as RF ultrasound, 4K surgical video, high-throughput cameras and lasers.

Huang showed a video of Holoscan accelerating images from a light-sheet microscope — which creates a “movie” of cells moving and dividing.

It typically takes an entire day to process the 3TB of data these instruments produce in an hour.

At the Advanced Bioimaging Center at UC Berkeley, however, researchers using Holoscan are able to process this data in real-time, enabling them to auto-focus the microscope while experiments are running.

Holoscan development platforms are available for early access customers today, generally available in May, and medical-grade readiness in the first quarter of 2023.

NVIDIA is also working with thousands of customers and developers who are building robots for manufacturing, retail, healthcare, agriculture, construction, airports and entire cities, Huang said.

NVIDIA’s robotics platforms consist of Metropolis and Isaac — Metropolis is a stationary robot tracking moving things, while Isaac is a platform for things that move, Huang explained.

To help robots navigate indoor spaces — like factories and warehouses — NVIDIA announced Isaac Nova Orin, built on Jetson AGX Orin, a state-of-the-art compute and sensor reference platform to accelerate autonomous mobile robot development and deployment.

In a video, Huang showed how PepsiCo uses Metropolis and an Omniverse digital twin together.

Four Layers, Five Dynamics

Huang ended by tying all the technologies, product announcements and demos back into a vision of how NVIDIA will drive forward the next generation of computing.

NVIDIA announced new products across its four-layer stack: hardware, system software and libraries, software platforms NVIDIA HPC, NVIDIA AI, and NVIDIA Omniverse; and AI and robotics application frameworks, Huang explained.

Huang also ticked through the five dynamics shaping the industry: million-X computing speedups, transformers turbocharging AI, data centers becoming AI factories, which is exponentially increasing demand for robotics systems, and digital twins for the next era of AI.

“Accelerating across the full stack and at data center scale, we will strive for yet another million-X in the next decade,” Huang said, concluding his talk. “I can’t wait to see what the next million-X brings.”

Noting that Omniverse generated “every rendering and simulation you saw today,” Huang then introduced a stunning video put together by NVIDIA’s creative team featuring viewers “on one more trip into Omniverse” for a surprising musical jazz number set in the heart of NVIDIA’s campus featuring a cameo from Huang’s digital counterpart, Toy Jensen.

Bringing simulation of real and virtual worlds to everything from self-driving vehicles to avatars to robotics to modeling the planet’s climate, NVIDIA founder and CEO Jensen Huang Tuesday introduced technologies to transform multitrillion-dollar industries.

Huang delivered a keynote at the company’s virtual GTC gathering where he unveiled NVIDIA Omniverse Avatar and NVIDIA Omniverse Replicator, among a host of announcements, demos and far-reaching initiatives.

Huang showed how NVIDIA Omniverse — the company’s virtual world simulation and collaboration platform for 3D workflows — brings NVIDIA’s technologies together.

And he showed a demonstration of Project Tokkio for customer support and Project Maxine for video conferencing with Omniverse Avatar.

“A constant theme you’ll see — how Omniverse is used to simulate digital twins of warehouses, plants and factories, of physical and biological systems, the 5G edge, robots, self-driving cars, and even avatars,” Huang said.

Huang ended by announcing NVIDIA will build a digital twin, called E-2, or Earth Two, to simulate and predict climate change.

‘Full Stack, Data Center Scale, and Open Platform’

Accelerated computing launched modern AI, and the waves it started are coming to science and the world’s industries, Huang said.

That starts with three chips, the GPU, CPU and DPU, and systems — DGX, HGX, EGX, RTX and AGX — spanning from cloud to the edge, he said.

NVIDIA has created 150 acceleration libraries for 3 million developers, from graphics and AI to sciences and robotics.

And NVIDIA is introducing 65 new and updated SDKs at GTC.

“NVIDIA accelerated computing is a full-stack, data-center-scale and open platform,” Huang said.

Huang introduced NVIDIA Quantum-2, “the most advanced networking platform ever built,” and with the BlueField-3 DPU, welcomes cloud-native supercomputing.

Quantum-2 offers the extreme performance, broad accessibility and strong security needed by cloud computing providers and supercomputing centers, he said.

Cybersecurity is a top issue for companies and nations, and Huang announced a three-pillar zero-trust framework to tackle the challenge.

BlueField DPUs isolate applications from infrastructure. NVIDIA DOCA 1.2 — the latest BlueField SDK — enables next-generation distributed firewalls. NVIDIA Morpheus, assuming the interloper is already inside, uses the “superpowers of accelerated computing and deep learning to detect intruder activities,” Huang said.

Omniverse Avatar and Omniverse Replicator

With Omniverse, “we now have the technology to create new 3D worlds or model our physical world,” Huang said.-

To help developers create interactive characters with Omniverse that can see, speak, converse on a wide range of subjects and understand naturally spoken intent, NVIDIA announced Omniverse Avatar.

Huang showed how Project Maxine for Omniverse Avatar connects computer vision, Riva speech AI, and avatar animation and graphics into a real-time conversational AI robot — the Toy Jensen Omniverse Avatar.

He also showed a demo of Project Tokkio, a customer-service avatar in a restaurant kiosk, able to see, converse with and understand two customers.

Huang additionally showed Project Maxine’s ability to add state-of-the-art video and audio features to virtual collaboration and content creation applications.

A demo showed a woman speaking English on a video call in a noisy cafe, but she can be heard clearly without background noise. As she speaks, her words are transcribed and translated in real time into French, German and Spanish. And, thanks to Omniverse, they’re spoken by an avatar able to engage in conversation with her same voice and intonation.

To help developers to create the huge amounts of data needed to train AI, NVIDIA announced Omniverse Replicator,  a synthetic-data-generation engine for training deep neural networks.

NVIDIA has developed two replicators: one for general robotics, Omniverse Replicator for Isaac Sim, and another for autonomous vehicles, Omniverse Replicator for DRIVE Sim.

Since its launch late last year, Omniverse has been downloaded 70,000 times by designers at 500 companies.

Omniverse Enterprise is now available starting at $9,000 a year.

AI Models and Systems

Huang introduced Nemo Megatron to train large language models. Such models “will be the biggest mainstream HPC application ever,” he said.

Graphs — a key data structure in modern data science — can now be projected into deep-neural network frameworks with Deep Graph Library, or DGL, a new Python package.

NVIDIA Modulus, introduced Tuesday, builds and trains physics-informed machine learning models that can learn and obey the laws of physics.

An upgrade to Triton, an inference server for all workloads, now inferences forest models and does multi-GPU multi-node inference for large language models.

And Huang introduced three new libraries.

• cuOpt – for the logistics industry.
• cuQuantum – to accelerate quantum computing research.
• cuNumeric – to accelerate NumPy for scientists, data scientists and machine learning and AI researchers in the Python community.

To help deliver services based on NVIDIA’s AI technologies to the edge, Huang announced NVIDIA Launchpad.

NVIDIA is partnering with data center powerhouse Equinix to pre-install and integrate NVIDIA AI into data centers worldwide.

Robotics

NVIDIA’s Isaac ecosystem now has over 700 companies and partners, a number that has grown 5 times in the last 4 years.

Huang announced the NVIDIA Isaac robotics platform can now be easily integrated into the Robot Operating System, or ROS, a widely-used set of software libraries and tools for robot applications.

Isaac Sim, built on Omniverse, is the most realistic robotics simulator ever created, Huang explained.

“The goal is for the robot to not know whether it is inside a simulation or the real world,” Huang said.

To aid this process, Isaac Sim Replicator can generate synthetic data to train robots.

Replicator simulates the sensors, generates data that is automatically labeled, and with a domain randomization engine, creates rich and diverse training data sets, Huang explained.

Autonomous Vehicles

Everything that moves will be autonomous—fully or mostly autonomous, Huang said. “By 2024, the vast majority of new EVs will have substantial AV capability,” he added.

NVIDIA DRIVE is NVIDIA’s full-stack and open platform for autonomous vehicles, and Hyperion 8 is NVIDIA’s latest complete hardware and software architecture.

Its sensor suite includes 12 cameras, nine radars, 12 ultrasonics and one front-facing lidar. All processed by two NVIDIA Orin SoCs.

Huang detailed several new technologies built into Hyperion, including Omniverse Replicator for DRIVE Sim, a synthetic data generator for autonomous vehicles built on Omniverse.

NVIDIA is now running Hyperion 8 sensors, 4d perception, deep learning-based multisensor fusion, feature tracking and a new planning engine.

The inside of the car will be revolutionized, too.  The technology of NVIDIA Maxine will reimagine how we interact with our cars.

“With Maxine, your car will become a concierge,” Huang said.

Earth Two

Huang wrapped up by announcing NVIDIA will build Earth Two, or E-2, to simulate and predict climate change.

“All the technologies we’ve invented up to this moment are needed to make Earth Two possible,” Huang said.

The demand for specialization in AI and robotics is growing more than ever.

Roles in these two fields are multiplying across many industries, with titles such as AI specialist and robotics engineer, respectively, placing as the first- and second-fastest emerging jobs in the U.S., according to the most recent Emerging Jobs Report from LinkedIn.

With demand high, upskilling in these two fields is vital.

NVIDIA offers several opportunities and initiatives for students and developers to get hands-on experience in AI and robotics.

NVIDIA GTC, running Nov. 8-11, will offer a number of sessions aimed at equipping educators with the appropriate tools to integrate into their curricula, and many more to inspire makers and enthusiasts just getting started or already innovating with AI and robotics.

Educators around the world — from Kenya and Taiwan, to Pittsburgh and Houston — are bringing AI and robotics into the hands of students in their local communities. And they’ve been doing it with NVIDIA resources, such as the NVIDIA Emerging Chapters program, the NVIDIA Deep Learning Institute, NVIDIA Jetson Nano 2GB Developer Kit Grant Program and learning opportunities offered through GTC.

Higher Ed at GTC

GTC will feature several institutions in higher education that are offering application-based learning within the AI and robotics space.

Professors at the University of Oxford and the University of Maryland Baltimore College (pictured above) have collaborated with NVIDIA to create an Edge AI and Robotics Teaching Kit with interactive lectures and hands-on labs.

The course uses the NVIDIA Jetson Edge AI platform to teach fundamentals of computer vision-based deep neural networks, autonomous navigation, reinforcement learning, security and ethics, and conversational AI. The teaching kit is freely available and open source for educators to integrate into their own courses and curricula.

Moreover, the University of Manchester and Coursera have partnered with NVIDIA to design a series of courses with content on both basic and advanced robotics principles, including camera-based navigation, map-based localization and computer vision.

The final project of the course will involve learning about intelligent navigation and exploration with the PuzzleBot Jetson Edition, a configurable robotics platform equipped with NVIDIA Jetson Nano for robot vision and intelligent robot navigation.

AI & Robotics 101

GTC also offers an abundance of courses that provide basic education on AI and robotics, all of which are meant to spur interactive applications.

Beginners who’d like to delve into the basics of AI should add these sessions to their calendars: “Begin Your AI Journey With NVIDIA Jetson Nano” (A31723) and “Getting Started With the Edge AI and Robotics Teaching Kit” (A31535).

For those interested in learning more about robotics and autonomous machines, GTC includes sessions on how to develop features and increase capabilities of robots with NVIDIA Isaac ROS (A31574), updates on the NVIDIA Isaac Gym environment for high-performance reinforcement learning (A31118), and how to leverage Isaac Sim 2021.2 for simulation manipulation, navigation and synthetic data generation (A31573).

To explore how to apply some of this knowledge to real-world tools and use cases, check out these sessions: “Your First Steps to Design an Intelligent Assistant for Hands-Free Applications” (A31186) and “Train. Adapt. Optimize. Supercharge Your AI Development Workflow and Application Development With NVIDIA TAO Toolkit” (A31193).

Register for GTC

Register for free to learn more about NVIDIA Jetson during GTC. And watch NVIDIA founder and CEO Jensen Huang’s GTC keynote address streaming on Nov. 9 and in replay.

Two artists, if given the same paint set, would create distinct works — each showcasing their own points of view. This is especially likely if the artists were to hail from opposite ends of the world.

Similarly, developers across the globe use the same tools to create different AI-based applications, each solving challenges relevant to their local community.

Give an NVIDIA Jetson Nano Developer Kit to an AI enthusiast in San Francisco, for example, and they will create an AI-based app that analyzes yoga poses. With that same tool, developers in the tinyML Kenya community are building AI systems for wildlife monitoring and conservation. Innovation is happening in each place, so long as the developers have access to technologies and enablement tools.

NVIDIA Emerging Chapters is a new program that enables local communities in emerging areas to build and scale their AI, data science and graphics projects. It provides technological tools, educational resources and co-marketing opportunities for these developers.

Members of Emerging Chapters have access to training and development opportunities through the NVIDIA Deep Learning Institute (DLI). This includes free passes to take select self- or instructor-led courses on AI and data science. Upon course completion, developers can receive an NVIDIA DLI Certificate to highlight their new skills and help advance their careers.

Since the program’s launch this year, 30+ African developer community groups — including seven founded by women — have joined Emerging Chapters, fostering a growing network of AI experts. The program is now expanding to Latin America, the Middle East, South Asia and other emerging markets that are hungry for more AI and related technology.

Reflecting the World’s Diversity

With Emerging Chapters, NVIDIA hopes to help mend what’s called the technology fracture — the gap between developers in the global North and those in emerging markets.

“This program is not about charity, it’s about innovation and business,” said Amulya Vishwanath, strategic program lead on NVIDIA’s emerging areas team. “It’s crucial to get communities in emerging areas access to AI technology, which they can then use to make their own, ensuring the developer community is more reflective of the world’s true diversity.”

By spurring innovation in collaboration with local communities, the program can cultivate AI solutions that most pertain to grassroot developers and their direct ecosystem, while also democratizing the global AI movement.

“With Emerging Chapters, NVIDIA is taking active steps toward positively contributing to the growing trend of technology in emerging markets,” said Michael Young, co-founder of the Python Ghana community. “NVIDIA helped us bring growing AI engineers from across the country together for live, online training sessions with experts.”

Fueling Innovation in Africa

Africa is an emerging market in which the AI revolution is underway. African developers are using AI and NVIDIA technology, for example, to maximize crop yields and honor Olympic athletes.

Early members of the Emerging Chapters program include DeepLearning.AI Kenya, an education technology company that empowers individuals in the AI workforce; NERD Ethiopia, a youth center that provides a hacker space and educational resources for AI research; and tinyML Kenya, a community of machine learning researchers and practitioners.

“TinyML Kenya joining NVIDIA Emerging Chapters was such a timely, game-changing move for our community,” said Clinton Oduor, the foundation’s lead organizer. “The program has allowed our members to meet industry leaders, learn new skills, earn certifications and solve real-world problems.”

Zindi, Africa’s first data science competition platform, is also a member. The organization has an ambassador program made up of data science community leaders from 20+ African countries.

“It’s great to be able to support people doing such excellent work with the help of the Emerging Chapters program,” said Celina Lee, Zindi’s co-founder and CEO.

Allowing AI Access for All

To bolster individuals who seek to work with AI, HPC, graphics and more, NVIDIA offers a range of opportunities including the NVIDIA Developer Program — which has more than 2.5 million members — and NVIDIA Inception, which offers go-to-market support, expertise and technology for AI and data science startups.

In addition, the company’s GTC conference, running Nov. 8-11, will feature a track focused on emerging markets. The conference is virtual, free and has sessions 24/7.

Over 20,000 developers from emerging markets tuned in to learn about AI innovation at our last GTC — and the appetite for developer resources is only growing.

For more on global AI innovation, register for GTC.

Read more stories about NVIDIA Emerging Chapters and join the program.

NVIDIA researchers are defining ways to make faster AI chips in systems with greater bandwidth that are easier to program, said Bill Dally, NVIDIA’s chief scientist, in a keynote released today for a virtual GTC China event.

He described three projects as examples of how the 200-person research team he leads is working to stoke Huang’s Law — the prediction named for NVIDIA CEO Jensen Huang that GPUs will double AI performance every year.

“If we really want to improve computer performance, Huang’s Law is the metric that matters, and I expect it to continue for the foreseeable future,” said Dally, who helped direct research at NVIDIA in AI, ray tracing and fast interconnects.

An Ultra-Efficient Accelerator

Toward that end, NVIDIA researchers created a tool called MAGNet that generated an AI inference accelerator that hit 100 tera-operations per watt in a simulation. That’s more than an order of magnitude greater efficiency than today’s commercial chips.

MAGNet uses new techniques to orchestrate the flow of information through a device in ways that minimize the data movement that burns most of the energy in today’s chips. The research prototype is implemented as a modular set of tiles so it can scale flexibly.

A separate effort seeks to replace today’s electrical links inside systems with faster optical ones.

Firing on All Photons

“We can see our way to doubling the speed of our NVLink [that connects GPUs] and maybe doubling it again, but eventually electrical signaling runs out of gas,” said Dally, who holds more than 120 patents and chaired the computer science department at Stanford before joining NVIDIA in 2009.

The team is collaborating with researchers at Columbia University on ways to harness techniques telecom providers use in their core networks to merge dozens of signals onto a single optical fiber.

Called dense wavelength division multiplexing, it holds the potential to pack multiple terabits per second into links that fit into a single millimeter of space on the side of a chip, more than 10x the density of today’s interconnects.

Besides faster throughput, the optical links enable denser systems. For example, Dally showed a mockup (below) of a future NVIDIA DGX system with more than 160 GPUs.

In software, NVIDIA’s researchers have prototyped a new programming system called Legate. It lets developers take a program written for a single GPU and run it on a system of any size — even a giant supercomputer like Selene that packs thousands of GPUs.

Legate couples a new form of programming shorthand with accelerated software libraries and an advanced runtime environment called Legion. It’s already being put to the test at U.S. national labs.

Rendering a Vivid Future

The three research projects make up just one part of Dally’s keynote, which describes NVIDIA’s domain-specific platforms for a variety of industries such as healthcare, self-driving cars and robotics. He also delves into data science, AI and graphics.

“In a few generations our products will produce amazing images in real time using path tracing with physically based rendering, and we’ll be able to generate whole scenes with AI,” said Dally.

He showed the first public demonstration that combines NVIDIA’s conversational AI framework called Riva with GauGAN, a tool that uses generative adversarial networks to create beautiful landscapes from simple sketches. The demo lets users instantly generate photorealistic landscapes using simple voice commands.

In an interview between recording sessions for the keynote, Dally expressed particular pride for the team’s pioneering work in several areas.

“All our current ray tracing started in NVIDIA Research with prototypes that got our product teams excited. And in 2011, I assigned [NVIDIA researcher] Bryan Catanzaro to work with [Stanford professor] Andrew Ng on a project that became CuDNN, software that kicked off much of our work in deep learning,” he said.

A First Foothold in Networking

Dally also spearheaded a collaboration that led to the first prototypes of NVLink and NVSwitch, interconnects that link GPUs running inside some of the world’s largest supercomputers today.

“The product teams grabbed the work out of our hands before we were ready to let go of it, and now we’re considered one of the most advanced networking companies,” he said.

With his passion for technology, Dally said he often feels like a kid in a candy store. He may hop from helping a group with an AI accelerator one day to helping another team sort through a complex problem in robotics the next.

“I have one of the most fun jobs in the company if not in the world because I get to help shape the future,” he said.

The keynote is just one of more than 220 sessions at GTC China. All the sessions are free and most are conducted in Mandarin.

Panel, Startup Showcase at GTC China

Following the keynote, a panel of senior NVIDIA executives will discuss how the company’s technologies in AI, data science, healthcare and other fields are being adopted in China.

The event also includes a showcase of a dozen top startups in China, hosted by NVIDIA Inception, an acceleration program for AI and data science startups.

Companies participating in GTC China include Alibaba, AWS, Baidu, ByteDance, China Telecom, Dell Technologies, Didi, New H3C Information Technologies, Inspur Electronic Information, Kuaishou, Lenovo, Microsoft, Ping An, Tencent, Tsinghua University and Xiaomi.

The NVIDIA DRIVE ecosystem is going multidimensional.

During the NVIDIA GPU Technology Conference this week, autonomous trucking startup Locomation and simulation company Blackshark.ai announced technological developments powered by NVIDIA DRIVE.

Locomation, a Pittsburgh-based provider of autonomous trucking technology, said it would integrate NVIDIA DRIVE AGX Orin in the upcoming rollout of its platooning system on public roads in 2022.

Innovating in the virtual world, Blackshark.ai detailed its toolset to create buildings and landscape assets for simulation environments on NVIDIA DRIVE Sim.

Together, these announcements mark milestones in the path toward safer, more efficient autonomous transportation.

Shooting for the Platoon

Locomation recently announced its first commercial system, Autonomous Relay Convoy, which allows one driver to pilot a lead truck while a fully autonomous follower truck operates in tandem.

The ARC system will be deployed with Wilson Logistics, which will operate more than 1,000 Locomation-equipped trucks, powered by NVIDIA DRIVE AGX Orin, starting in 2022.

NVIDIA DRIVE AGX Orin is a highly advanced software-defined platform for autonomous vehicles.  The system features the new Orin system-on-a-chip, which delivers more than 200 trillion operations per second — nearly 7x the performance of NVIDIA’s previous-generation Xavier SoC.

In August, Locomation and Wilson Logistics successfully completed the first-ever on-road pilot program transporting commercial freight using ARC. Two Locomation trucks, hauling Wilson Logistics trailers and freight, were deployed on a 420-mile long route along I-84 between Portland, Ore., and Nampa, Idaho. This stretch of interstate has some of the most challenging road conditions for truck driving, with curvatures, inclines and wind gusts.

“We’re moving rapidly toward autonomous trucking commercialization, and NVIDIA DRIVE presents a solution for providing a robust, safety-forward platform for our team to work with,” said Çetin Meriçli, CEO and cofounder of Locomation.

Constructing a New Dimension

While Locomation is deploying autonomous vehicles in the real world, Blackshark.ai is making it easier to create building and landscape assets used to enhance the virtual world on a global scale.

The startup has developed a digital twin platform that uses AI and cloud computing to automatically transform satellite data, aerial images or map and sensor data into building, landscape and infrastructure assets that contribute to a semantic photorealistic 3D environment.

During the opening GTC keynote, NVIDIA founder and CEO Jensen Huang showcased the technology on NVIDIA DRIVE Sim. DRIVE Sim uses high-fidelity simulation to create a safe, scalable and cost-effective way to bring self-driving vehicles to our roads.

It taps into the computing horsepower of NVIDIA RTX GPUs to deliver a powerful, scalable, cloud-based computing platform. One that is capable of generating billions of qualified miles for autonomous vehicle testing.

In the demo video, Blackshark’s AI automatically generated the trees and buildings used to reconstruct the city of San Jose in simulation for an immersive, authentic environment.

These latest announcements from Locomation and Blackshark.ai demonstrate the breadth of the DRIVE ecosystem, spanning the real and virtual worlds to push autonomous innovation further.

Watch NVIDIA CEO Jensen Huang recap all the news from GTC. It’s not too late to get access to hundreds of live and on-demand talks — register now through Oct. 9 using promo code CMB4KN to get 20 percent off.

Humans are wielding AI to create art, and a virtual exhibit that’s part of NVIDIA’s GPU Technology Conference showcases the stunning results.

The AI Art Gallery at NVIDIA GTC features pieces by a broad collection of artists, developers and researchers from around the world who are using AI to push the limits of artistic expression.

When AI is introduced into the artistic process, the artist feeds the machine data and code, explains NVIDIA’s Heather Schoell senior art director at NVIDIA, who curated the online exhibit.

Once the output reveals itself, it’s up to the artist to determine if it stands up to their artistic style and desired message, or if the input needs to be adjusted, according to Schoell.

“The output reflects both the artist’s hand and the medium, in this case data, used for creation,” Schoell says.

The exhibit complements what has become the world’s premier AI conference.

GTC, running Oct. 5-9, will bring together researchers from industry and academia, startups and Fortune 500 companies.

So it’s only natural that artists would be among those putting modern AI to work.

“Through this collection we aim to share how the artist can partner with AI as both an artistic medium and creative collaborator,” Schoell explains.

The artists featured in the AI Art Gallery include:

• Daniel Ambrosi – “Dreamscapes” fuses computational photography and AI to create a deeply textural environment.
• Refik Anadol – “Machine Hallucinations,” by the Turkish-born, Los Angeles-based conceptual artist known for his immersive architectural digital installations, such as a project at New York’s Chelsea Market that used projectors to splash AI generated images based of New York cityscapes to create what Anadol called a “machine hallucination.”
• Sofia Crespo and Dark Fractures – Work from the Argentina-born artist and Berlin-based studio led by Feileacan McCormick uses GANs and NLP models to generate 3D insects in a virtual, digital space.
• Scott Eaton – An artist, educator and creative technologist residing in London combines a deep understanding of human anatomy, traditional art techniques and modern digital tools in his uncanny, figurative artworks.
• Oxia Palus – The NVIDIA Inception startup will uncover a new masterpiece by Leonardo Da Vinci that resurrects a hidden sketch and reconstructs the painting style from one of the most famous artists of all time, DaVinci.
• Anna Ridler – Three displays showing images of tulips that change based on Bitcoin’s price, created by a U.K. artist and researcher known for her work exploring the intersection of machine learning, nature and history.
• Helena Sarin – Using her own drawings, sketches, and photographs as datasets, Helena trains her models to generate new visuals that serve as the basis of her compositions — in this case with type of neural network known as a generative adversarial network, or GAN. The Moscow-born artist has embedded 12 of these creations in a book of puns on the acronym GAN.
• Pindar Van Arman – Driven by a collection of algorithms programmed to work with — and against — one another, the U.S.-based artist and roboticist’s creation uses a paintbrush, paint and canvas to create portraits that fuse the look and feel of a photo and a handmade sketch.

For a closer look, registered GTC attendees can go on a live, personal tour of two of our featured artists’ studios.

On Thursday, Oct. 8, you can virtually tour Van Arman’s Fort Worth, Texas, studio between 11-12 p.m. Pacific time. And at 2 p.m. Pacific, you can tour Refik Anadol’s Los Angeles studio.

In addition, a pair of panel discussions, Thursday, October 8, with AI Gallery artists will explore what led them to connect AI and fine art.

And starting Oct. 5, you can tune in to an on-demand GTC session featuring Oxia Palus co-founder George Cann, a PhD candidate in space and climate physics at University College London.

Join us at the AI Art Gallery.

Register for GTC. 

Strap in, because this fall you can access the greatest minds in autonomous driving from around the world at any hour, from anywhere.

GTC returns Oct. 5, running 24 hours a day through Oct. 9 to deliver the best in AI sessions, training and more from the foremost experts. NVIDIA founder and CEO Jensen Huang will kick off the conference with a keynote video highlighting the latest innovations in GPU computing.

With digital networking events and live sessions, GTC provides the unique opportunity to interact with researchers, engineers, developers and technologists in the autonomous vehicle space, as well as healthcare, robotics, graphics and numerous other industries.

Here’s a sneak peek at the automotive offerings at this year’s event.

End-to-End Experts

Global automakers, suppliers, startups and researchers will present their most recent work in live and recorded sessions. These talks will cover every step of the autonomous vehicle development process, from training, to testing and validation, to deployment at scale.

• Alexander Amini, a doctoral candidate at MIT, details a new end-to-end learning strategy for training self-driving neural networks, from perception to control.
• Vijitha Chekuri, business strategy director at Microsoft, discusses how hyperscale cloud and cutting-edge compute can be used for comprehensive autonomous vehicle simulation, training and validation.
• Chuck Price, chief product officer at TuSimple, outlines the future deployment of autonomous trucks, including capabilities, market expectations and the unique challenges in developing a self-driving commercial vehicle.
• Neda Cvijetic, senior manager of autonomous vehicles at NVIDIA, applies an engineering focus to widely acknowledged autonomous vehicle challenges, and explains how NVIDIA is tackling them.
• Karl Greb, safety engineering director at NVIDIA, describes the key elements of the processes, methodologies and technologies behind the safe and robust NVIDIA DRIVE platform.

Round-the-Clock Training

In addition to the virtual keynote and sessions, GTC will offer hands-on deep learning training across a variety of time zones.

Hosted by the NVIDIA Deep Learning Institute, these courses will provide a foundation for developing AI applications, including autonomous driving. Topics include the fundamentals of deep learning, natural language processing and deep learning for multiple GPUs.

Register today for GTC Early Bird pricing, available for just $49 through Friday, Sept. 25. DLI training — which includes competency certification for many sessions — can be added to a Digital Conference Pass for just $99 per session.

As much of the world continues to conduct business from home, NVIDIA’s autonomous test vehicles are hard at work in the cloud.

During the GTC 2020 keynote, NVIDIA CEO Jensen Huang demonstrated how NVIDIA DRIVE technology is being developed and tested in simulation. While physical testing is temporarily paused, the cloud-based NVIDIA DRIVE Constellation platform makes it possible to dispatch virtual vehicles in virtual environments to continue making great progress in self-driving technology.

In the video demonstration, a virtual NVIDIA BB8 test vehicle drives near NVIDIA headquarters in Silicon Valley, traveling through highways and urban streets — all in simulation. The 17-mile loop shows the NVIDIA DRIVE AV Software navigating the roadways, pedestrians and traffic in a highly accurate replica environment.

Data Center Proving Ground

NVIDIA DRIVE Constellation is a cloud-based simulation platform, designed from the ground up to support the development and validation of autonomous vehicles. The data center-based platform consists of two side-by-side servers.

The first server uses NVIDIA GPUs running DRIVE Sim software and generates the sensor output from the virtual car driving in a virtual world. The second server contains the actual vehicle computer, processing the simulated sensor data running the exact same DRIVE AV and DRIVE IX software that’s being deployed in the real car.

The driving decisions from the second server are fed back into the first, enabling real-time, bit-accurate, hardware-in-the-loop development and testing.

The system is designed to be deployed in a data center as a scalable virtual fleet. This provides development engineers with a vehicle on demand, and gives them the ability to conduct testing at scale. It also makes it possible to consistently test rare and dangerous scenarios that are difficult or impossible to encounter in the real world.

Development and Testing from End to End

Building an autonomous vehicle requires testing at every level — starting at subsystems and continuing all the way to full vehicle integration tests. DRIVE Constellation enables this type of end-to-end development and testing for autonomous vehicles in simulation, similar to developing a physical car.

End-to-end tests ensure timing and performance accuracy as well as accurate modeling of the complex interdependency of different systems in autonomous vehicle software.

Achieving this level of fidelity at scale is a major undertaking. The environment, traffic behavior, sensor inputs and vehicle dynamics must appear, act and feed into the car computer just as they would in the real world.

This requires multiple GPUs to generate synthetic data in sync with precise timing. The vehicle software and hardware signals and interfaces must be replicated in simulation — and everything has to run in real time.

Simulating Silicon Valley

Comprehensive simulation starts with the environment. To accurately recreate the Silicon Valley driving loop, 3D Mapping, a member of the NVIDIA DRIVE ecosystem, scanned the roadways to within 5 centimeters of accuracy. The raw scanned data was then processed into a dataset format known as OpenDRIVE.

From there, NVIDIA developed a content creation pipeline to generate a highly accurate 3D environment using the NVIDIA Omniverse collaboration platform. The environment includes accurate road networks and roadmarks. Material properties are also applied to ensure it interacts with light rays, radio waves and lidar rays in the same way real sensors interact with the physical world.

Recreating Sensor Data

With an accurate environment in place, high-fidelity development and testing next requires accurately generated sensor data. The sensor models include those typically found on an autonomous test vehicle, such as camera, lidar, radar and inertial measurement unit. DRIVE Sim provides a flexible sensor pipeline and APIs that allow configuring sensors to match real-world vehicle architectures.

For camera data, the image pipeline starts by rendering an HDR image that is warped according to the lens properties of the camera used on the vehicle. Exposure control, black and white level balancing, and color grading are applied to the image to match the sensor profile. Finally, the pixel data is converted to its native output format using a sensor-specific encoder.

In addition to camera models, DRIVE Sim provides physically based lidar and radar sensors using ray tracing. NVIDIA RTX GPUs enable DRIVE Sim to run highly computationally intensive radar and lidar models in real time.

Modeling Vehicle Behavior

Finally, vehicle models are critical for accurate simulation. As control signals — steering, acceleration and braking — are sent to the in-vehicle computer, the car must respond just as it would in the physical world.

To do so, the simulation platform must recreate motion properly, including details such as interaction with the road surface. Vehicle models in DRIVE Sim are handled using a plugin system with the included PhysX models or third-party vehicle dynamics models from NVIDIA DRIVE ecosystem partners such as Mechanical Simulation or IPG.

Vehicle dynamics also play a key role in accurate sensor data generation. As the vehicle operates, the position and pose of the vehicle changes significantly, affecting a sensor’s viewpoint. For example, the forward-facing cameras will pitch downward when a car is braking. Modeling vehicle dynamics correctly is important to generating sensor data properly.

By accurately simulating each of these components — environment, sensors, vehicle dynamics — on a single, end-to-end platform, NVIDIA DRIVE Constellation and DRIVE Sim are critical pieces to a comprehensive development and testing pipeline. They enable NVIDIA and its partners to work toward safer and more efficient autonomous vehicles as physical fleets remain in the garage.

NVIDIA DRIVE AGX is giving automotive industry startups an AI-powered boost.

During GTC Digital, electric and autonomous vehicle startups Pony.ai, Canoo and Faraday Future announced they are developing vehicles using the NVIDIA DRIVE AGX compute platform. The high-performance, energy-efficient platform enables automated and autonomous driving across all levels for robust, software-defined vehicle development.

These companies are joining a wide, international ecosystem of automakers, tier 1 suppliers, truck makers, sensor suppliers, robotaxi companies and software startups developing on NVIDIA DRIVE.

By selecting an open and scalable platform, the DRIVE ecosystem is developing autonomous vehicles that are always improving, with over-the-air update capabilities, for a safer, more efficient transportation future.

Redefining Mobility and Delivery

By augmenting the human driver with AI, autonomous driving technology promises to significantly improve everyday mobility and logistics.

Pony.ai, an autonomous driving technology company, is developing its upcoming robotaxi fleet on NVIDIA DRIVE AGX Pegasus. The company has been operating autonomous ride-hailing test vehicles since 2018, in California and China.

In April, Pony.ai began providing autonomous delivery services in Irvine, Calif., to help those in the area sheltering in place due to COVID-19.

The startup said it will leverage the DRIVE AGX Pegasus self-driving platform to meet the massive computing demands required to bring robotaxis to market. The AI compute platform achieves 320 trillion operations per second (TOPS) of deep learning and integrates two NVIDIA Xavier processors and two NVIDIA Turing Tensor Core GPUs.

With the ability to process a range of redundant and diverse deep neural networks simultaneously, Pony.ai can focus on developing safe and sustainable mobility technology for passenger travel and delivery.

A New Vision for Personal Vehicles

Electric vehicle startup Canoo has unveiled a sleek EV resembling a futuristic take on the iconic Volkswagen Microbus. The vehicles, purpose-built for a shared mobility system, will go into production in late 2021.

Canoo’s vehicles will feature AI-assisted driving features powered by NVIDIA DRIVE AGX Xavier. The compute platform delivers 30 TOPS of performance for object detection and sensor fusion, running state-of-the-art algorithms to provide cross-traffic alerts, blind spot detection and pedestrian detection, as well as convenience features such as adaptive cruise control and lane-centering control.

The software-defined DRIVE AGX Xavier also allows for more advanced features, like auto lane change, traffic light recognition and evasive steering, to be introduced when they become available.

Luxury EV maker Faraday Future also announced this week it will develop its upcoming FF91 vehicle using DRIVE AGX Xavier. With high-performance, energy-efficient compute at its core, the FF91 incorporates more than 36 sensors for advanced autonomous driving capabilities. The flagship EV is expected to begin deliveries by the end of this year.

By developing on the scalable DRIVE AGX platform, these startups, as well as the entire DRIVE ecosystem, can continue to build more advanced features and continuously deliver truly intelligent transportation.

Intelligent vehicles require intelligent development.

That’s why NVIDIA has built a complete AI-powered portfolio — from data centers to in-vehicle computers — that enables software-defined autonomous vehicles. And this month during GTC Digital, we’re providing an inside look at how this development process works, plus how we’re approaching safer, more efficient transportation.

Autonomous vehicles must be able to operate in thousands of conditions around the world to be truly driverless. The key to reaching this level of capability is mountains of data.

To put that in perspective, a fleet of just 50 vehicles driving six hours a day generates about 1.6 petabytes of sensor data a day. If all that data were stored on standard 1GB flash drives, they’d cover more than 100 football fields. This data must then be curated and labeled to train the deep neural networks (DNNs) that will run in the car, performing a variety of dedicated functions, such as object detection and localization.

The infrastructure to train and test this software must include high-performance supercomputers to handle these enormous data needs. To run efficiently, the system must be able to intelligently curate and organize this data. Finally, it must be traceable — making it easy to find and fix bugs in the process — and repeatable, going over the same scenario over and over again to ensure a DNN’s proficiency.

As part of the GTC Digital series, we present this complete development and training infrastructure as well as some of the DNNs it has produced, driving progress toward deploying the car of the future.

Born and Raised in the Data Center

While today’s vehicles are put together on the factory floor assembly line, autonomous vehicles are born in the data center. In a GTC digital session, Clemént Farabet, vice president of AI Infrastructure at NVIDIA, details this high-performance, end-to-end platform for autonomous vehicle development.

The NVIDIA internal AI infrastructure includes NVIDIA DGX servers that store and process the petabytes of driving data. For comprehensive training, developers must work with five to 10 billion frames to develop and then evaluate a DNN’s performance.

High-performance data center GPUs help speed up the time it takes to process this data. In addition, Farabet’s team optimizes development times using advanced learning methods such as active learning.

Rather than rely solely on humans to curate and label driving data for DNN training, active learning makes it possible for the DNN to choose the data it needs to learn from. A dedicated neural network goes through a pool of frames, flagging those in which it demonstrates uncertainty. The flagged frames are then labeled manually and used to train the DNN, ensuring that it’s learning from the exact data that’s new or confusing.

Once trained, these DNNs can then be tested and validated on the NVIDIA DRIVE Constellation simulation platform. The cloud-based solution enables millions of miles to be driven in virtual environments across a broad range of scenarios — from routine driving to rare or even dangerous situations — with greater efficiency, cost-effectiveness and safety than what is possible in the real world.

DRIVE Constellation’s high-fidelity simulation ensures these DNNs can be tested over and over, in every possible scenario and every possible condition before operating on public roads.

When combined with data center training, simulation allows developers to constantly improve upon their software in an automated, traceable and repeatable development process.

• Watch on-demand: NVIDIA’s AI Infrastructure for Self-Driving Cars

DNNs at the Edge

Once trained and validated, these DNNs can then operate in the car.

During GTC Digital, Neda Cvijetic, NVIDIA senior manager of autonomous vehicles and host of the DRIVE Labs video series, gave  an inside look at a sampling of self-driving DNNs we’ve developed.

Autonomous vehicles run an array of DNNs covering perception, mapping and localization to operate safely. To humans, these tasks seem straightforward, however, they’re all complex processes that require intelligent approaches to be performed successfully.

For example, to classify road objects, pedestrians and drivable space, one DNN uses a process known as panoptic segmentation, which can identify a scene with pixel-level accuracy.

To help it perceive parking spaces in a variety of environments, developers taught the ParkNet DNN to identify a spot as a four-sided polygon rather than a rectangle, so it could discern slanted spaces as well as their entry points.

And our LidarNet DNN addresses challenges in processing lidar data for localization by fusing multiple perspectives for accurate and complete perception information.

By combining these and other DNNs and running them on high-performance in-vehicle compute, such as the NVIDIA DRIVE AGX platform, an autonomous vehicle can perform comprehensive perception and planning and control without a human driver.

• Watch on-demand: NVIDIA DRIVE Labs: An Inside Look at Autonomous Vehicle Software

The GTC Digital site hosts these and other free sessions, with new content from NVIDIA experts and the DRIVE ecosystem added every Thursday until April 23. Stay up to date and register here.