At the Adobe MAX creativity conference this week, Adobe announced updates to its Adobe Creative Cloud products, including Premiere Pro and After Effects, as well as to Substance 3D products and the Adobe video ecosystem.

These apps are accelerated by NVIDIA RTX and GeForce RTX GPUs — in the cloud or running locally on RTX AI PCs and workstations.

One of the most highly anticipated features is Generative Extend in Premiere Pro (beta), which uses generative AI to seamlessly add frames to the beginning or end of a clip. Powered by the Firefly Video Model, it’s designed to be commercially safe and only trained on content Adobe has permission to use, so artists can create with confidence.

Adobe Substance 3D Collection apps offer numerous RTX-accelerated features for 3D content creation, including ray tracing, AI delighting and upscaling, and image-to-material workflows powered by Adobe Firefly.

Substance 3D Viewer, entering open beta at Adobe MAX, is designed to unlock 3D in 2D design workflows by allowing 3D files to be opened, viewed and used across design teams. This will improve interoperability with other RTX-accelerated Adobe apps like Photoshop.

Adobe Firefly integrations have also been added to Substance 3D Collection apps, including Text to Texture, Text to Pattern and Image to Texture tools in Substance 3D Sampler, as well as Generative Background in Substance 3D Stager, to further enhance the 3D content creation with generative AI.

The October NVIDIA Studio Driver, designed to optimize creative apps, will be available for download tomorrow. For automatic Studio Driver notifications, as well as easy access to apps like NVIDIA Broadcast, download the NVIDIA app beta.

Video Editing Evolved

Adobe Premiere Pro has transformed video editing workflows over the last four years with features like Auto Reframe and Scene Edit Detection.

The recently launched GPU-accelerated Enhance Speech, AI Audio Category Tagging and Filler Word Detection features allow editors to use AI to intelligently cut and modify video scenes.

The Adobe Firefly Video Model — now available in limited beta at Firefly.Adobe.com — brings generative AI to video, marking the next advancement in video editing. It allows users to create and edit video clips using simple text prompts or images, helping fill in content gaps without having to reshoot, extend or reframe takes. It can also be used to create video clip prototypes as inspiration for future shots.

Topaz Labs has introduced a new plug-in for Adobe After Effects, a video enhancement software that uses AI models to improve video quality. This gives users access to enhancement and motion deblur models for sharper, clearer video quality. Accelerated on GeForce RTX GPUs, these models run nearly 2.5x faster on the GeForce RTX 4090 Laptop GPU compared with the MacBook Pro M3 Max.

Stay tuned for NVIDIA TensorRT enhancements and more Topaz Video AI effects coming to the After Effects plug-in soon.

3D Super Powered

The Substance 3D Collection is revolutionizing the ideation stage of 3D creation with powerful generative AI features in Substance 3D Sampler and Stager.

Sampler’s Text to Texture, Text to Pattern and Image to Texture tools, powered by Adobe Firefly, allow artists to rapidly generate reference images from simple prompts that can be used to create parametric materials.

Stager’s Generative Background feature helps designers explore backgrounds for staging 3D models, using text descriptions to generate images. Stager can then match lighting and camera perspective, allowing designers to explore more variations faster when iterating and mocking up concepts.

Substance 3D Viewer also offers a connected workflow with Photoshop, where 3D models can be placed into Photoshop projects and edits made to the model in Viewer will be automatically sent back to the Photoshop project. GeForce RTX GPU hardware acceleration and ray tracing provide smooth movement in the viewport, producing up to 80% higher frames per second on the GeForce RTX 4060 Laptop GPU compared to the MacBook M3 Pro.

There are also new Firefly-powered features in Substance 3D Viewer, like Text to 3D and 3D Model to Image, that combine text prompts and 3D objects to give artists more control when generating new scenes and variations.

The latest After Effects release features an expanded range of 3D tools that enable creators to embed 3D animations, cast ultra-realistic shadows on 2D objects and isolate effects in 3D space.

After Effects now also has an RTX GPU-powered Advanced 3D Renderer that accelerates the processing-intensive and time-consuming task of applying HDRI lighting — lowering creative barriers to entry while improving content realism. Rendering can be done 30% faster on a GeForce RTX 4090 GPU over the previous generation.

Pairing Substance 3D with After Effects native and fast 3D integration allows artists to significantly boost the visual quality of 3D in After Effects with precision texturing and access to more than 20,000 parametric 3D materials, IBL environment lights and 3D models.

Follow NVIDIA Studio on Instagram, X and Facebook. Access tutorials on the Studio YouTube channel and get updates directly in your inbox by subscribing to the Studio newsletter. 

Generative AI is transforming gaming, videoconferencing and interactive experiences of all kinds. Make sense of what’s new and what’s next by subscribing to the AI Decoded newsletter.

As hurricanes, tornadoes and other extreme weather events occur with increased frequency and severity, it’s more important than ever to improve and accelerate climate research and prediction using the latest technologies.

Amid peaks in the current Atlantic hurricane season, NVIDIA Research today announced a new generative AI model, dubbed StormCast, for emulating high-fidelity atmospheric dynamics. This means the model can enable reliable weather prediction at mesoscale — a scale larger than storms but smaller than cyclones — which is critical for disaster planning and mitigation.

Detailed in a paper written in collaboration with the Lawrence Berkeley National Laboratory and the University of Washington, StormCast arrives as extreme weather phenomena are taking lives, destroying homes and causing more than $150 billion in damage annually in the U.S. alone.

It’s just one example of how generative AI is supercharging thundering breakthroughs in climate research and actionable extreme weather prediction, helping scientists tackle challenges of the highest stakes: saving lives and the world.

NVIDIA Earth-2 — a digital twin cloud platform that combines the power of AI, physical simulations and computer graphics — enables simulation and visualization of weather and climate predictions at a global scale with unprecedented accuracy and speed.

In Taiwan, for example, the National Science and Technology Center for Disaster Reduction plans to predict fine-scale details of typhoons using CorrDiff, an NVIDIA generative AI model offered as part of Earth-2.

CorrDiff can super-resolve 25-kilometer-scale atmospheric data by 12.5x down to 2 kilometers — 1,000x faster and using 3,000x less energy for a single inference than traditional methods.

That means the center’s potentially lifesaving work, which previously cost nearly $3 million on CPUs, can be accomplished using about $60,000 on a single system with an NVIDIA H100 Tensor Core GPU. It’s a massive reduction that shows how generative AI and accelerated computing increase energy efficiency and lower costs.

The center also plans to use CorrDiff to predict downwash — when strong winds funnel down to street level, damaging buildings and affecting pedestrians — in urban areas.

Now, StormCast adds hourly autoregressive prediction capabilities to CorrDiff, meaning it can predict future outcomes based on past ones.

A Global Impact From a Regional Focus

Global climate research begins at a regional level.

Physical hazards of weather and climate change can vary dramatically on regional scales. But reliable numerical weather prediction at this level comes with substantial computational costs. This is due to the high spatial resolution needed to represent the underlying fluid-dynamic motions at mesoscale.

Regional weather prediction models — often referred to as convection-allowing models, or CAMs — have traditionally forced researchers to face varying tradeoffs in resolution, ensemble size and affordability.

CAMs are useful to meteorologists for tracking the evolution and structure of storms, as well as for monitoring its convective mode, or how a storm is organized when it forms. For example, the likelihood of a tornado is based on a storm’s structure and convective mode.

CAMs also help researchers understand the implications for weather-related physical hazards at the infrastructure level.

For example, global climate model simulations can be used to inform CAMs, helping them translate slow changes in the moisture content of large atmospheric rivers into flash-flooding projections in vulnerable coastal areas.

At lower resolutions, machine learning models trained on global data have emerged as useful emulators of numerical weather prediction models that can be used to improve early-warning systems for severe events. These machine learning models typically have a spatial resolution of about 30 kilometers and a temporal resolution of six hours.

Now, with the help of generative diffusion, StormCast enables this at a 3-kilometer, hourly scale.

Despite being in its infancy, the model — when applied with precipitation radars — already offers forecasts with lead times of up to six hours that are up to 10% more accurate than the U.S. National Oceanic and Atmospheric Administration (NOAA)’s state-of-the-art 3-kilometer operational CAM.

Plus, outputs from StormCast exhibit physically realistic heat and moisture dynamics, and can predict over 100 variables, such as temperature, moisture concentration, wind and rainfall radar reflectivity values at multiple, finely spaced altitudes. This enables scientists to confirm the realistic 3D evolution of a storm’s buoyancy — a first-of-its-kind accomplishment in AI weather simulation.

NVIDIA researchers trained StormCast on approximately three-and-a-half years of NOAA climate data from the central U.S., using NVIDIA accelerated computing to speed calculations.

More Innovations Brewing

Scientists are already looking to harness the model’s benefits.

“Given both the outsized impacts of organized thunderstorms and winter precipitation, and the major challenges in forecasting them with confidence, the production of computationally tractable storm-scale ensemble weather forecasts represents one of the grand challenges of numerical weather prediction,” said Tom Hamill, head of innovation at The Weather Company. “StormCast is a notable model that addresses these challenges, and The Weather Company is excited to collaborate with NVIDIA on developing, evaluating and potentially using these deep learning forecast models.”

“Developing high-resolution weather models requires AI algorithms to resolve convection, which is a huge challenge,” said Imme Ebert-Uphoff, machine learning lead at Colorado State University’s Cooperative Institute for Research in the Atmosphere. “The new NVIDIA research explores the potential of accomplishing this with diffusion models like StormCast, which presents a significant step toward the development of future AI models for high-resolution weather prediction.”

Alongside the acceleration and visualization of physically accurate climate simulations, as well as a digital twin of our planet, such research breakthroughs signify how NVIDIA Earth-2 is enabling a new, vital era of climate research.

Learn more about sustainable computing and NVIDIA Research, a global team of hundreds of scientists and engineers focused on topics including climate AI, computer graphics, computer vision, self-driving cars and robotics.

Featured image courtesy of NASA.

See notice regarding software product information.

Editor’s note: This post is part of our In the NVIDIA Studio series, which celebrates featured artists, offers creative tips and tricks, and demonstrates how NVIDIA Studio technology improves creative workflows. We’re also deep diving on new GeForce RTX GPU features, technologies and resources, and how they dramatically accelerate content creation.

Every month brings new creative app updates and optimizations powered by the NVIDIA Studio platform — supercharging creative processes with NVIDIA RTX and AI.

RTX-powered video editing app CyberLink PowerDirector now has a setting for high-efficiency video encoding (HEVC). 3D artists can access new features and faster workflows in Adobe Substance 3D Modeler and SideFX: Houdini. And content creators using Topaz Video AI Pro can now scale their photo and video touchups faster with NVIDIA TensorRT acceleration.

The August Studio Driver is ready to install via the NVIDIA app beta — the essential companion for creators and gamers — to keep GeForce RTX PCs up to date with the latest NVIDIA drivers and technology.

And this week’s featured In the NVIDIA Studio artist Stavros Liaskos is creating physically accurate 3D digital replicas of Greek Orthodox churches, holy temples, monasteries and other buildings using the NVIDIA Omniverse platform for building and connecting Universal Scene Description (OpenUSD) apps.

Discover the latest breakthroughs in graphics and generative AI by watching the replay of NVIDIA founder and CEO Jensen Huang’s firechat chats with Lauren Goode, senior writer at WIRED, and Meta founder and CEO Mark Zuckerberg at SIGGRAPH. 

There’s a Creative App for That

The NVIDIA NVENC video encoder is built into every RTX graphics card, offloading the compute-intensive task of video encoding from the CPU to a dedicated part of the GPU.

CyberLink PowerDirector, a popular video editing program that recently added support for RTX Video HDR, now has a setting to increase HEVC with NVIDIA NVENC HEVC Ultra-High-Quality mode.

The new functionality reduces bit rates and improves encoding efficiency by 10%, significantly boosting video quality. Using the custom setting, content creators can offer audiences superior viewing experiences.

Alpha exporting allows users to add overlay effects to videos by exporting HEVC video with an alpha channel. This technique can be used to create transparent backgrounds and rapidly process animated overlays, making it ideal for creating social media content.

With an alpha channel, users can export HEVC videos up to 8x faster compared with run-length encoding supported by other processors, and with a 100x reduction in file size.

Adobe Substance 3D Modeler, a multisurface 3D sculpting tool for artists, virtual effects specialists and designers, released Block to Stock, an AI-powered, geometry-based feature for accelerating the prototyping of complex shapes.

It allows rough 3D shapes to be quickly replaced with pre-existing, similarly shaped 3D models that have greater detail. The result is a highly detailed shape crafted in no time.

The recently released version 20.5 of SideFX: Houdini, a 3D procedural software for modeling, animation and lighting, introduced NVIDIA OptiX 8 and NVIDIA’s Shader Execution Reordering feature to its Karma XPU renderer — exclusively on NVIDIA RTX GPUs.

With these additions, computationally intensive tasks can now be executed up to 4x faster on RTX GPUs.

Topaz Video AI Pro, a photo and video enhancement software for noise reduction, sharpening and upscaling, added TensorRT acceleration for multi-GPU configurations, enabling parallelization across multiple GPUs for supercharged rendering speeds — up to 2x faster with two GPUs over a single GPU system, with further acceleration in systems with additional GPUs.

Virtual Cultural Sites to G(r)eek Out About

Anyone can now explore over 30 Greek cultural sites in virtual reality, thanks to the immersive work of Stavros Liaskos, managing director of visual communications company Reyelise.

“Many historical and religious sites are at risk due to environmental conditions, neglect and socio-political issues,” he said. “By creating detailed 3D replicas, we’re helping to ensure their architectural splendor is preserved digitally for future generations.”

Liaskos dedicated the project to his father, who passed away last year.

“He taught me the value of patience and instilled in me the belief that nothing is unattainable,” he said. “His wisdom and guidance continue to inspire me every day.”

Churches are architecturally complex structures. To create physically accurate 3D models of them, Liaskos used the advanced real-time rendering capabilities of Omniverse, connected with a slew of content-creation apps.

The OpenUSD framework enabled a seamless workflow across the various apps Liaskos used. For example, after using Trimble X7 for highly accurate 3D scanning of structures, Liaskos easily moved to Autodesk 3ds Max and Blender for modeling and animation.

Then, with ZBrush, he sculpted intricate architectural details on the models and refined textures with Adobe Photoshop and Substance 3D. It was all brought together in Omniverse for real-time lighting and rendering.

For post-production work, like adding visual effects and compiling rendered scenes, Liaskos used OpenUSD to transfer his projects to Adobe After Effects, where he finalized the video output. Nearly every element of his creative workflow was accelerated by his NVIDIA RTX A4500 GPU. 

Liaskos also explored developing extended reality (XR) applications that allow users to navigate his 3D projects in real time in virtual reality (VR).

First, he used laser scanning and photogrammetry to capture the detailed geometries and textures of the churches.

Then, he tapped Autodesk 3ds Max and Maxon ZBrush for retopology, ensuring the models were optimized for real-time rendering without compromising detail.

After importing them into NVIDIA Omniverse with OpenUSD, Liaskos packaged the XR scenes so they could be streamed to VR headsets  using either the NVIDIA Omniverse Create XR spatial computing app or Unity Engine, enabling immersive viewing experiences.

“This approach will even more strikingly showcase the architectural beauty and cultural significance of these sites,” Liaskos said. “The simulation must be as good as possible to recreate the overwhelming, impactful feeling of calm and safety that comes with visiting a deeply spiritual space.”

The project is co-funded by the European Union within the framework of the operational program Digital Transformation 2021-2027 for the Greek Holy Archbishopric of Athens.

Follow NVIDIA Studio on Instagram, X and Facebook. Access tutorials on the Studio YouTube channel and get updates directly in your inbox by subscribing to the Studio newsletter. 

Stay up to date on NVIDIA Omniverse with Instagram, Medium and X. For more, join the Omniverse community and check out the Omniverse forums, Discord server, Twitch and YouTube channels. 

NVIDIA researchers used NVIDIA Edify, a multimodal architecture for visual generative AI, to build a detailed 3D desert landscape within a few minutes in a live demo at SIGGRAPH’s Real-Time Live event on Tuesday.

During the event — one of the prestigious graphics conference’s top sessions — NVIDIA researchers showed how, with the support of an AI agent, they could build and edit a desert landscape from scratch within five minutes. The live demo highlighted how generative AI can act as an assistant to artists by accelerating ideation and generating custom secondary assets that would otherwise have been sourced from a repository.

By drastically decreasing ideation time, these AI technologies will empower 3D artists to be more productive and creative — giving them the tools to explore concepts faster and expedite parts of their workflows. They could, for example, generate the background assets or 360 HDRi environments that the scene needs in minutes, instead of spending hours finding or creating them.

From Idea to 3D Scene in Three Minutes

Creating a full 3D scene is a complex, time-consuming task. Artists must support their hero asset with plenty of background objects to create a rich scene, then find an appropriate background and an environment map to light it. Due to time constraints, they’ve often had to make a trade-off between rapid results and creative exploration.

With the support of AI agents, creative teams can achieve both goals: quickly bring concepts to life and continue iterating to achieve the right look.

In the Real-Time Live demo, the researchers used an AI agent to instruct an NVIDIA Edify-powered model to generate dozens of 3D assets, including cacti, rocks and the skull of a bull — with previews produced in just seconds.

They next directed the agent to harness other models to create potential backgrounds and a layout of how the objects would be placed in the scene — and showcased how the agent could adapt to last-minute changes in creative direction by quickly swapping the rocks for gold nuggets.

With a design plan in place, they prompted the agent to create full-quality assets and render the scene as a photorealistic image in NVIDIA Omniverse USD Composer, an app for virtual world-building.

NVIDIA Edify Accelerates Environment Generation 

NVIDIA Edify models can help creators focus on hero assets while accelerating the creation of background environments and objects using AI-powered scene generation tools. The Real-Time Live demo showcased two Edify models: 

• Edify 3D generates ready-to-edit 3D meshes from text or image prompts. Within seconds, the model can generate previews, including rotating animations of each object, to help creators rapidly prototype before committing to a specific design.
• Edify 360 HDRi uses text or image prompts to generate up to 16K high-dynamic range images (HDRi) of nature landscapes, which can be used as backgrounds and to light scenes.

During the demo, the researchers also showcased an AI agent powered by a large language model, and USD Layout, an AI model that generates scene layouts using OpenUSD, a platform for 3D workflows.

At SIGGRAPH, NVIDIA also announced that two leading creative content companies are giving designers and artists new ways to boost productivity with generative AI using tools powered by NVIDIA Edify.

Shutterstock has launched in commercial beta its Generative 3D service, which lets creators quickly prototype and generate 3D assets using text or image prompts. Its 360 HDRi generator based on Edify also entered early access.

Getty Images updated its Generative AI by Getty Images service with the latest version of NVIDIA Edify. Users can now create images twice as fast, with improved output quality and prompt adherence, and advanced controls and fine-tuning.

Harnessing Universal Scene Description in NVIDIA Omniverse

The 3D objects, environment maps and layouts generated using Edify models are structured with USD, a standard format for describing and composing 3D worlds. This compatibility allows artists to immediately import Edify-powered creations into Omniverse USD Composer.

Within Composer, they can use popular digital content creation tools to further modify the scene by, for example, changing the position of objects, modifying their appearance or adjusting lighting.

Real-Time Live is one of the most anticipated events at SIGGRAPH, featuring about a dozen real-time applications including generative AI, virtual reality and live performance capture technology. Watch the replay below.

Designers and artists have new and improved ways to boost their productivity with generative AI trained on licensed data.

Shutterstock, a leading platform for creative content, launched its Generative 3D service in commercial beta. It lets creators quickly prototype 3D assets and generate 360 HDRi backgrounds that light scenes, using just text or image prompts.

Getty Images, a premier visual content creator and marketplace, turbocharged its Generative AI by Getty Images service so it creates images twice as fast, improves output quality, brings advanced controls and enables fine-tuning.

The services are built with NVIDIA’s visual AI foundry using NVIDIA Edify, a multimodal generative AI architecture. The AI models are then optimized and packaged for maximum performance with NVIDIA NIM, a set of accelerated microservices for AI inference.

Edify enables service providers to train responsible generative models on their licensed data and scale them quickly with NVIDIA DGX Cloud, the cloud-first way to get the best of NVIDIA AI.

Generative AI Speeds 3D Modeling

Available now for enterprises in commercial beta, Shutterstock’s service lets designers and artists quickly create 3D objects that help them prototype or populate virtual environments. For example, tapping generative AI, they can quickly create the silverware and plates on a dining room table so they can focus on designing the characters around it.

The 3D assets the service generates are ready to edit using digital content creation tools, and available in a variety of popular file formats. Their clean geometry and layout gives artists an advanced starting point for adding their own flair.

The AI model first delivers a preview of a single asset in as little as 10 seconds. If users like it, the preview can be turned into a higher-quality 3D asset, complete with physically based rendering materials like concrete, wood or leather.

At this year’s SIGGRAPH computer graphics conference, designers will see just how fast they can make their ideas come to life.

Shutterstock will demo a workflow in Blender that lets artists generate objects directly within their 3D environment. In the Shutterstock booth at SIGGRAPH, HP will show 3D prints and physical prototypes of the kinds of assets attendees can design on the show floor using Generative 3D.

Shutterstock is also working with global marketing and communications services company WPP to bring ideas to life with Edify 3D generation for virtual production (see video below).

Explore Generative 3D by Shutterstock on the company’s website, or test-drive the application programming interface (API) at build.nvidia.com.

Virtual Lighting Gets Real

Lighting a virtual scene with accurate reflections can be a complicated task. Creatives need to operate expensive 360-degree camera rigs and go on set to create backgrounds from scratch, or search vast libraries for something that approximates what they want.

With Shutterstock’s Generative 3D service, users can now simply describe the exact environment they need in text or with an image, and out comes a high-dynamic-range panoramic image, aka 360 HDRi, in brilliant 16K resolution. (See video below.)

Want that beautiful new sports car shown in a desert, a tropical beach or maybe on a winding mountain road? With generative AI, designers can shift gears fast.

Three companies plan to integrate Shutterstock’s 360 HDRi APIs directly into their workflows — WPP, CGI studio Katana and Dassault Systèmes, developer of the 3DEXCITE applications for creating high-end visualizations and 3D content for virtual worlds.

Great Images Get a Custom Fit

Generative AI by Getty Images has upgraded to a more powerful Edify AI model with a portfolio of new features that let artists control image composition and style.

Want a red beach ball floating above that perfect shot of a coral reef in Fiji? Getty Images’ service can get it done in a snap.

The new model is twice as fast, boosts image quality and prompt accuracy, and lets users control camera settings like the depth of field or focal length of a shot. Users can generate four images in about six seconds and scale them up to 4K resolution.

In addition, the commercially safe foundational model now serves as the basis for a fine-tuning capability that lets companies customize the AI with their own data. That lets them generate images tailored to the creative style of their specific brands.

New controls in the service support the use of a sketch or depth map to guide the composition or structure of an image.

Creatives at Omnicom, a global leader in marketing and sales solutions, are using Getty Images’ service to streamline advertising workflows and safely create on-brand content. The collaboration with Getty Images is part of Omnicom’s strategy to infuse generative AI into every facet of its business, helping teams move from ideas to outcomes faster.

Generative AI by Getty Images is available through the Getty Images and iStock websites, and via an API.

For more about NVIDIA’s offerings, read about the AI foundry for visual generative AI built on NVIDIA DGX Cloud, and try it on ai.nvidia.com.

To get the big picture, listen to NVIDIA founder and CEO Jensen Huang in two fireside chats at SIGGRAPH.

See notice regarding software product information.

Editor’s note: This post is part of the AI Decoded series, which demystifies AI by making the technology more accessible, and showcases new hardware, software, tools and accelerations for RTX PC users.

Adobe Creative Cloud applications, which tap NVIDIA RTX GPUs, are designed to enhance the creativity of users, empowering them to work faster and focus on their craft.

These tools seamlessly integrate into existing creator workflows, enabling greater productivity and delivering power and precision.

Look to the Light

Generative AI creates new data in forms such as images or text by learning from existing data. It effectively visualizes and generates content to match what a user describes and helps open up fresh avenues for creativity.

Adobe Firefly is Adobe’s family of creative generative AI models that offer new ways to ideate and create while assisting creative workflows using generative AI. They’re designed to be safe for commercial use and were trained, using NVIDIA GPUs, on licensed content, like Adobe Stock Images, and public domain content where copyright has expired.

Firefly features are integrated in Adobe’s most popular creative apps.

Adobe Photoshop features the Generative Fill tool, which uses simple description prompts to easily add content from images. With the latest Reference Image feature currently in beta, users can also upload a sample image to get image results closer to their desired output.

Generative Expand allows artists to extend the border of their image with the Crop tool, filling in bigger canvases with new content that automatically blends in with the existing image.

RTX-accelerated Neural Filters, such as Photo Restoration, enable complex adjustments such as colorizing black-and-white photos and performing style transfers using AI. The Smart Portrait filter, which allows non-destructive editing with filters, is based on work from NVIDIA Research.

The brand-new Generative Shape Fill (beta) in Adobe Illustrator, powered by the latest Adobe Firefly Vector Model, allows users to accelerate design workflows by quickly filling shapes with detail and color in their own styles. With Generative Shape Fill, designers can easily match the style and color of their own artwork to create a wide variety of editable and scalable vector graphic options.

Adobe Illustrator’s Generative Recolor feature lets creators type in a text prompt to explore custom color palettes and themes for their vector artwork in seconds.

NVIDIA will continue working with Adobe to support advanced generative AI models, with a focus on deep integration into the apps the world’s leading creators use.

Making Moves on Video

Adobe Premiere Pro is one of the most popular and powerful video editing solutions.

Its Enhance Speech tool, accelerated by RTX, uses AI to remove unwanted noise and improve the quality of dialogue clips so they sound professionally recorded. It’s up to 4.5x faster on RTX PCs.

Auto Reframe, another Adobe Premiere feature, uses GPU acceleration to identify and track the most relevant elements in a video, and intelligently reframes video content for different aspect ratios. Scene Edit Detection automatically finds the original edit points in a video, a necessary step before the video editing stage begins.

Visual Effects

Separating a foreground object from a background is a crucial step in many visual effects and compositing workflows.

Adobe After Effects has a new feature that uses a matte to isolate an object, enabling capabilities including background replacement and the selective application of effects to the foreground.

Using the Roto Brush tool, artists can draw strokes on representative areas of the foreground and background elements. After Effects uses that information to create a segmentation boundary between the foreground and background elements, delivering cleaner cutouts with fewer clicks.

Creating 3D Product Shots

The Substance 3D Collection is Adobe’s solution for 3D material authoring, texturing and rendering, enabling users to rapidly create stunningly photorealistic 3D content, including models, materials and lighting.

Visualizing products and designs in the context of a space is compelling, but it can be time-consuming to find the right environment for the objects to live in. Substance 3D Stager’s Generative Background feature, powered by Adobe Firefly, solves this issue by letting artists quickly explore generated backgrounds to composite 3D models.

Once an environment is selected, Stager can automatically match the perspective and lighting to the generated background.

Material Authoring With AI

Adobe Substance 3D Sampler, also part of the Substance 3D Collection, is designed to transform images of surfaces and objects into photorealistic physically based rendering (PBR) materials, 3D models and high-dynamic range environment lights. With the recent introduction of new generative workflows powered by Adobe Firefly, Sampler is making it easier than ever for artists to explore variations when creating materials for everything from product visualization projects to the latest AAA games.

Sampler’s Text-to-Texture feature allows users to generate tiled images from detailed text prompts. These generated images can then be edited and transformed into photorealistic PBR materials using the machine learning-powered Image-to-Material feature or any Sampler filter.

Image-to-Texture similarly enables the creation of tiled textures from reference images, providing an alternate way to prompt and generate variations from existing visual content.

Sampler’s Text-to-Pattern feature uses text prompts to generate tiling patterns, which can be used as base colors or inputs for various filters, such as the Cloth Weave filter for creating original fabric materials.

All of these generative AI features in the Substance 3D Collection, supercharged with RTX GPUs, are designed to help 3D creators ideate and create faster.

Photo-tastic Features

Adobe Lightroom’s AI-powered Raw Details feature produces crisp detail and more accurate renditions of edges, improves color rendering and reduces artifacts, enhancing the image without changing its original resolution. This feature is handy for large displays and prints, where fine details are visible.

Super Resolution helps create an enhanced image with similar results as Raw Details but with 2x the linear resolution. This means that the enhanced image will have 2x the width and height of the original image — or 4x the total pixel count. This is especially useful for increasing the resolution of cropped imagery.

For faster editing, AI-powered, RTX-accelerated masking tools like Select Subject, which isolates people from an image, and Select Sky, which captures skies, enable users to create complex masks with the click of a button.

Visit Adobe’s AI features page for a complete list of AI features using RTX.

Looking for more AI-powered content creation apps? Consider NVIDIA Broadcast, which transforms any room into a home studio, free for RTX GPU owners. 

Generative AI is transforming gaming, videoconferencing and interactive experiences of all kinds. Make sense of what’s new and what’s next by subscribing to the AI Decoded newsletter.

Editor’s note: This post is part of our In the NVIDIA Studio series, which celebrates featured artists, offers creative tips and tricks, and demonstrates how NVIDIA Studio technology improves creative workflows. We’re also deep diving on new GeForce RTX GPU features, technologies and resources, and how they dramatically accelerate content creation.

Wondershare Filmora — a video editing app with AI-powered tools — now supports NVIDIA RTX Video HDR, joining editing software like Blackmagic Design’s DaVinci Resolve and Cyberlink PowerDirector.

RTX Video HDR significantly enhances video quality, ensuring the final output is suitable for the best monitors available today.

Livestreaming software OBS Studio and XSplit Broadcaster now support Twitch Enhanced Broadcasting, giving streamers more control over video quality through client-side encoding and automatic configurations. The feature, developed in collaboration between Twitch, OBS and NVIDIA, also paves the way for more advancements, including vertical live video and advanced codecs such as HEVC and AV1.

A summer’s worth of creative app updates are included in the July Studio Driver, ready for download today. Install the NVIDIA app beta — the essential companion for creators and gamers — to keep GeForce RTX PCs up to date with the latest NVIDIA drivers and technology.

Join NVIDIA at SIGGRAPH to learn about the latest breakthroughs in graphics and generative AI, and tune in to a fireside chat featuring NVIDIA founder and CEO Jensen Huang and Lauren Goode, senior writer at WIRED, on Monday, July 29 at 2:30 p.m. MT. Register now.

And this week’s featured In the NVIDIA Studio artist, Kevin Stratvert, shares all about AI-powered content creation in Wondershare Filmora.

(Wonder)share the Beauty of RTX Video

RTX Video HDR analyzes standard dynamic range video and transforms it into HDR10-quality video, expanding the color gamut to produce clearer, more vibrant frames and enhancing the sense of depth for greater immersion.

With RTX Video HDR, Filmora users can create high-quality content that’s ideal for gaming videos, travel vlogs or event filmmaking.

Combining RTX Video HDR with RTX Video Super Resolution — another AI-powered tool that uses trained models to sharpen edges, restore features and remove artifacts in video — further enhances visual quality. RTX Video HDR requires an NVIDIA RTX GPU connected to an HDR10-compatible monitor or TV. For more information, check out the RTX Video FAQ.

Those with a RTX GPU-powered PC can send files to the Filmora desktop app and continue to edit with local RTX acceleration, doubling the speed of the export process with dual encoders on GeForce RTX 4070 Ti or above GPUs.

Learn more about Wondershare Filmora’s AI-powered features.

Maximizing AI Features in Filmora

Kevin Stratvert has the heart of a teacher — he’s always loved to share his technical knowledge and tips with others.

One day, he thought, “Why not make a YouTube video to explain stuff directly to users?” His first big hit was a tutorial on how to get Microsoft Office for free through Office.com. The video garnered millions of views and tons of engagement — and he’s continued creating content ever since.

“The more content I created, the more questions and feedback I got from viewers, sparking this cycle of creativity and connection that I just couldn’t get enough of,” said Stratvert.

Explaining the benefits of AI has been an area of particular interest for Stratvert, especially as it relates to AI-powered features in Wondershare Filmora. In one YouTube video, Filmora Video Editor Tutorial for Beginners, he breaks down the AI effects video editors can use to accelerate their workflows.

Examples include:

• Smart Edit: Edit footage-based transcripts generated automatically, including in multiple languages.
• Smart Cutout: Remove unwanted objects or change the background in seconds.
• Speech-to-Text: Automatically generate compelling descriptions, titles and captions.

“AI has become a crucial part of my creative toolkit, especially for refining details that really make a difference,” said Stratvert. “By handling these technical tasks, AI frees up my time to focus more on creating content, making the whole process smoother and more efficient.”

Stratvert has also been experimenting with NVIDIA ChatRTX, a technology that lets users interact with their local data, installing and configuring various AI models, effectively prompting AI for both text and image outputs using CLIP and more.

NVIDIA Broadcast has been instrumental in giving Stratvert a professional setup for web conferences and livestreams. The app’s features, including background noise removal and virtual background, help maintain a professional appearance on screen. It’s especially useful in home studio settings, where controlling variables in the environment can be challenging.

Stratvert stresses the importance of his GeForce RTX 4070 graphics card in the content creation process.

“With an RTX GPU, I’ve noticed a dramatic improvement in render times and the smoothness of playback, even in demanding scenarios,” he said. “Additionally, the advanced capabilities of RTX GPUs support more intensive tasks like real-time ray tracing and AI-driven editing features, which can open up new creative possibilities in my edits.”

Check out Stratvert’s video tutorials on his website.

Follow NVIDIA Studio on Instagram, X and Facebook. Access tutorials on the Studio YouTube channel and get updates directly in your inbox by subscribing to the Studio newsletter. 

NVIDIA is taking an array of advancements in rendering, simulation and generative AI to SIGGRAPH 2024, the premier computer graphics conference, which will take place July 28 – Aug. 1 in Denver.

More than 20 papers from NVIDIA Research introduce innovations advancing synthetic data generators and inverse rendering tools that can help train next-generation models. NVIDIA’s AI research is making simulation better by boosting image quality and unlocking new ways to create 3D representations of real or imagined worlds.

The papers focus on diffusion models for visual generative AI, physics-based simulation and increasingly realistic AI-powered rendering. They include two technical Best Paper Award winners and collaborations with universities across the U.S., Canada, China, Israel and Japan as well as researchers at companies including Adobe and Roblox.

These initiatives will help create tools that developers and businesses can use to generate complex virtual objects, characters and environments. Synthetic data generation can then be harnessed to tell powerful visual stories, aid scientists’ understanding of natural phenomena or assist in simulation-based training of robots and autonomous vehicles.

Diffusion Models Improve Texture Painting, Text-to-Image Generation

Diffusion models, a popular tool for transforming text prompts into images, can help artists, designers and other creators rapidly generate visuals for storyboards or production, reducing the time it takes to bring ideas to life.

Two NVIDIA-authored papers are advancing the capabilities of these generative AI models.

ConsiStory, a collaboration between researchers at NVIDIA and Tel Aviv University, makes it easier to generate multiple images with a consistent main character — an essential capability for storytelling use cases such as illustrating a comic strip or developing a storyboard. The researchers’ approach introduces a technique called subject-driven shared attention, which reduces the time it takes to generate consistent imagery from 13 minutes to around 30 seconds.

NVIDIA researchers last year won the Best in Show award at SIGGRAPH’s Real-Time Live event for AI models that turn text or image prompts into custom textured materials. This year, they’re presenting a paper that applies 2D generative diffusion models to interactive texture painting on 3D meshes, enabling artists to paint in real time with complex textures based on any reference image.

Kick-Starting Developments in Physics-Based Simulation

Graphics researchers are narrowing the gap between physical objects and their virtual representations with physics-based simulation — a range of techniques to make digital objects and characters move the same way they would in the real world.

Several NVIDIA Research papers feature breakthroughs in the field, including SuperPADL, a project that tackles the challenge of simulating complex human motions based on text prompts (see video at top).

Using a combination of reinforcement learning and supervised learning, the researchers demonstrated how the SuperPADL framework can be trained to reproduce the motion of more than 5,000 skills — and can run in real time on a consumer-grade NVIDIA GPU.

Another NVIDIA paper features a neural physics method that applies AI to learn how objects — whether represented as a 3D mesh, a NeRF or a solid object generated by a text-to-3D model — would behave as they are moved in an environment.

A paper written in collaboration with Carnegie Mellon University researchers develops a new kind of renderer — one that, instead of modeling physical light, can perform thermal analysis, electrostatics and fluid mechanics. Named one of five best papers at SIGGRAPH, the method is easy to parallelize and doesn’t require cumbersome model cleanup, offering new opportunities for speeding up engineering design cycles.

In the example above, the renderer performs a thermal analysis of the Mars Curiosity rover, where keeping temperatures within a specific range is critical to mission success. 

Additional simulation papers introduce a more efficient technique for modeling hair strands and a pipeline that accelerates fluid simulation by 10x.

Raising the Bar for Rendering Realism, Diffraction Simulation

Another set of NVIDIA-authored papers present new techniques to model visible light up to 25x faster and simulate diffraction effects — such as those used in radar simulation for training self-driving cars — up to 1,000x faster.

A paper by NVIDIA and University of Waterloo researchers tackles free-space diffraction, an optical phenomenon where light spreads out or bends around the edges of objects. The team’s method can integrate with path-tracing workflows to increase the efficiency of simulating diffraction in complex scenes, offering up to 1,000x acceleration. Beyond rendering visible light, the model could also be used to simulate the longer wavelengths of radar, sound or radio waves.

Path tracing samples numerous paths — multi-bounce light rays traveling through a scene — to create a photorealistic picture. Two SIGGRAPH papers improve sampling quality for ReSTIR, a path-tracing algorithm first introduced by NVIDIA and Dartmouth College researchers at SIGGRAPH 2020 that has been key to bringing path tracing to games and other real-time rendering products.

One of these papers, a collaboration with the University of Utah, shares a new way to reuse calculated paths that increases effective sample count by up to 25x, significantly boosting image quality. The other improves sample quality by randomly mutating a subset of the light’s path. This helps denoising algorithms perform better, producing fewer visual artifacts in the final render.

Teaching AI to Think in 3D

NVIDIA researchers are also showcasing multipurpose AI tools for 3D representations and design at SIGGRAPH.

One paper introduces fVDB, a GPU-optimized framework for 3D deep learning that matches the scale of the real world. The fVDB framework provides AI infrastructure for the large spatial scale and high resolution of city-scale 3D models and NeRFs, and segmentation and reconstruction of large-scale point clouds.

A Best Technical Paper award winner written in collaboration with Dartmouth College researchers introduces a theory for representing how 3D objects interact with light. The theory unifies a diverse spectrum of appearances into a single model.

And a collaboration with University of Tokyo, University of Toronto and Adobe Research introduces an algorithm that generates smooth, space-filling curves on 3D meshes in real time. While previous methods took hours, this framework runs in seconds and offers users a high degree of control over the output to enable interactive design.

NVIDIA at SIGGRAPH

Learn more about NVIDIA at SIGGRAPH. Special events include a fireside chat between NVIDIA founder and CEO Jensen Huang and Meta founder and CEO Mark Zuckerberg, as well as a fireside chat with Huang and Lauren Goode, senior writer at WIRED, on the impact of robotics and AI in industrial digitalization. 

NVIDIA researchers will also present OpenUSD Day by NVIDIA, a full-day event showcasing how developers and industry leaders are adopting and evolving OpenUSD to build AI-enabled 3D pipelines.

NVIDIA Research has hundreds of scientists and engineers worldwide, with teams focused on topics including AI, computer graphics, computer vision, self-driving cars and robotics. See more of their latest work.

Multi-die chips, known as three-dimensional integrated circuits, or 3D-ICs, represent a revolutionary step in semiconductor design. The chips are vertically stacked to create a compact structure that boosts performance without increasing power consumption.

However, as chips become denser, they present more complex challenges in managing electromagnetic and thermal stresses. To understand and address this, advanced 3D multiphysics visualizations become essential to design and diagnostic processes.

At this week’s Design Automation Conference, a global event showcasing the latest developments in chips and systems, Ansys — a company that develops engineering simulation and 3D design software — will share how it’s using NVIDIA technology to overcome these challenges to build the next generation of semiconductor systems.

To enable 3D visualizations of simulation results for their users, Ansys uses NVIDIA Omniverse, a platform of application programming interfaces, software development kits, and services that enables developers to easily integrate Universal Scene Description (OpenUSD) and NVIDIA RTX rendering technologies into existing software tools and simulation workflows.

The platform powers visualizations of 3D-IC results from Ansys solvers so engineers can evaluate phenomena like electromagnetic fields and temperature variations to optimize chips for faster processing, increased functionality and improved reliability.

With Ansys Icepak on the NVIDIA Omniverse platform, engineers can simulate temperatures across a chip according to different power profiles and floor plans. Finding chip hot-spots can lead to better design of the chips themselves, as well as auxiliary cooling devices. However, these 3D-IC simulations are computationally intensive, limiting the number of simulations and design points users can explore.

Using NVIDIA Modulus, combined with novel techniques for handling arbitrary power patterns in the Ansys RedHawk-SC electrothermal data pipeline and model training framework, the Ansys R&D team is exploring the acceleration of simulation workflows with AI-based surrogate models. Modulus is an open-source AI framework for building, training and fine-tuning physics-ML models at scale with a simple Python interface.

With the NVIDIA Modulus Fourier neural operator (FNO) architecture, which can parameterize solutions for a distribution of partial differential equations, Ansys researchers created an AI surrogate model that efficiently predicts temperature profiles for any given power profile and a given floor plan defined by system parameters like heat transfer coefficient, thickness and material properties. This model offers near real-time results at significantly reduced computational costs, allowing Ansys users to explore a wider design space for new chips.

Following a successful proof of concept, the Ansys team will explore integration of such AI surrogate models for its next-generation RedHawk-SC platform using NVIDIA Modulus.

As more surrogate models are developed, the team will also look to enhance model generality and accuracy through in-situ fine-tuning. This will enable RedHawk-SC users to benefit from faster simulation workflows, access to a broader design space and the ability to refine models with their own data to foster innovation and safety in product development.

To see the joint demonstration of 3D-IC multiphysics visualization using NVIDIA Omniverse APIs, visit Ansys at the Design Automation Conference, running June 23-27, in San Francisco at booth 1308 or watch the presentation at the Exhibitor Forum.

NVIDIA researchers are at the forefront of the rapidly advancing field of visual generative AI, developing new techniques to create and interpret images, videos and 3D environments.

More than 50 of these projects will be showcased at the Computer Vision and Pattern Recognition (CVPR) conference, taking place June 17-21 in Seattle. Two of the papers — one on the training dynamics of diffusion models and another on high-definition maps for autonomous vehicles — are finalists for CVPR’s Best Paper Awards.

NVIDIA is also the winner of the CVPR Autonomous Grand Challenge’s End-to-End Driving at Scale track — a significant milestone that demonstrates the company’s use of generative AI for comprehensive self-driving models. The winning submission, which outperformed more than 450 entries worldwide, also received CVPR’s Innovation Award.

NVIDIA’s research at CVPR includes a text-to-image model that can be easily customized to depict a specific object or character, a new model for object pose estimation, a technique to edit neural radiance fields (NeRFs) and a visual language model that can understand memes. Additional papers introduce domain-specific innovations for industries including automotive, healthcare and robotics.

Collectively, the work introduces powerful AI models that could enable creators to more quickly bring their artistic visions to life, accelerate the training of autonomous robots for manufacturing, and support healthcare professionals by helping process radiology reports.

“Artificial intelligence, and generative AI in particular, represents a pivotal technological advancement,” said Jan Kautz, vice president of learning and perception research at NVIDIA. “At CVPR, NVIDIA Research is sharing how we’re pushing the boundaries of what’s possible — from powerful image generation models that could supercharge professional creators to autonomous driving software that could help enable next-generation self-driving cars.”

At CVPR, NVIDIA also announced NVIDIA Omniverse Cloud Sensor RTX, a set of microservices that enable physically accurate sensor simulation to accelerate the development of fully autonomous machines of every kind.

Forget Fine-Tuning: JeDi Simplifies Custom Image Generation

Creators harnessing diffusion models, the most popular method for generating images based on text prompts, often have a specific character or object in mind — they may, for example, be developing a storyboard around an animated mouse or brainstorming an ad campaign for a specific toy.

Prior research has enabled these creators to personalize the output of diffusion models to focus on a specific subject using fine-tuning — where a user trains the model on a custom dataset — but the process can be time-consuming and inaccessible for general users.

JeDi, a paper by researchers from Johns Hopkins University, Toyota Technological Institute at Chicago and NVIDIA, proposes a new technique that allows users to easily personalize the output of a diffusion model within a couple of seconds using reference images. The team found that the model achieves state-of-the-art quality, significantly outperforming existing fine-tuning-based and fine-tuning-free methods.

JeDi can also be combined with retrieval-augmented generation, or RAG, to generate visuals specific to a database, such as a brand’s product catalog.

New Foundation Model Perfects the Pose

NVIDIA researchers at CVPR are also presenting FoundationPose, a foundation model for object pose estimation and tracking that can be instantly applied to new objects during inference, without the need for fine-tuning.

The model, which set a new record on a popular benchmark for object pose estimation, uses either a small set of reference images or a 3D representation of an object to understand its shape. It can then identify and track how that object moves and rotates in 3D across a video, even in poor lighting conditions or complex scenes with visual obstructions.

FoundationPose could be used in industrial applications to help autonomous robots identify and track the objects they interact with. It could also be used in augmented reality applications where an AI model is used to overlay visuals on a live scene.

NeRFDeformer Transforms 3D Scenes With a Single Snapshot

A NeRF is an AI model that can render a 3D scene based on a series of 2D images taken from different positions in the environment. In fields like robotics, NeRFs can be used to generate immersive 3D renders of complex real-world scenes, such as a cluttered room or a construction site. However, to make any changes, developers would need to manually define how the scene has transformed — or remake the NeRF entirely.

Researchers from the University of Illinois Urbana-Champaign and NVIDIA have simplified the process with NeRFDeformer. The method, being presented at CVPR, can successfully transform an existing NeRF using a single RGB-D image, which is a combination of a normal photo and a depth map that captures how far each object in a scene is from the camera.

VILA Visual Language Model Gets the Picture

A CVPR research collaboration between NVIDIA and the Massachusetts Institute of Technology is advancing the state of the art for vision language models, which are generative AI models that can process videos, images and text.

The group developed VILA, a family of open-source visual language models that outperforms prior neural networks on key benchmarks that test how well AI models answer questions about images. VILA’s unique pretraining process unlocked new model capabilities, including enhanced world knowledge, stronger in-context learning and the ability to reason across multiple images.

The VILA model family can be optimized for inference using the NVIDIA TensorRT-LLM open-source library and can be deployed on NVIDIA GPUs in data centers, workstations and even edge devices.

Read more about VILA on the NVIDIA Technical Blog and GitHub.

Generative AI Fuels Autonomous Driving, Smart City Research

A dozen of the NVIDIA-authored CVPR papers focus on autonomous vehicle research. Other AV-related highlights include:

• NVIDIA’s AV applied research, which won the CVPR Autonomous Grand Challenge, is featured in this demo.
• Sanja Fidler, vice president of AI research at NVIDIA, will present on vision language models at the Workshop on Autonomous Driving on June 17.
• Producing and Leveraging Online Map Uncertainty in Trajectory Prediction, a paper authored by researchers from the University of Toronto and NVIDIA, has been selected as one of 24 finalists for CVPR’s best paper award.

Also at CVPR, NVIDIA contributed the largest ever indoor synthetic dataset to the AI City Challenge, helping researchers and developers advance the development of solutions for smart cities and industrial automation. The challenge’s datasets were generated using NVIDIA Omniverse, a platform of APIs, SDKs and services that enable developers to build Universal Scene Description (OpenUSD)-based applications and workflows.

NVIDIA Research has hundreds of scientists and engineers worldwide, with teams focused on topics including AI, computer graphics, computer vision, self-driving cars and robotics. Learn more about NVIDIA Research at CVPR.

NVIDIA launched NVIDIA Studio at COMPUTEX in 2019. Five years and more than 500 NVIDIA RTX-accelerated apps and games later, it’s bringing AI to even more creators with an array of new RTX technology integrations announced this week at COMPUTEX 2024.

Newly announced NVIDIA GeForce RTX AI laptops — including the ASUS ProArt PX13 and P16 and MSI Stealth 16 AI+ laptops — will feature dedicated RTX Tensor Cores to accelerate AI performance and power-efficient systems-on-a-chip with Windows 11 AI PC features. They join over 200 laptops already accelerated with RTX AI technology.

NVIDIA RTX Video, a collection of technologies including RTX Video Super Resolution and RTX Video HDR that enhance video content streamed in browsers like Google Chrome, Microsoft Edge and Mozilla Firefox, is coming to the free VLC Media Player. And for the first time in June, creators can enjoy these AI-enhanced video effects in popular creative apps like DaVinci Resolve and Wondershare Filmora.

DaVinci Resolve and Cyberlink PowerDirector are adding NVIDIA’s new H.265 Ultra-High-Quality (UHQ) mode, which uses the NVIDIA NVENC to increase high-efficiency video coding (HEVC) and encoding efficiency by 10%.

NVIDIA RTX Remix, a modding platform for remastering classic games with RTX, will soon be made open source, allowing more modders to streamline how assets are replaced and scenes are relit. RTX Remix will also be made accessible via a new REST application programming interface (API) to connect the platform to other modding tools like Blender and Hammer.

Creative apps are continuing to adopt AI-powered NVIDIA DLSS for higher-quality ray-traced visuals in the viewport, with 3D modeling platform Womp being the latest to integrate DLSS 3.5 with Ray Reconstruction.

NVIDIA unveiled Project G-Assist, an RTX-powered AI-assistant technology demo that provides context-aware help for PC games and apps.

The new NVIDIA app beta update adds 120 frames per second AV1 video capture and one-click performance-tuning.

And the latest Game Ready Driver and NVIDIA Studio Driver are available for installation today.

Video Gets the AI Treatment

RTX Video is a collection of real-time, AI-based video enhancements — powered by RTX GPUs equipped with AI Tensor Cores — to dramatically improve video quality.

It includes RTX Video Super Resolution — an upscaling technology that removes compression artifacts and generates additional pixels to improve video sharpness and clarity up to 4K — and RTX Video HDR, which transforms standard dynamic range videos into stunning high-dynamic range on HDR10 displays.

NVIDIA has released the RTX Video software development kit, which allows app developers to add RTX Video effects to creator workflows.

Blackmagic Design’s DaVinci Resolve, a powerful video editing app with color correction, visual effects, graphics and audio post-production capabilities, will be one of the first to integrate RTX Video. The integration is being demoed on the COMPUTEX show floor.

Wondershare Filmora, a video editing app with AI tools and pro-level social media video editing features, will support RTX Video HDR, coming soon.

VLC Media Player — an open-source, cross-platform media player, has added RTX Video HDR in its latest beta release, following its recently added support for Mozilla Firefox.

NVIDIA also released a new UHQ mode in NVENC, a dedicated hardware encoder on RTX GPUs, for the HEVC video compression standard (also known as H.265). The new mode increases compression by 10% without diminishing quality, making NVENC HEVC 34% more efficient than the typically used x264 Medium compression standard.

DaVinci Resolve and Cyberlink PowerDirector video editing software will be adding support for the new UHQ mode in their next updates. Stay tuned for official launch dates.

RTX Remix Open Sources Creator Toolkit

NVIDIA RTX Remix allows modders to easily capture game assets, automatically enhance materials with generative AI tools and create stunning RTX remasters with full ray tracing.

RTX Remix open beta recently added DLSS 3.5 support featuring Ray Reconstruction, an AI model that creates higher-quality images for intensive ray-traced games and apps.

Later this month, NVIDIA will make the RTX Remix Toolkit open source, allowing more modders to streamline how assets are replaced and scenes are relit. The company is also increasing the supported file formats for RTX Remix’s asset ingestor and bolstering RTX Remix’s AI Texture Tools with new models.

NVIDIA is also making the capabilities of RTX Remix accessible via a new powerful REST API, allowing modders to livelink RTX Remix to other DCC tools such as Blender and modding tools such as Hammer. NVIDIA is also providing an SDK for the RTX Remix runtime to allow modders to deploy RTX Remix’s renderer into other applications and games beyond DirectX 8 and 9 classics.

Catch Some Rays

NVIDIA DLSS 3.5 with Ray Reconstruction enhances ray-traced image quality on NVIDIA RTX and GeForce RTX GPUs by replacing hand-tuned denoisers with an NVIDIA supercomputer-trained AI network that generates higher-quality pixels in between sampled rays.

Previewing content in the viewport, even with high-end hardware, can sometimes offer less-than-ideal image quality, as traditional denoisers require hand-tuning for every scene. With DLSS 3.5, the AI neural network recognizes a wide variety of scenes, producing high-quality preview images and drastically reducing time spent rendering.

The free browser-based 3D modeling platform Womp has added DLSS 3.5 to enhance interactive, photorealistic modeling in the viewport.

Chaos Vantage and D5 Render, two popular professional-grade 3D apps that feature real-time preview modes with ray tracing, have also seen drastic performance increases with DLSS 3.5 — up to a 60% boost from Ray Reconstruction and 4x from all DLSS technologies.

Tools That Accelerate AI Apps

The vast ecosystem of open-source AI models currently available are usually pretrained for general purposes and run in data centers.

To create more effective app-specific AI tools that run on local PCs, NVIDIA has introduced the RTX AI Toolkit — an end-to-end workflow for the customization, optimization and deployment of AI models on RTX AI PCs.

Partners such as Adobe, Topaz and Blackmagic Design are integrating RTX AI Toolkit within their popular creative apps to accelerate AI performance on RTX PCs.

Developers can learn more on the NVIDIA Technical Blog.

Editor’s note: This post is part of Into the Omniverse, a series focused on how artists, developers and enterprises can transform their workflows using the latest advances in OpenUSD and NVIDIA Omniverse.

Industrial digitalization is driving automotive innovation.

In response to the industry’s growing demand for seamless, connected driving experiences, SoftServe, a leading IT consulting and digital services provider, worked with Continental, a leading German automotive technology company, to develop Industrial Co-Pilot, a virtual agent powered by generative AI that enables engineers to streamline maintenance workflows.

SoftServe helps manufacturers like Continental to further optimize their operations by integrating the Universal Scene Description, or OpenUSD, framework into virtual factory solutions — such as Industrial Co-Pilot — developed on the NVIDIA Omniverse platform.

OpenUSD offers the flexibility and extensibility organizations need to harness the full potential of digital transformation, streamlining operations and driving efficiency. Omniverse is a platform of application programming interfaces, software development kits and services that enable developers to easily integrate OpenUSD and NVIDIA RTX rendering technologies into existing software tools and simulation workflows.

Realizing the Benefits of OpenUSD

SoftServe and Continental’s Industrial Co-Pilot brings together generative AI and immersive 3D visualization to help factory teams increase productivity during equipment and production line maintenance. With the copilot, engineers can oversee production lines and monitor the performance of individual stations or the shop floor.

They can also interact with the copilot to conduct root cause analysis and receive step-by-step work instructions and recommendations, leading to reduced documentation processes and improved maintenance procedures. It’s expected that these advancements will contribute to increased productivity and a10% reduction in maintenance effort and downtime.

In a recent Omniverse community livestream, Benjamin Huber, who leads advanced automation and digitalization in the user experience business area at Continental, highlighted the significance of the company’s collaboration with SoftServe and its adoption of Omniverse.

The Omniverse platform equips Continental and SoftServe developers with the tools needed to build a new era of AI-enabled industrial applications and services. And by breaking down data silos and fostering multi-platform cooperation with OpenUSD, SoftServe and Continental developers enable engineers to work seamlessly across disciplines and systems, driving efficiency and innovation throughout their processes.

“Any engineer, no matter what tool they’re working with, can transform their data into OpenUSD and then interchange data from one discipline to another, and from one tool to another,” said Huber.

This sentiment was echoed by Vasyl Boliuk, senior lead and test automation engineer at SoftServe, who shared how OpenUSD and Omniverse — along with other NVIDIA technologies like NVIDIA Riva, NVIDIA NeMo and NVIDIA NIM microservices — enabled SoftServe and Continental teams to develop custom large language models and connect them to new 3D workflows.

“OpenUSD allows us to add any attribute or any piece of metadata we want to our applications,” he said.

Boliuk, Huber and other SoftServe and Continental representatives joined the livestream to share more about the potential unlocked from these OpenUSD-powered solutions. Watch the replay:

By embracing cutting-edge technologies and fostering collaboration, SoftServe and Continental are helping reshape automotive manufacturing.

Get Plugged Into the World of OpenUSD

Watch SoftServe and Continental’s on-demand NVIDIA GTC talks to learn more about their virtual factory solutions and experience developing on NVIDIA Omniverse with OpenUSD:

• Transforming Factory Planning and Manufacturing Operations Within Digital Mega Plants
• Getting Started With Generative AI and OpenUSD for Industrial Metaverse Applications

Learn about the latest technologies driving the next industrial revolution by watching NVIDIA founder and CEO Jensen Huang’s COMPUTEX keynote on Sunday, June 2, at 7 p.m. Taiwan time.

Check out a new video series about how OpenUSD can improve 3D workflows. For more resources on OpenUSD, explore the Alliance for OpenUSD forum and visit the AOUSD website.

Get started with NVIDIA Omniverse by downloading the standard license free, access OpenUSD resources and learn how Omniverse Enterprise can connect teams. Follow Omniverse on Instagram, Medium and X. For more, join the Omniverse community on the forums, Discord server, Twitch and YouTube channels. 

Featured image courtesy of SoftServe.

Editor’s note: This post is part of our In the NVIDIA Studio series, which celebrates featured artists, offers creative tips and tricks, and demonstrates how NVIDIA Studio technology improves creative workflows. We’re also deep diving on new GeForce RTX GPU features, technologies and resources, and how they dramatically accelerate content creation.

A superbloom of creative app updates, included in the May Studio Driver, is ready for download today.

New GPU-accelerated and AI-powered apps and features are now available, backed by the NVIDIA Studio platform.

And this week’s featured In the NVIDIA Studio artist, Yao Chan, created the whimsical, spring-inspired 3D scene By the Window using her NVIDIA RTX GPU.

May’s Creative App Rundown

RTX Video is a collection of AI enhancements that improves the quality of video played on apps like YouTube, Prime Video and Disney+. RTX Video Super Resolution (VSR) upscales video for cleaner, crisper imagery, while RTX Video HDR transforms standard dynamic range video content to high-dynamic range (HDR10), improving its visibility, details and vibrancy.

Mozilla Firefox, the third most popular PC browser, has added support for RTX VSR and HDR, including AI-enhanced upscaling, de-artifacting and HDR effects for most streamed videos.

NVIDIA RTX Remix allows modders to easily capture game assets, automatically enhance materials with generative AI tools and create stunning RTX remasters with full ray tracing. RTX Remix recently added DLSS 3.5 support featuring Ray Reconstruction, an AI model that creates higher-quality images for intensive ray-traced games and apps, to the modding toolkit.

Game developers interested in creating their own ray-traced mod for a classic game can download the RTX Remix Beta and watch tutorial videos to get a head start.

Maxon’s Cinema 4D modeling software empowers 3D video effects artists and motion designers to create complex scenes with ease. The integration of the software’s Version 2024.4 with C4D’s Unified Simulation systems now enables control of emission fields to modify behaviors more precisely.

This integration unlocks the ability to orchestrate object interactions with different simulation types, including Pyro, Cloth, soft bodies and rigid bodies. These simulations run considerably faster depending on the RTX GPU in use.

The NVIDIA Omniverse Audio2Face app for iClone 8 uses AI to produce expressive facial animations solely from audio input. In addition to generating natural lip-sync animations for multilingual dialogue, the latest standalone release supports multilingual lip-sync and singing animations, as well as full-spectrum editing with slider controls and a keyframe editor.

Along with accurate lip-sync, facial animations are significantly enhanced by nuanced facial expressions. Pairing Audio2Face with the iClone AccuFACE plug-in, powered by NVIDIA Maxine, Reallusion provides a flexible and multifaceted approach to facial animation, laying the groundwork with audio tracks and adding subtle expressions with webcams.

These latest AI-powered tools and creative app power ups are available for NVIDIA and GeForce RTX GPU owners.

All Things Small, Bright and Beautiful

China-based 3D visual effects artist Yao Chan finds inspiration and joy in the small things in life.

“As the weather gradually warms up, everything is rejuvenating and flowers are blooming,” said Chan. “I want to create an illustration that captures the warm and bright atmosphere of spring.”

Her 3D scene By the Window closely resembles a corner of her home filled with various succulent plants, pots and neatly arranged gardening tools.

“I think everyone has a place or moment that warms their heart in one way or another, and that’s an emotion I want to share with my audience,” said the artist.

Chan usually first sketches out her ideas in Adobe Photoshop, but with her real-life reference already set, she dove right into blocking out the scene in Blender.

Since she wanted to use a hand-painted texture style for modeling the vases and pots, Chan added Blender’s displace modifier and used a Voronoi texture to give the shapes a handcrafted effect.

Blender Cycles’ RTX-accelerated OptiX ray tracing in the viewport, unlocked by Chan’s GeForce RTX GPU, ensured smooth, interactive modeling throughout her creative workflow.

For texturing, Chan referred to former In the NVIDIA Studio featured artist SouthernShotty’s tutorial, using the precision of geometry nodes to highlight the structure of objects and gradient nodes to control the color and transparency of plants.

Chan then used the “pointiness” node to simulate the material of ceramic flower pots.

Lighting was fairly straightforward, consisting of sunlight, a warm-toned key light, a cool-toned fill light and a small light source to illuminate the area beneath the table.

Chan also added a few volume lights in front of the camera.

Finally, to give the image a more vintage look, Chan added noise to the final rendered image in compositing.

Chan’s AI-powered simulations and viewport renderings were powered by her RTX GPU.

“RTX GPUs accelerate workflows and ensure fluent video editing,” she said.

Check out Chan’s latest work on Instagram.

Follow NVIDIA Studio on Instagram, X and Facebook. Access tutorials on the Studio YouTube channel and get updates directly in your inbox by subscribing to the Studio newsletter. 

A collaboration between NVIDIA and academic researchers is prepping robots for surgery.

ORBIT-Surgical — developed by researchers from the University of Toronto, UC Berkeley, ETH Zurich, Georgia Tech and NVIDIA — is a simulation framework to train robots that could augment the skills of surgical teams while reducing surgeons’ cognitive load.

It supports more than a dozen maneuvers inspired by the training curriculum for laparoscopic procedures, aka minimally invasive surgery, such as grasping small objects like needles, passing them from one arm to another and placing them with high precision.

The physics-based framework was built using NVIDIA Isaac Sim, a robotics simulation platform for designing, training and testing AI-based robots. The researchers trained reinforcement learning and imitation learning algorithms on NVIDIA GPUs and used NVIDIA Omniverse, a platform for developing and deploying advanced 3D applications and pipelines based on Universal Scene Description (OpenUSD), to enable photorealistic rendering.

Using the community-supported da Vinci Research Kit, provided by the Intuitive Foundation, a nonprofit supported by robotic surgery leader Intuitive Surgical, the ORBIT-Surgical research team demonstrated how training a digital twin in simulation transfers to a physical robot in a lab environment in the video below.

ORBIT-Surgical will be presented Thursday at ICRA, the IEEE International Conference on Robotics and Automation, taking place this week in Yokohama, Japan. The open-source code package is now available on GitHub.

A Stitch in AI Saves Nine

ORBIT-Surgical is based on Isaac Lab, a modular framework for robot learning built on Isaac Sim. Isaac Lab includes support for various libraries for reinforcement learning and imitation learning, where AI agents are trained to mimic ground-truth expert examples.

The surgical framework enables developers to train robots like the da Vinci Research Kit robot, or dVRK, to manipulate both rigid and soft objects using reinforcement learning and imitation learning frameworks running on NVIDIA RTX GPUs.

ORBIT-Surgical introduces more than a dozen benchmark tasks for surgical training, including one-handed tasks such as picking up a piece of gauze, inserting a shunt into a blood vessel or lifting a suture needle to a specific position. It also includes two-handed tasks, like handing a needle from one arm to another, passing a threaded needle through a ring pole and reaching two arms to specific positions while avoiding obstacles.

One of ORBIT-Surgical’s benchmark tests is inserting a shunt — shown on left with a real-world robot and on right in simulation.

By developing a surgical simulator that takes advantage of GPU acceleration and parallelization, the team is able to boost robot learning speed by an order of magnitude compared to existing surgical frameworks. They found that the robot digital twin could be trained to complete tasks like inserting a shunt and lifting a suture needle in under two hours on a single NVIDIA RTX GPU.

With the visual realism enabled by rendering in Omniverse, ORBIT-Surgical also allows researchers to generate high-fidelity synthetic data, which could help train AI models for perception tasks such as segmenting surgical tools in real-world videos captured in the operating room.

A proof of concept by the team showed that combining simulation and real data significantly improved the accuracy of an AI model to segment surgical needles from images — helping reduce the need for large, expensive real-world datasets for training such models.

Read the paper behind ORBIT-Surgical, and learn more about NVIDIA-authored papers at ICRA.

Editor’s note: This post is part of our In the NVIDIA Studio series, which celebrates featured artists, offers creative tips and tricks, and demonstrates how NVIDIA Studio technology improves creative workflows. We’re also deep diving on new GeForce RTX 40 Series GPU features, technologies and resources, and how they dramatically accelerate content creation.

Video editors have more to look forward to than just April showers.

Blackmagic Design’s DaVinci Resolve released version 19, adding the IntelliTrack AI point tracker and UltraNR AI-powered features to further streamline video editing workflows.

The NAB 2024 trade show is bringing together thousands of content professionals
from all corners of the broadcast, media and entertainment industries, with video editors and livestreamers seeking ways to improve their creative workflows with NVIDIA RTX technology.

The recently launched Design app SketchUp 2024 introduced a new graphics engine that uses DirectX 12, which renders scenes 2.5x faster than the previous engine.

April also brings the latest NVIDIA Studio Driver, which optimizes the latest creative app updates, available for download today.

And this week’s featured In the NVIDIA Studio artist Rakesh created his captivating 3D scene The Rooted Vault using RTX acceleration.

Video Editor’s DaVinci Code

DaVinci Resolve is a powerful video editing package with color correction, visual effects, motion graphics and audio post-production all in one software tool. Its elegant, modern interface is easy to learn for new users, while offering powerful capabilities for professionals.

Two new AI features make video editing even more efficient: the IntelliTrack AI point tracker for object tracking, stabilization and audio panning, and UltraNR, which uses AI for spatial noise reduction — doing so 3x faster on the GeForce RTX 4090 vs. the Mac M2 Ultra.

All DaVinci Resolve AI effects are accelerated on RTX GPUs by NVIDIA TensorRT, boosting AI performance by up to 2x. The update also includes acceleration for Beauty, Edge Detect and Watercolor effects, doubling performance on NVIDIA GPUs.

For more information, check out the DaVinci Resolve website.

SketchUp Steps Up

SketchUp 2024 is a professional-grade 3D design software toolkit for designing buildings and landscapes, commonly used by designers and architects.

The new app, already receiving positive reviews, introduced a robust graphics engine that uses DirectX 12, which increases frames per second (FPS) by a factor of 2.5x over the previous engine. Navigating and orbiting complex models feels considerably lighter and faster with quicker, more predictable performance.

In testing, the scene below runs 4.5x faster FPS using the NVIDIA RTX 4090 vs. the Mac M2 Ultra and other competitors.

SketchUp 2024 also unlocks import and export functionality for OpenUSD files to efficiently manage the interoperability of complex 3D scenes and animations across numerous 3D apps.

Get the full release details.

Art Rooted in Nature

Rakesh’s passion for 3D modeling and animation stemmed from his love for gaming and storytelling.

“My goal is to inspire audiences and take them to new realms by showcasing the power of immersive storytelling, captivating visuals and the idea of creating worlds and characters that evoke emotions,” said Rakesh.

His scene The Rooted Vault aims to convey the beauty of the natural world, transporting viewers to a serene setting filled with the soothing melodies of nature.

Rakesh began by gathering reference material.

He then used Autodesk Maya to block out the basic structure and piece together the house as a series of modules. GPU-accelerated viewport graphics ensured fast, interactive 3D modeling and animations.

Next, Rakesh used ZBrush to sculpt high-resolution details into the modular assets.

He used the ZBrush decimation tool alongside Unreal Engine’s Nanite workflow to efficiently create most of the modular building props.

Traditional poly-modeling workflows for the walls enabled vertex blending shaders for seamless texture transitions.

Textures were created with Adobe Substance 3D Painter. Rakesh’s RTX GPU used RTX-accelerated light and ambient occlusion to bake and optimize assets in mere seconds.

Rakesh moved the project to Unreal Engine 5, where near-final finishing touches such as lighting, shadows and visual effects were applied.

GPU acceleration played a crucial role in real-time rendering, allowing him to instantly see and adjust the scene.

Rakesh then moved to Blackmagic Design’s DaVinci Resolve to color grade the scene for the desired mood and aesthetic, before he began final editing in Premiere Pro, adding transitions and audio.

“While the initial concept required significant revisions, the final result demonstrates the iterative nature of artistic creation — all inspired by my mentors, friends and family, who were always there to support me,” Rakesh said.

Check out Rakesh’s latest work on Instagram.

Follow NVIDIA Studio on Instagram, X and Facebook. Access tutorials on the Studio YouTube channel and get updates directly in your inbox by subscribing to the Studio newsletter. 

From designing dream cars to customizing clothing, 3D product configurators are ringing in a new era of hyper-personalization that will benefit retailers and consumers.

Developers are delivering innovative virtual product experiences and automated personalization using Universal Scene Description (aka OpenUSD), NVIDIA RTX technologies from NVIDIA Omniverse software development kits (SDKs) and application programming interfaces (APIs), and generative AI from NVIDIA Edify models.

Together, these technologies enable developers to create configurator applications that deliver physically accurate, photoreal digital twins of products, revolutionizing the way brands personalize buyer journeys at unprecedented scale.

For example, Dassault Systèmes’ 3DEXCITE brand is adopting Omniverse Cloud APIs to enable interoperability with generative AI services, such as Shutterstock’s Edify3D or Edify 360, directly inside its web-based application.

By using NVIDIA Edify-powered models, trained by Shutterstock, Dassault Systèmes can generate stunning 3D environments from text prompts to instantly personalize scenes representing physically accurate products. And with Omniverse APIs, the company can supercharge the web-based app with real-time ray-traced rendering.

Many other developers are also building 3D product configurator software and solutions with NVIDIA Omniverse SDKs and APIs.

CGI studio Katana has developed a content creation application, COATCreate, used by manufacturers such as Nissan, that allows marketing assets to be staged and created faster with product digital twins. COATCreate also enables users to view and customize the digital twin while wearing an Apple Vision Pro headset, unlocking real-time ray-traced extended reality experiences.

Brickland, another CGI studio, is developing real-time virtual experiences that allow users to customize clothing by choosing from predefined parameters such as color and material. Through their Digitex initiative, Brickland is expanding into digital textures and allowing consumers to visualize and interact with extreme levels of detail in their 3D assets thanks to RTX real-time rendering

Configit connected its powerful configurator logic tool Configit Ace to Omniverse and OpenUSD by streamlining the management of the complex rules system behind the creation of configurators and combining it with the rendering capabilities of Omniverse and RTX. This allows for rapid creation of articulated product configurators and enables the configurator developers to power real-time ray-traced rendering in their solutions.

WPP has developed a content engine that harnesses OpenUSD and AI to enable creative teams to produce high-quality commercial content faster, more efficiently and at scale while remaining aligned with a client’s brand.

Media.Monks has developed an AI-centric professional managed service that leverages Omniverse called Monks.Flow, which helps brands virtually explore different customizable product designs and unlock scale and hyper-personalization across any customer journey.

Accenture Song, the world’s largest tech-powered creative group, is using Omniverse SDKs to generate marketing content for Defender vehicles. Using it with the Edify-powered generative AI microservice, Accenture Song is enabling the creation of cinematic 3D environments via conversational prompts.

Product Digital Twins in the Era of Industrial Digitalization

Forecasts indicate that consumer purchases, including high-value items like vehicles and luxury goods, will increasingly take place online in the coming decade. 3D product digital twins and automated personalization with generative AI serve as invaluable tools for brands to showcase their products and enhance customer engagement in the changing retail landscape.

3D configurators provide tangible benefits for businesses, including increased average selling prices, reduced return rates and stronger brand loyalty. Once a digital twin is built, it can serve many purposes and be updated to meet shifting consumer preferences with minimal time, cost and effort.

Creating a 3D Product Configurator

The process of creating a 3D product configurator begins with harnessing OpenUSD’s powerful composition engine and interoperability. These features enable developers to create dynamic, interactive experiences that accurately reflect the nuances of each product.

Teams can also integrate generative AI technologies into OpenUSD-based product configurators using NVIDIA Omniverse APIs to enhance the realism and customization options available to users. By leveraging AI, configurators can intelligently adapt to user inputs, offering personalized recommendations and dynamically adjusting product configurations in real time. And with NVIDIA Graphics Delivery Network , high-quality, real-time viewports can be embedded into web applications so consumers can browse products in full fidelity, on nearly any device.

The possibilities for 3D product configurators are virtually limitless, applicable across a wide range of industries and use cases.

To start, get NVIDIA Omniverse and follow along with a tutorial series.

NVIDIA researchers have pumped a double shot of acceleration into their latest text-to-3D generative AI model, dubbed LATTE3D.

Like a virtual 3D printer, LATTE3D turns text prompts into 3D representations of objects and animals within a second.

Crafted in a popular format used for standard rendering applications, the generated shapes can be easily served up in virtual environments for developing video games, ad campaigns, design projects or virtual training grounds for robotics.

“A year ago, it took an hour for AI models to generate 3D visuals of this quality — and the current state of the art is now around 10 to 12 seconds,” said Sanja Fidler, vice president of AI research at NVIDIA, whose Toronto-based AI lab team developed LATTE3D. “We can now produce results an order of magnitude faster, putting near-real-time text-to-3D generation within reach for creators across industries.”

This advancement means that LATTE3D can produce 3D shapes near instantly when running inference on a single GPU, such as the NVIDIA RTX A6000, which was used for the NVIDIA Research demo.

Ideate, Generate, Iterate: Shortening the Cycle

Instead of starting a design from scratch or combing through a 3D asset library, a creator could use LATTE3D to generate detailed objects as quickly as ideas pop into their head.

The model generates a few different 3D shape options based on each text prompt, giving a creator options. Selected objects can be optimized for higher quality within a few minutes. Then, users can export the shape into graphics software applications or platforms such as NVIDIA Omniverse, which enables Universal Scene Description (OpenUSD)-based 3D workflows and applications.

While the researchers trained LATTE3D on two specific datasets — animals and everyday objects — developers could use the same model architecture to train the AI on other data types.

If trained on a dataset of 3D plants, for example, a version of LATTE3D could help a landscape designer quickly fill out a garden rendering with trees, flowering bushes and succulents while brainstorming with a client. If trained on household objects, the model could generate items to fill in 3D simulations of homes, which developers could use to train personal assistant robots before they’re tested and deployed in the real world.

LATTE3D was trained using NVIDIA A100 Tensor Core GPUs. In addition to 3D shapes, the model was trained on diverse text prompts generated using ChatGPT to improve the model’s ability to handle the various phrases a user might come up with to describe a particular 3D object — for example, understanding that prompts featuring various canine species should all generate doglike shapes.

NVIDIA Research comprises hundreds of scientists and engineers worldwide, with teams focused on topics including AI, computer graphics, computer vision, self-driving cars and robotics.

Researchers shared work at NVIDIA GTC this week that advances the state of the art for training diffusion models. Read more on the NVIDIA Technical Blog, and see the full list of NVIDIA Research sessions at GTC, running in San Jose, Calif., and online through March 21.

For the latest NVIDIA AI news, watch the replay of NVIDIA founder and CEO Jensen Huang’s keynote address at GTC: 

Editor’s note: This post is part of our In the NVIDIA Studio series, which celebrates featured artists, offers creative tips and tricks, and demonstrates how NVIDIA Studio technology improves creative workflows. We’re also deep diving on new GeForce RTX 40 Series GPU features, technologies and resources, and how they dramatically accelerate content creation.

Creators are getting a generative AI boost with tools announced at NVIDIA GTC, a global AI conference bringing together the brightest minds in AI content creation and accelerated computing.

Adobe Substance 3D Stager and Sampler via Adobe Firefly, the OBS 30.1 YouTube HDR Beta and NVIDIA Omniverse Audio2Face for iClone 8 will also receive sizable upgrades.

DLSS 3.5 with Ray Reconstruction is coming soon to the NVIDIA RTX Remix Open Beta, enabling modders to upgrade their projects with the power of AI. Sample this leap in high graphical fidelity with the new Portal With RTX update available on Steam with DLSS Ray Reconstruction, which provides enhanced ray-traced imagery. Learn more about the DLSS 3.5 update to Portal With RTX.

The March NVIDIA Studio Driver, optimizing the latest creative app updates, is available for download today.

A March of Creative App Upgrades

The Adobe Substance 3D Stager beta announced a new Generative Background feature — powered by Adobe Firefly — to create backdrops for rendered images. Stager’s Match Image tool uses machine learning to accurately place 3D models within the generated background, optimizing lighting and perspective for greater flexibility and realism.

Meanwhile, Substance 3D Sampler’s announced Text to Texture beta — also powered by Adobe Firefly — gives artists a new way to source texture imagery using only a description. All Text to Texture images are square and tileable with proper perspective, ready for material-creation workflows.

Learn more about both apps in the GTC session “Elevating 3D Concepts: GenAI-Infused Design.” Search the GTC session catalog and check out the “Content Creation / Rendering / Ray Tracing” and “Generative AI” topics for additional creator-focused sessions.

The recently launched OBS 30.1 beta will enable content creators to use Real-Time Messaging Protocol — an Adobe open-source protocol designed to stream audio and video by maintaining low-latency connections — to stream high-dynamic range, high-efficiency video coding content to YouTube. Download OBS Beta 30.1 on the OBS website to get started.

NVIDIA Omniverse Audio2Face for iClone 8 uses AI to produce expressive facial animations solely from audio input. In addition to generating natural lip-sync animations for multilingual dialogue, the latest standalone release supports multilingual lip-sync and singing animations, as well as full-spectrum editing with slider controls and a keyframe editor.

For more information on how RTX is powering premium AI capabilities and performance, check out the new AI Decoded blog series and sign up to receive updates weekly.

Follow NVIDIA Studio on Instagram, X and Facebook. Access tutorials on the Studio YouTube channel and get updates directly in your inbox by subscribing to the Studio newsletter.