University of Massachusetts Amherst researchers are using X-ray scans and computational models to learn the secrets of mantis shrimp, crustaceans who fire their appendages with amazing speed and force to ward off enemies and capture prey. On the left is a freeze frame from a high-speed video of an experiment in which a materials-testing machine compresses a mantis shrimp appendage to mimic the way the crustacean would prepare to strike. On the right is a finite element computer model of the appendage under similar loading conditions. Blue, or cold, regions represent areas with low calculated strain energy density. Red, or hot, regions have high calculated strain energy density. The comparisons show the model’s predicted behavior resembles the appendage’s physical behavior. (Images: Michael Rosario, University of Massachusetts Amherst. A video, "An inside look at the mantis shrimp's punching mechanism," is available in the Related Links box at right.)

An archery avocation got Michael Rosario thinking seriously about killer shrimp.

As an undergraduate integrative biology student at the University of California, Berkeley, Rosario co-founded an archery club. As he won competitions, he also studied with Sheila Patek, an evolutionary biology, biomechanics and animal behavior researcher. The intersection of the two interests has led Rosario, a Department of Energy Computational Science Graduate Fellowship (DOE CSGF) recipient, to apply computers to crustaceans and to a featured role on a National Geographic television program.

Rosario and Patek focus on the mantis shrimp, a group of around 400 species named for their resemblance to true shrimp and to the praying mantis, the garden insect hunter. These denizens of the ocean floor and coral reefs deliver powerful, lightning-fast blows. Their club-like appendages, which also can open to deploy needle-sharp claws, reach top speeds of around 50 miles per hour in less than 3 milliseconds. That’s so powerful the claws pull water molecules away from each other to form a cavitation bubble that delivers an additional shock, emits sound waves and a spark of light, and creates a temperature spike of up to 7,000 degrees centigrade – about as hot as the sun’s surface.

What’s more, a mantis shrimp appendage delivers a blow measured at up to 300 pounds, thousands of times the creature’s body weight. It’s one of the highest peak forces any animal produces.

“I kept looking at this animal and trying to figure out its relationship to how I knew bows and arrows work,” Rosario says. Shrimp claws and bows both store muscle power gradually and release it in a burst. But while each of a modern bow’s parts has a specific duty – acting as a brace or storing elastic energy – “when you look at a mantis shrimp appendage, it’s just the exoskeleton. It’s one structure that has to deal with these competing demands.”

As an undergraduate, Rosario largely focused on experiments testing the exoskeletons of mantis shrimp appendages. By using wire to replace the muscle that loads the appendage’s spring, he could measure the amount of force required as a function of displacement in the claw over time. Rosario found it took 40 to 50 Newtons to fully compress it – or the force necessary to lift between 9 and 11 pounds. “I couldn’t close the appendage with my hands.”

Rosario felt stifled by the experiment’s limitations. “What I was really interested in was elastic energy” – the potential energy stored in a crumpled or stretched or otherwise deformed bendable object, like a retracted bowstring. “There’s no good way to measure elastic energy in these systems.” He wanted to know how a mantis shrimp appendage, as a single structure, handled demands that usually require specialized structures.

Rosario joined Patek as a graduate student after she moved her lab to the University of Massachusetts Amherst. Computational models were the key, he decided, to discovering the mantis shrimp’s elastic energy secrets. In an intensive summer course he participated in Biomesh, a finite element analysis (FEA) workshop that Elizabeth Dumont, another UMass biology professor, had helped create. FEA, more typically used in engineering applications, decomposes a model into a series of simple digital bricks, Rosario says, then computes the physical properties and changes in each. Starting with computer tomography (CT) scans of mantis shrimp appendages, he created models to calculate what parts of the appendage bear the load when it’s compressed to strike – in essence, how and where it stores elastic energy.

Multiscale modeling of arterial blood flow can capture adhesion of red blood cells to the arterial wall, clot formation and other small-scale phenomena while capturing events at macro-scales – for instance, clot-induced changes in flow patterns. This is a still from an animation that illustrates a flow of healthy (red) and diseased (blue) blood cells using a method called Dissipative Particle Dynamics (DPD). Each blood cell is represented by a mesh made of 500 DPD-particles, and small spheres show a subset of the DPD particles representing the blood plasma; instantaneous streamlines and slices represent the ensemble average velocity. (Science: Leopold Grinberg and George Karniadakis, Brown University. Visualization: Joseph A. Insley and Michael E. Papka, Argonne National Laboratory.)

Despite gains in identifying and treating them, the cerebral blood vessel dilations known as aneurysms cause suffering and death for up to 5 percent of Americans. Aneurysms can rupture to start hemorrhages with often rapid catastrophic consequences. Clots formed at the site of an aneurysm may detach, block arteries and trigger a stroke.

Blood supply to the brain relies on a highly complex system where, at any point, an aneurysm may occur. Angiograms can show the presence of aneurysms and clots but don’t necessarily reveal their cause – the interactions among and between platelets and red and white blood cell and the endothelial cells that line blood vessels. When injured, endothelial cells trigger platelet aggregation, leading to a clot.

Conventional imaging also doesn’t show the big picture of blood flow throughout the brain – where blood is coming from and where it’s going. Precisely understanding these elements at the level of both gross blood flow and its microscopic properties would greatly improve neurosurgeons’ ability to predict when and where an aneurysm might rupture and when to operate.

High-performance computing (HPC) shows promise for simulating and visualizing brain blood flow at multiple scales. Paving the way is a team led by Leopold Grinberg, senior research associate in the Division of Applied Mathematics at Brown University. Other researchers include Brown’s George Em Karniadakis, Argonne National Laboratory’s (ANL) Joseph A. Insley, Vitali Morozov, Michael E. Papka and Kalyan Kumaran, and Dmitry Fedosov of the Institute of Complex Systems (ICS) in Jülich, Germany.

In 2006, Grinberg began developing arithmetical and software methods capable of simulating 3-D blood flow in complex arterial networks. “The methodology I started to build was based on functional decomposition, with many tasks assigned to different groups of processors, plus multilevel communicating interfaces capable of connecting the data computed by different tasks and synchronizing the processors assigned to different tasks,” Grinberg says.

Working on three fronts – parallel computing, mathematical algorithms and visualization tools –  the team from Brown and ANL received 50 million processor hours on Intrepid, Argonne’s IBM Blue Gene/P, HPC, and 23 million hours on Jaguar, the Cray XT system at Oak Ridge National Laboratory (ORNL). The allocations were granted through the Department of Energy’s Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program. The researchers also had access to Jugene, ICS’s Blue Gene/P, which runs at a peak speed of 1 petaflops – a quadrillion calculations per second – and is almost twice the size of Intrepid.

As they reported in November at the SC11 high-performance computing conference in Seattle, Grinberg and colleagues have created what they think is the first truly multiscale simulation and visualization of an actual biological system. The team’s ambitious target was brain blood flow, the most complex arteriovenous system in the human body. A paper describing the research was one of five finalists for the prestigious 2011 Gordon Bell Prize, which recognizes outstanding results in the application of parallel computing to practical scientific problems.

(a) Traditional approaches to address volume-change in battery materials use acetylene black as the conductive additive and PVDF polymer as the mechanical binder. (b) Conductive polymer with dual functionality, as a conductor and binder, could keep both the electric and mechanical integrity of the electrode during the battery cycles. (c) PF-type conductive polymers' molecular structure, with two key function groups in PFFOMB (carbonyl and methylbenzoic ester) tailor the conduction band and improve the mechanical binding force. (Click to enlarge schematic, courtesy of Lin-Wang Wang, Lawrence Berkeley National Laboratory.)

Electric cars will remain a tough sell until they can travel beyond 100 miles before needing to recharge their batteries. The battery-life bottleneck has driven the search for technologies that extend the energy storage capacity of lithium-ion batteries.

A collaboration, led by Gao Liu of Lawrence Berkeley National Laboratory’s Environmental Energy Technologies Division and Wanli Yang of Berkeley’s Advanced Light Source, has developed a next-generation battery that could fill the need.

Combining computational modeling and advanced materials synthesis, the Berkeley Lab scientists sought a low-cost anode to provide the needed battery boost. Their solution involves replacing inert graphite with silicon nanoparticles bound to a polymer that absorbs eight times the lithium and becomes electrically conductive as it does. The researchers hope the advance will help power the next generation of electronics and extend the range of electric vehicles.

They key to the advance was making silicon a practical anode material.

Scientists have known for decades that a silicon atom can absorb almost four lithium atoms. But in doing so, it balloons to three times its size. The shrinking and swelling during each round of discharging and charging have made silicon an impractical choice for battery anodes.

Materials scientists have tried a work-around by forming the silicon into nanoparticles and connecting them with an inert polymer binder and a graphite electrical conductor to improve performance. After a few charging and discharging cycles, however, the graphite tends to lose contact with the silicon nanoparticles, reducing its conductivity.

The new-generation polymer acts as both a mechanical binder and an electrical conductor. After testing several variations, the Berkeley Lab group designed a variant of a polyfluorine-based polymer hat worked much better than a previously tested conducting polymer. They hypothesized that a particular modification to the variant polymer – the addition of a carbonyl group (a double-bonded carbon-oxygen group) – contributed to its unique electronic properties. Indeed, the researchers used Berkley’s Advanced Light Source to produce an X-ray absorption spectrum that showed there is an additional peak below the electron conduction band. But that didn’t explain how the carbonyl group and an associated additional X-ray absorption peak were related to the enhanced battery performance.

An analysis of aerosol particle and cloud compositions at various altitudes demonstrates the power of so-called n-dimensional computation, which gives researchers information about the role of aerosols on global and local climate change. The parallel coordinate display allows users to view and filter the composition of the particles and clouds; the linked Google Earth display enables a spatial co-reference. Users also may select any sample point or group of points along the flight path to view compositions in the parallel coordinate display. In addition, users may enlist pie charts to summarize the data at that location or area. The dataset was acquired by a single particle mass spectrometer (SPLAT II) on Flight 26 (F26) on April 19-20, 2008 as part of the Indirect and Semi-Direct Aerosol Campaign (ISDAC), a month-long field campaign at the North Slope of Alaska. (Click on image for larger, detailed view.)

To give audiences a better feel for life on a fictional planet, movie director James Cameron made the film Avatar in 3-D and moviegoers donned 3-D glasses. For scientists trying to get a better grasp on what drives climate on Earth, 3-D isn’t good enough. Instead, they’re turning to new techniques to help them see the details and interconnections that ultimately shape the big picture.

“I’m interested in high-dimensional visualization, which is an extension from 3-D visualization into n-dimensional visualization,” says Klaus Mueller, director of the visual analytics and imaging lab in the Center for Visual Computing at Stony Brook University. “Three dimensions is a somewhat solved problem in computer visualizations; n-dimensions is the new frontier. There are all kinds of problems in terms of the user interface and how to make people understand what it means.”

A traditional graph with an x-and-y axis is a two-dimensional visualization, showing the relationship between two variables, or dimensions. An n-dimensional visualization involves showing the relationships between four or more variables.

Mueller is collaborating with the Brookhaven National Lab-led FASTER (FAst-physics System TEstbed and Research) project to develop novel multi-dimensional visualization techniques and tools to help the FASTER researchers see how fine-level, fast physics processes, such as local aerosols and raindrops, shape global climate patterns.

For Mueller, the bigger question that connects all his research is how to effectively convey multi-dimensional information. He can’t give researchers the equivalent of n-dimensional glasses, so instead turns to how we see and interpret information.

We’re familiar with seeing four dimensions – 3-D plus time – and how to visually represent this – motion blur, for example, to show movement. Five dimensions and beyond, however, is a poorly explored territory in visualization yet one that’s crucial for dealing with complex simulations, such as climate. For example, in the FASTER models there are a dozen variables, or dimensions, that are linked to changes in atmospheric pressure.

One thing that’s clear: There’s a learning process in moving to n-dimensional viewing, Mueller says. With two-dimensional imaging sensors – our retinas – we’re able to see in 3-D. In fact, we teach ourselves to develop this fine-tuned spatial sense. When babies reach out to touch objects around them, they’re calibrating their depth and 3-D geometry perception.

“People are familiar with bar charts and pie charts and scatter plots for the display of two- and three-dimensional information,” says Mueller, who’s also an adjunct scientist with BNL’s Computational Science Center. “Anything else is a challenge for many people. This is the challenge for n-dimensional information visualization – to bring this to the masses.”

FASTER and better

The FASTER project is an ideal test bed for these new high-dimensional visualization tools, Mueller says. FASTER is itself an effort to explore how the computational modeling of rapid, small-scale fast physics processes, such as precipitation and cloud dynamics, fits into and shapes global climate models. It’s believed that errors in modeling these fast physics processes are responsible for major uncertainties in global climate model predictions.

“Virtually all of the fast-physics processes interact,” says Yangang Liu, the Brookhaven atmospheric scientist leading FASTER, “so once we get a handle on the individual processes, we want to see how they interact and how to evaluate these interactions. Data integration and visualization are an essential part of this analysis.”

Working with FASTER scientists and Stony Brook doctoral student Zhiyuan Zhang, Mueller has created a unique visualization system that links sophisticated multidimensional information displays with geographical context.

At DOE’s Pacific Northwest National Laboratory, Zelenyuk specializes in using single-particle mass spectrometry to analyze the real-time transformations of nanoparticles. This includes atmospheric particles, such as aerosols, crucial to determining climate. Her experimental runs produce a jungle of spectral data in 450 dimensions for millions of particles.

Automated methods to analyze data with multiple variables often fail when the number of variables exceeds a dozen, Zelenyuk says. “So with 450-dimensional spectral data we needed new tools for visualizing and analyzing our data.”

Mueller, Zelenyuk and collaborators developed a two-part interactive data mining and visual analytics software package. SpectraMiner creates a unique hierarchical dynamical tree or cluster dendogram that can incorporate hundreds of clusters. Data then can be exported to ClusterSculptor so scientist can tune and explore parameters in search of important relationships.

“At each step the scientist is in control of the level of detail and the visualization format,” Mueller says, noting that the visualization tools are now used daily. “This allows them to refine, steer and control the data-mining process.”

A visualization of a lean hydrogen flame simulation shows three computed fields simultaneously. A bowl-shaped turbulent flame floats over the exit flow from a pipe that is swirling as it moves upward. The gray filaments at the bottom depict regions of high turbulence, the transparent red surface highlights the mixing region between the fuel from the pipe and the air outside, and the purple-to-red zone shows the concentration of nitrogen-based emissions from the flame.

With help from a Cray supercomputer, Lawrence Berkeley National Laboratory (LBNL) researchers John Bell and Marcus Day are studying the combustion properties of hydrogen – a potential fuel that science has been trying to tame for decades.

“Hydrogen-based fuels are going to play a key role in low-emissions energy sources of the future,” explains Day, a staff scientist in LBNL’s Center for Computational Sciences and Engineering. To design devices that can use hydrogen-based fuels, “we need to understand much more than we do now about how these flames respond to intense turbulence and high pressures. Both are fundamental features of practical power-generation combustors, but both have significant and complicated effects on the hydrogen flame.”

To understand those effects normally requires experimentation. But experiments aren’t sufficient here because flames in those conditions are extremely hard to measure accurately – and the generated data defies easy interpretation.

“Experiments with real flames are in a relatively hostile environment – it is very hot – and you need to not disturb the flame when you take measurements,” says Bell, an LBNL senior staff scientist. And “many of the things you want to look at are chemical species that live only for a short time in the actual flame zone. They can’t be measured directly.”

Although computer-simulated flames obviously don’t take place in a combustor and are not subject to Mother Nature’s unpredictability, they nevertheless are quite difficult to produce and study. Experimental-quality combustion simulations are complex and require loads of computer time. Bell and Day have access to 40 million processor hours on Jaguar, based at Oak Ridge National Laboratory, through the Department of Energy’s Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program.

Running the simulations is only part of the challenge. The huge volumes of data that result must be compared to experimental data and further explored to understand the new information that the data contain.

“Simulations don’t match experiments exactly,” Bell says, “but they do provide useful insights into how the flames are behaving.” The data help paint a more complete picture of the combustion system – a picture that is simply inaccessible by any other means. As Bell and Day figure out how to quantify uncertainties in the models they use to describe these systems, they can improve predictions and design more practical systems.

What’s more, Bell says, “we hope our work will open the door to an increasingly rich set of activities that couple computational, theoretical and experimental components of combustion science.”

Stabilizing the flame

Unlike fossil fuels that pump carbon dioxide and other pollutants into the air, hydrogen is virtually waste-free. It’s also a basic component of Earth’s atmosphere. Hydrogen is oxygen’s partner in forming water ­– and, it turns out, flames.

In an idealized setting oxygen and hydrogen burn together with near-zero emissions but only if the mix is sufficiently lean, meaning a low fuel-to-oxygen ratio. However, at these leaner mixtures, the flame becomes highly unstable and difficult to use in practical scenarios. High-pressure conditions tend to exacerbate the difficulties.

The same physics that makes hydrogen flames hard to study in the lab lead to difficulties on the computer. The flames blow off or flash back into the fuel supply. Strong turbulence can also have a huge impact – the flames will be twisted, violently shredded and possibly extinguished.

The tiny white yeast colonies in the right panel interspersed with larger normal colonies are cells that have had a synthetic chromosome inserted and their DNA shuffled by the lab-induced SCRaMbLE system, which introduces changes that slow cell growth. By comparison, all colonies on the left are grown from the standard lab yeast strain and appear uniform. (Click on image to enlarge.)

Last year the J. Craig Venter Institute made waves by creating the first fully synthetic bacterial genome. Now a group from Johns Hopkins University has extended that work to yeast, producing a built-from-scratch chromosome that works just like the natural chromosome it replaced.

The project, described today in the advance online issue of the journal Nature, is the first step in creating a modular, synthetic organism that its makers hope will act as a biological factory for churning out medicines or substances that break down toxic waste.

Led by biologists Joel Bader, Jef Boeke and Srinivasan Chandrasegaran, the team relied on the computational skills of Sarah Richardson, a graduate student in Bader’s lab and an alumna of the Department of Energy Computational Science Graduate Fellowship (DOE CSGF), to design the chromosome and to oversee its construction. Software assisted biologists in fashioning a chromosome containing millions of individual DNA bases and thousands of functioning genes in a highly structured, modular system.

“Probably the most computationally difficult algorithm is the segmentation of a chromosome into assemble-able bits,” Richardson says. To manage such a data-intensive project, Richardson searched for programs she could modify for her team’s task. She spoke to geneticists who wrote widely used gene annotation software but quickly discovered that the tools, which ensure genes are correctly sequenced and labeled, fell flat at breaking the chromosome down and moving genes around.

“The biggest problem was that there are not (publicly available) algorithms to edit chromosomes or genomes,” Richardson says. “So I set out to write those algorithms and create that framework for editing sequence on a large scale.”

The result was a software suite called BioStudio and an associated program called GeneDesign. Together, the software assists in designing genetic constructs and tracking the progress of synthesis and assembly. Richardson specifically designed the programs to be as generic and user friendly as possible. She wove in touches adapted from open-source packages, such as a collaborative wiki-like interface with revision-control systems and color-coding graphics to assist editing tasks.

In yeast, all the essential genes – ones the organism can’t survive without – are known. With that information in place, the visualization software colors all the essential genes red. “The red flag on essential genes,” she says, “really lets you know if you are editing a particular gene, you are potentially affecting the fitness of the yeast.” All genes with known functions follow the color-coding system, enabling the designers to monitor changes they are making.

Although the computer can automate many tasks, deciding which genes to move around requires a scientist’s experienced eye for subtle detail.

“It turns out it is pretty hard for the computer to decide what stays and what goes” in the genome design, she says. “First you need to know what you want. Then you can apply the algorithms.”

As part of her Department of Energy Computational Science Graduate Fellowship, Richardson served a 2009 practicum under Adam Arkin, director of Lawrence Berkeley National Laboratory’s Physical Biosciences Division. She made important contributions to Arkin’s research into an RNA-based transcription attenuator found in Staphylococcus aureus.

Attenuators are regulatory sequences that halt gene transcription in bacteria and other prokaryotes. The researchers want to make it a standardized tool for fine-grained gene expression control.

“This RNA-regulated attenuator provides an opportunity to, for one, engineer it for better function so it really is an off switch and has a large dynamic range,” Arkin says. Because it blocks transcription, an engineered attenuator might be built into a family of parts operating similarly but orthogonally – without interfering with other functions – in cells, Arkin says.

Attenuators have two pieces, like a lock and key, Richardson says. Arkin’s group wants to modify them so a specific lock works only with a particular key. Two or more could be inserted in the same transcript, working together much like a logic circuit on a computer chip.

Richardson’s practicum focused on improving and augmenting existing computer code to generate new attenuators orthogonal to wild type S. aureus. She learned the Python programming language and wrote an interface between the group’s existing code and the Vienna RNA Package – software designed to predict and compare RNA secondary structures. The bridge made it possible to interchange Vienna with mFold, another RNA secondary structure prediction package, and to compare the two programs’ predictions.

The group’s code starts with the wild-type lock and key and then mutates it. Richardson devised an algorithm to choose a large set of mutually orthogonal or nonorthogonal attenuators from the mutants. She then worked on a cloning technique and lab protocol to synthesize and evaluate proposed attenuators.

Months after Richardson finished her practicum, Arkin’s group was still using software she helped create. The group also improved and automated her lab protocol.

A frame from an animation showing the possible route into the Atlantic Ocean of oil and dispersant from the spot of the Deepwater Horizon spill in the Gulf of Mexico.

Within days of the Deepwater Horizon oil well blowout on April 20, 2010, Los Alamos National Laboratory (LANL) oceanographer Mathew Maltrud, working at the New Mexico Supercomputing Applications Center, ran a simulation of the spilled oil and sent a visualization to collaborator Synte Peacock of the National Center for Atmospheric Research (NCAR).

Maltrud and Peacock had been studying – and continue to study – ocean dynamics and the relationship between the ocean, climate, the environment and Earth’s atmosphere. All of the gushing oil was at once fascinating and fearful, and it piqued their curiosity.

“Even very experienced ocean modelers looked at the speed of the ocean currents in the simulations and thought, ‘Man, we forget how fast the ocean can move sometimes.’” says Maltrud. “Most of us are working on climate time scales; we’re not used to thinking in terms of weeks to months. It was very impressive in terms of speed.”

About the same time, the two heard from Peacock’s former Ph.D. adviser and occasional collaborator Martin Visbeck, who lives in Germany. Visbeck mentioned that a low-grade panic was sweeping parts of Europe, where citizens began to fear the oil might someday pollute their coasts. With visions of the Alaskan Exxon Valdez spill still fresh after 20 years, many were wondering when Gulf oil might reach their continent and how diluted might it be. Visbeck told his German colleagues of his friends who had a global model that might provide some answers. Suddenly, real-time events were driving science at a frantic pace.

The U.S. Department of Energy allotted Maltrud generous computing hours on the Jaguar Cray XT system at Oak Ridge National Laboratory’s Leadership Computing Facility. He began exhaustive weeks programming a series of ensembles, each a varied group of simulations in which a certain parameter is changed. In this case, the only variant was  the ocean’s initial state. The researchers sought to answer a riveting question: What were the chances that oil, exuding at a rate ranging from an estimated 1,000 barrels a day in April to upward of 62,000 barrels a day in August, would escape the Gulf and lurch up the Atlantic Seaboard and beyond to the coasts of Europe?

Meanwhile, other models were capturing world attention because they used real-time data and monitored the situation the way weather is forecast. Businesses, tourists, governments, environmentalists – all wanted to know the spill’s effect on local beaches, on ocean flora and fauna, and on seafood safety.

“We sought a different approach,” Maltrud says. “We didn’t try to make predictions, but tried instead to understand statistical distributions of what’s possible because that’s the kind of thing that our model can do. Those other models tried to represent exactly what was going on in the ocean at a given time, so we figured that we would try to shed some light on how long it might take oil to leave the Gulf, where it might go, and in what relative concentrations in relation to what was being released at the spill site.

The model Maltrud ran is called POP (Parallel Ocean Program). He made major contributions to its development at LANL in the ’90s; it is the ocean component of NCAR’s Community Climate System Model.

The crux of the modeling problem was a Gulf feature known as the Loop Current, a complex “clockwise surface circulation” entering the Gulf through the Yucatan Channel, and exiting in the Florida Current through the Florida Straits. Maltrud calls the Loop Current the “big meander” because its behavior is akin to a river on a flood plain: now and then changing course, creating oxbow lakes and mini-currents. In the Gulf, these small currents are eddies, subsets of the major loop circulation. The eddies are complex because there may be two or three going at any given time, each broken off from the Loop Current at its own interval and moving in its own pattern.

These tracers include carbon and radiocarbon isotopes, paleotracers (fossils from the sea, in sediments and shells) and theoretical tracers called transit time distributions, which get at how long it’s been since water at the ocean’s interior was lost to the surface – useful for tracking the behavior of greenhouse gases carbon dioxide and methane and the long-banned chlorofluorcarbons (CFCs).

Add passive dye to that list of tracers. In fact, a key reason she and her Los Alamos National Laboratory collaborator Mathew Maltrud could start simulations so quickly during the Deepwater Horizon incident was that passive dye already was part of their modeling repertoire and the code had been worked out for previous simulations.

“It was easy to set this model up,” Peacock says, “like flipping a switch and letting the model go. You specify the latitude and longitude for where you want to inject (the dye), the depth, and how long you want to inject for.”

Earlier in 2010, Peacock and Maltrud had reported on their simulation of CFCs and numerous other tracers in a 100-year ocean circulation model. The model’s extreme high resolution of .1 degree let the researchers follow the movements of eddies, which previous CFC distribution studies characterized poorly because their resolutions were coarse in comparison. Though the eddying model did not directly lead to the Deepwater Horizon incident study, it laid the groundwork for the researchers to act and implement similar theory.

The model the researchers ran is considered to be one of the most realistic global fine-resolution eddying models, and the only one to simulate such a large set of tracer distributions, thanks to the power of the Jaguar supercomputer at Oak Ridge National Laboratory.

CFC modeling is vital in understanding how the ocean can store chemicals for up to thousands of years and globally circulate them at various depths from the surface to hundreds of meters down, a process called ventilation that has multiple effects on climate.

Maltrud wants “to know how gases get transferred from the surface of the ocean down into the depths, and CFCs are a really good way to do this.”

Adds Peacock: “It was the first time that CFCs had been carried for many decades in a model of one-tenth degree resolution. So, the grid was small enough that we could resolve these eddies.”

Peacock refers to eddies as “the weather of the ocean. They happen on small spatial and time scales, are about 1 to 10 kilometers in diameter, with circular features that rotate. You get very large temperature gradients in relationship with surrounding water and very different properties within the core of an eddy. They’re important in transporting, for example, heat and mixing properties in the ocean. One of the reasons we run the high resolution models is to see how accurate we are in parameterizing them.”

CFCs are directly measured in real-time on ocean-going ships with devices called CTD sensors (for conductivity, temperature, depth) from water samples captured in bottles dunked to various depths. Peacock says many measurements have been made over the past 20 years. She and Maltrud used the real-time measurements to test their simulated CFC distributions.

“We didn’t use the real-time observations to push the model in any way, just for the validation,” Peacock says. “This was a very powerful tool that gives us a snapshot of what is going on and clues to how the ocean is being ventilated over the past couple of decades.”

Describing all of the nuclei and the reactions between them, however, demands powerful algorithms running on high-performance computers.

The Universal Nuclear Energy Density Functional (UNEDF) collaboration, which was created by the Department of Energy’s Scientific Discovery through Advanced Computing (SciDAC) program, focuses on developing such descriptions.

An optimized sequence of parameter values in nuclear simulations. (Image courtesy of Stefan Wild.)

The UNEDF collaboration includes researchers from seven national laboratories – Ames, Argonne, Lawrence Berkeley, Lawrence Livermore, Los Alamos, Oak Ridge, and Pacific Northwest – and nine universities: Central Michigan, Iowa State, Michigan State, Ohio State, San Diego State, North Carolina at Chapel Hill, Tennessee-Knoxville, Texas A&M in Commerce and University of Washington. Recently, researchers in this collaboration made a significant advance through the use of density functional theory (DFT).

On Earth, only about 300 kinds of nuclei – specific combinations of protons and neutrons – exist. In accelerators and stars, the number of known nuclei grows to about 3,000, and it could eventually expand to around 6,000. Many of these tiny systems prove extremely difficult to study, largely because they live such short lives before decaying.

Consequently, researchers need ways to accurately simulate these elusive species. Other applications also require extremely precise simulations of interacting nuclei. For example, the National Nuclear Security Administration (NNSA) Stockpile Stewardship Program requires such simulations to assess the safety and functionality of the weapons in the U.S. nuclear stockpile.

Witold Nazarewicz, professor of physics at the University of Tennessee and co-director of UNEDF, describes the basic structure: “A nucleus resembles a droplet of liquid, where there’s a high density inside and a surface area where it drops, and there’s little outside.” Moreover, the quantum behavior of the protons and neutrons at that surface determines the energy of the nucleus and how it interacts with other nuclei. “We need to know how the nuclear energy is generated in a nucleus to use it.”

Talented teamwork

DFT provides an extremely useful, but not necessarily easy, approach to modeling nuclei. For one thing, DFT includes many parameters that must be determined. As Stefan Wild, assistant computational mathematician in the Laboratory for Advanced Numerical Simulations at  Argonne and a fellow in the Computation Institute at the University of Chicago, asks, “What are the best parameters to calibrate these new models to experimental data?”

In the past, scientists searched for the best parameters with what Wild, an alumnus of DOE’s Computational Science Graduate Fellowship, calls “a lot of hand-tuning. They used intuition to pick the values of parameters, ran a simulation, saw how close the answer came to observed data, made small adjustments and ran the simulation again.”

Given the increasing complexity of nuclear simulations, however, “hand-tuning was like looking for a needle in a haystack and far too time consuming to do anything rigorous or thorough.”

As high-performance computing grew more powerful, though, Wild says that “people started thinking about doing something more mathematical” with DFT. For example, Wild and Jorge Moré, an Argonne Distinguished Fellow and director of Argonne’s Laboratory for Advanced Numerical Simulations, developed an algorithm and computer code called POUNDERS (for “practical optimization using no derivatives for sums of squares”).

“If we can determine those densities precisely,” says Witold Nazarewicz, professor of physics at the University of Tennessee, “we can determine the binding energy – the energy stored in the nucleus.”

The energy density functional (EDF) in DFT is an integral of a function of those particle densities. The corresponding energy density is composed of proton and neutron densities, spins, momentum and more. Such an EDF includes variables, or coupling constants, that must be adjusted. The goal of the Universal Nuclear Energy Density Functional (UNEDF) collaboration is to find a universal EDF that works across the entire nuclear landscape, or chart of nuclides, which is a two-dimensional table that collects all nuclei represented by their proton and neutron numbers. Building that requires simulating the properties of thousands of nuclei and doing so repeatedly.

The first step in using DFT to model a nucleus is finding the densities that minimize the energy in a nucleus. This creates an optimization problem that can be tackled by solving the so-called Hartree–Fock–Bogoliubov (HFB) equations of the nuclear DFT.

For the coupling constants, says Nazarewicz, “some of those – but very few – are basically given by theory. Most of them have to be found by comparing DFT calculations with experiments.”

The results of the optimization experiments and starting values for coupling constants can be used to generate nuclear features, such as binding energies, radii, shape deformations and others.

The UNEDF team started with about a dozen coupling constants and used them in HFB equations to calculate more than 100 features of nuclei.

“We compare those observables with experiment and design a chi-square to see if the results are good or not,” Nazarewicz says. “Then we try to adjust the parameters so the chi-square becomes minimum. So this constitutes a huge optimization-minimization problem.”

Fortunately, some of the parameters cannot be varied broadly, because physical bounds limit some of them. That provides some small simplifications to this process.

To make this technique work, Nazarewicz and his colleagues came up with the form of the functional and selected the experimental fit-observables to use.

“We provided our codes to the Argonne group,” Nazarewicz says, “and they designed an optimized procedure that minimized the chi-square. That is POUNDERS (for “practical optimization using no derivatives for sums of squares”), which is a tremendous algorithm because it saved orders of magnitude of time.”

Without POUNDERS, the chi-square converged on a minimum so slowly that it would have taken hundreds of iterations. That takes too long when the minimization involves what Nazarewicz describes as “a problem that is highly nonlinear for more than 100 observables and a dozen or so coupling constants.”

With POUNDERS, DFT can be applied to more nuclei and other applications of this technique will surely keep emerging.

“To do it with only four other people is pretty unusual,” Minnich says. “I would assume a normal company would allocate at least 10 times as many people to the effort. A lot of things we’ve done are new and we weren’t sure if they were even possible.”

Minnich’s collaborators include David Eckhardt of Carnegie-Mellon University, who “knows an awful lot about Plan 9 and has been involved in different parts of the software at different times,” he says.

And Charles Forsyth at Plan 9 distributor Vita Nuova, who “got the compilers going that we needed going and wrote a lot of the code that allowed us to boot the operating system.”

James McKie is “one of the famous names at Bell Labs in the Plan 9 community. He has done things that I don’t know how many other people, if anyone, could have.”

And Eric Van Hensbergen of IBM “was the guy I talked to in 2006 about changing the focus of my work to a Blue Gene. He’s developed almost all the software to make the vision of desktop extension happen.”

Meanwhile, Minnich, as principal investigator, also has been creating code, writing research proposals and coordinating the group’s efforts.

The team is nearing the end of nearly four years of desktop extension funding, largely provided by DOE’s FAST-OS program to promote research in operating and runtime systems for extreme-scale scientific computing.

“We’ve succeeded to a point,” Minnich says, noting the group was working on a key message-passing interface that allows processes to communicate with each other.

“This is a research program,” he says. “We weren’t creating a product, but we did create ideas and learned some things that will live on beyond this little Plan 9 effort. So I do think that, coming up in another year or two, I’ll look at some software offerings and be able to recognize the influence of the work we did.”

“If you go to such a bank to deposit money or take money out, you fill out a form and you give it to somebody and then at some point later they come back and tell you how it went,” he says. “But you’re always at least one step away from your money. And that’s like the ‘batch processing’ model of supercomputing.

This is what it might look like if a user's laptop and supercomputer had access to a common set of files. (Image courtesy of Ronald Minnich, Sandia National Laboratories.)

“What we’ve wanted to do is make it look like the supercomputer is directly attached to your laptop so that they appear to have a common set of files, and the laptop resources, such as the display, are directly available to the supercomputer. So it puts you, in a sense, directly in touch with the money.”

A distinguished technical staff member at Sandia’s Livermore, Calif., site, Minnich is principal investigator of a small research team developing novel software to give programmers what he calls “desktop extension” supercomputer access. His five-person collaboration has 1 million processing hours on the IBM Blue Gene/P supercomputer at Argonne National Laboratory near Chicago, awarded by the U.S. Department of Energy’s competitive INCITE (Innovative and Novel Computational Impact on Theory and Experiment) program.

Beating the batch is not easy in the supercomputing world. A fixture in the mainframe community since the 1950s, batch processing is a way of sharing time on highly prized computational resources by placing programs and data in a queue, processed in order, much as customers must stand in line at a bank waiting for an available teller. If there is a problem, a client goes to the back of the line and starts over.

As with time-sharing office printers, supercomputer programs are run from start to finish without stopping midstream for corrections or other adjustments. That contrasts with interactive processing that allows programmers to intervene on the fly.

Years ago, all computer programmers had to pre-write their programs onto sheets of paper that they then took to keypunch operators to convert into holes in cards. The punch cards were run though a reader, Minnich says, and eventually the programmers got back their results in an interoffice mailing envelope.

“It usually takes anywhere from five minutes to 10 hours before your program gets on the supercomputer,” Minnich says. “If your program started going off course in a direction you didn’t plan, with a batch you have to try and figure out afterwards what happened. And then you’ve got to sit there and resubmit that batch job.”

Although keypunchers have gone the way of telephone operators, programmers still submit their work to software queue managers who control the supercomputer. “You do a lot of work ahead of time. And in the end you surrender control.”

The batch system is like forcing a director who wants to alter a film to “go though the whole process of making the movie again to make a single change.” Minnich and his collaborators saw “value in being able to make supercomputer programs be interactive so you could observe what was going on and change them while they were running.”

Because the medical tools that can measure and predict these problems involve expensive, painful and invasive tests, doctors want less invasive ways to tease out factors that put a patient at risk. Computational science provides one way to better understand blood flow in this complex biological system.

In research presented today at the SC10 supercomputing conference in New Orleans, a Harvard University-led team describes the first large-scale simulation of blood flow in coronary arteries. The simulation uses a realistic description of the arteries’ geometries, and it accounts for fluid flow and the shape and movement of 300 million red blood cells through this system. This multiscale simulation was carried out on the nearly 295,000 processors of the IBM Blue Gene/P system. The research is a finalist for the Gordon Bell Prize, awarded each year at the conference.

The geometry of human coronary arteries from a CTA scan, shown at 12.5 micron resolution. The inset shows blood-flow geometry detail. The red in the detail highlights red blood cells, not endothelial shear stress (ESS), which is represented as a color map on the arterial walls. (Image courtesy of the author.)

One goal of these simulations is to calculate the force – the endothelial shear stress – along the walls of the coronary arteries as blood flows through. Research suggests that such forces play a critical role in forming arterial plaques that lead to ruptured vessels and heart attacks.

“The only way to really determine the shear stress for a patient’s heart is to run a simulation like this,” says first author Amanda Peters, a recipient of the Department of Energy Computational Science Graduate Fellowship and a Harvard applied physics Ph.D. student.

The work, which requires expertise in physics, parallel computing methods and visualization, has grown into an interdisciplinary and international partnership to look at blood flow through the coronary arteries, says Efthimios Kaxiras, Peters’ advisor and John H. Van Vleck professor of pure and applied physics in Harvard’s  Department of Physics and School of Engineering and Applied Sciences.

Kaxiras developed the original project with Sauro Succi, the research director at the Istituto per le Applicazioni del Calcolo  (IAC) of the National Research Council of Italy. In the current study, Peters collaborated with Simone Melchionna, now a senior research scientist at the National Institute of Condensed Matter, Italy, and collaborateur scientifique at the Institute of Materials at École Polytechnique Fédérale de Lausanne (EPFL); and Massimo Bernaschi, IAC chief technology officer. Other co-authors include Mauro Bisson, IAC; Jonas Latt, EPFL; and Joy Sircar, Harvard.

Lecturer of Medicine Charles Feldman and cardiologist Peter Stone, both with Brigham and Women’s Hospital, have been looking at the relationship between plaques and blood flow forces and the progression of heart disease. Their radiologist colleague Frank Rybicki provided detailed medical imaging of the tree of coronary arteries for the study.

Calculating the geometry of coronary artery fluid flow is far more complex than it would be if the arteries were straight tubes, Peters says. The computational mesh of data points designed to emulate the system has to be small enough to account for the subtle kinks and branches of the 12 arteries that feed the heart. The resulting simulation was at a resolution of 12.5 microns, just larger than the average red blood cell, which led to a billion fluid nodes. The simulation looked at these effects for about 1 second, the length of an average heartbeat.

Seeing the need to expand course offerings and graduate student research opportunities, Cherry Murray, dean of the Harvard School of Engineering and Applied Sciences, announced in September the formation of a new Institute of Applied Computational Sciences, directed by Efthimios Kaxiras, John H. Van Vleck Professor of pure and applied physics at Harvard.

So far, the idea is to expand course offerings while building a full curriculum, Kaxiras says. This fall the university began assembling an advisory committee including Harvard faculty from engineering and physical sciences and the medical and business schools.

“We are bringing together a range of people who have knowledge in computation but also interest in pursuing computational approaches,” Kaxiras says. That focus also extends outside the academy to include individuals from industry and national laboratories to understand computational needs in “the real world.”

David Brown, deputy associate director for science and technology at Lawrence Livermore National Laboratory and an advisor to the new program, says that “like experimentation, computation is a scientific approach, and there is a learning curve. We need scientists who are conversant in computation, which is only just starting to gain the maturity of experimentation and observation.”

Brown noted that Harvard students have access to an IBM Blue Gene testbed, a small machine that shares architecture with the massively parallel supercomputers at the national laboratories. “It enables students to get the codes running there at the Institute, then take them to a big machine and run their simulations.”

In the spring, Harvard plans to offer a few new computational courses that could include computational approaches in fluids or materials science or stochastic methods in computational science. Over the next few years, Kaxiras hopes that the curriculum and framework might be in place for a full-fledged computational science graduate program.

It’s called FASTER, for FAst-physics System TEstbed and Research. Its goal is to characterize and evaluate the “fast physics” phenomena of clouds, aerosols (particles suspended in the atmosphere) and precipitation in current and future global climate models.

A frame from a WRF, or weather research and forecasting model, that shows an area over Oklahoma where the FASTER fast-cloud physics project will be put to the test.

Many physical processes that influence Earth’s climate occur on scales of time and space that are too small to be portrayed by most global climate models. Fast physics denotes all these climatic processes collectively, primarily focusing on processes related to clouds and precipitation. These include cloud microphysics, convection, boundary layer processes, radiation and aerosol-cloud interactions.

“We chose the acronym for its obvious relevance, but also because the project will develop the testbed and perform research and address the fast processes as an interacting system, as they occur in nature,” says Liu, a scientist in the Atmospheric Sciences Division (ASD) at Brookhaven National Laboratory (BNL). “Also, global climate models’ resolution increases over time as computer technology advances, and the fast processes in climate models will actually become faster as a result.  And it’s a reminder: We will always try to perform evaluations faster than others.”

FASTER is an elaborate collaboration between atmospheric scientists and climate modelers from BNL, Lawrence Berkeley National Laboratory, NASA, the National Oceanic and Atmospheric Administration and seven other universities and institutions around the world. Many came to BNL, on New York’s Long Island, for two days last November to launch the ambitious Department of Energy (DOE) initiative.

“The kickoff meeting gave us a forum and the opportunity to review some preliminary results, discuss collaborations and lay out items for immediate actions, as well as let some external experts see the extent and scope of the project,” says ASD head Robert McGraw. Kiran Alapaty of DOE’s Atmospheric System Research program is co-manager of the project, and Peter Daum, chairman of the BNL environmental sciences department, rounds out the project management team.

“The DOE anticipates that the outcome from this project will lead to major improvement in climate models, “Alapaty says. “The number of scientists who are participating in this effort is an indication of how important accurate climate change projections are now considered worldwide.”

Because most climate models can’t capture fast physics, the processes must be parameterized, or characterized in a way that models can interpret. Global climate models often differ in how they represent the fast physics driving processes like precipitation and aerosol and cloud formation. Those differences are largely responsible for significant uncertainties in predictions of climate sensitivity and indirect effects from aerosols, Liu says.

“One of the main goals of the fast physics project is to devise parameterizations based on scientific research and observation and bring them to a form suitable for testbed evaluation and use in the climate model,” McGraw says.

Liu adds: “Despite tremendous effort over the past three decades, progress has been frustratingly slow.  In particular, representation of cloud-related processes has remained one of the greatest sources of uncertainty in global climate models.”

Two chief reasons advancement has lagged involve observations and compartmentalized research activities. Modeling and parameterization progress relies heavily on comparing models with observations to discern systematic biases and parameterization problems, Liu says. Traditionally, global-scale evaluation of the way climate models represent clouds has involved comparison with observations such as those from the International Satellite Cloud Climatology Project. Satellite observations are unmatched in covering the globe spatially, but they often aren’t well resolved across time and lack vertical details, such as cloud microphysics and liquid water content, that are much needed to address the fast physics problem.

DOE’s Atmospheric Radiation Measurement Program (ARM) was started more than a decade ago to fill this critical need and scientists around the world use its measurements.

The testbed integrates two major “fast” components: a single column model (SCM), a roughly 100 kilometer by 100 km column that complements traditional global climate models; and a numerical weather prediction model (NWP). The testbeds allow researchers to evaluate fast physics parameterizations quickly.

Wuyin Lin of Brookhaven National Laboratory oversees the FASTER testbed.

Lin has “strong experience in climate model development and climate simulations,” he says, “with a special interest in understanding the physics of cloud and convective processes and their parameterizations in large-scale models, and cloud-climate feedback in general.”

“I’ve incorporated various computational technologies through previous projects on developing an automated framework for numerical weather prediction and seasonal forecasts based on the National Center for Atmospheric Research Community Atmosphere Model.  My research focus is on climate modeling, the impacts of clouds and the uncertainties due to clouds, from regional to global scales.”

Lin says both the SCM and NWP testbeds will have online components and are not computationally intensive, but a high-resolution cloud-resolving platform is in the works that will require supercomputing power. For the SCM, for example, Lin says the online interface provides a convenient platform for parameterization within participating models against well-established case studies.  NWP forecast and analysis products are readily available from national centers, so the NWP testbed will be focused on evaluating the available NWP cloud products against long-term data from the Department of Energy’s Atmospheric Radiation Measurement program, instead of the NWP simulation itself.

“The Web framework is designed such that researchers, including those who are not even typical model parameterization developers, can test their original ideas in full climate models quickly and effectively,” Lin says.  “To aid in the evaluation and development process, a multi-regime case library will be built to integrate and categorize available high-quality forcing and evaluation data.”

Forcing is atmospheric heating or cooling and moistening or drying induced by large-scale motions.  Forcing largely sets the atmospheric environment that may produce clouds and precipitation.

Lin says development of the testbed is moving forward quickly.  Currently, researchers have three versions of the NCAR CAM models (CAM3, CAM4 and CAM5) in the testbed and are collaborating with Geophysical Fluid Dynamics Laboratory and Goddard Institute for Space Studies researchers to work their models in.

“The main theme of the testbed is to confront the models with data – quantify performance of the models under a variety of atmospheric conditions, “ he says.  “The data include the large-scale forcing data that are needed to drive the models and the detailed cloud property products that are needed for model evaluation.”

Lin says the testbed is not only capable of evaluating model physics but also can evaluate the quality of forcing data, in particular when new forcing data are being developed. The Lawrence Livermore National Laboratory group is expanding its continuous forcing data development beyond the currently available three years of 1999-2001.

FASTER team members are putting in a “tremendous effort,” Lin says, to enable the popular Weather Research and Forecast model (WRF) for improving cloud simulations constrained by observational data. In the FASTER framework, the WRF model will take inputs from the ARM observations to produce detailed cloud-scale dynamic and thermodynamic properties.

In astrophysics, dark matter is akin to the idiomatic 800-pound gorilla – a dominating influence that throws its weight around, dictating how stars and galaxies move. But unlike gorillas, dark matter is invisible and its nature is elusive. Physicists can only surmise its existence from its effects on visible matter.

“If you didn’t have dark matter the stars would not be able to move at the velocity they’re moving at. They would fly apart,” says Michael Kuhlen, a postdoctoral researcher at the University of California, Berkeley. Physicists theorize that “halos” of invisible matter host galaxies, providing the gravity that holds them together.

Kuhlen is part of a research team using Jaguar, Oak Ridge National Laboratory’s world-leading Cray XT computer, to help understand dark matter distribution. Their Via Lactea II model first gained attention when it elucidated the “lumpy” nature of the dark matter halo enfolding a galaxy like the Milky Way.

Via Lactea II was made possible with a grant of 1.5 million processor hours on Jaguar, provided through INCITE, the Innovative and Novel Computational Impact on Theory and Experiment program, supported by the Department of Energy’s Office of Science. The researchers, led by Piero Madau of the University of California, Santa Cruz (UCSC), now have a 5 million processor-hour INCITE grant to run their next, more detailed dark matter simulation.

There’s a lot to detail. As much as 83 percent of the matter comprising the universe is dark matter, physicists say. Researchers have offered several theories describing it, but the one gaining the most acceptance casts the mysterious material as a low-temperature fundamental particle that interacts only weakly with ordinary matter – except through gravity.

Via Lactea II is based on this cold dark matter representation. Although its name is Latin for Milky Way, it’s not designed to precisely simulate evolution of our galaxy but of one similar to it, Kuhlen says. “By similar, we mean it has the right mass, the right rotation curve and roughly the correct accretion history” – the process that formed the galaxy.

The latest INCITE grant will enable more precise simulations. Instead of resolving structures on a scale of hundreds of parsecs (a parsec is equal to about 3.26 light-years, or about 19 trillion miles), the group’s model could have a resolution in the tens of parsecs. It’s like switching from a microscope capable of enlarging objects by 10 times to one capable of enlarging objects by 100 times.

“We could actually resolve scales in these dark matter halos that also are accessible to observational astronomy. We can get a correlation between what observers see and what we can predict,” potentially providing further data on dark matter’s nature, Kuhlen says.

]]>