Display of a multijet event from a CMS experiment at the Large Hadron Collider. (CERN.)

Answers to two of the most stubborn questions in science may come down to one point – the point where one particle smashes into another.

The first question comes from physics: What is the strong force? Particles called gluons exert this force inside subatomic particles, keeping protons and neutrons from flying apart in a shower of quarks.

Cosmology gives rise to the second question: What happened in the first microseconds after the Big Bang? A portion of the energy released in that cosmos-spawning event was transformed into the matter that makes up the known universe. But how did that matter form?

Last fall, Dragos Velicanu revealed clues to these mysteries. His presentation won  the Klaus Kinder-Geiger Award for best talk at the Hot Quarks meeting, a biennial workshop for young scientists who study the physics of ultrarelativistic (near light speed) collisions between atomic nuclei.

Velicanu, a Department of Energy Computational Science Graduate Fellowship recipient and doctoral candidate in high-energy physics at the Massachusetts Institute of Technology, reported findings from a series of collisions that lasted 4 hours, 20 minutes in September 2012. The experiment was conducted by the CMS Collaboration, an international group of more than 3,000 scientists, engineers and students using the Compact Muon Solenoid detector at the Large Hadron Collider (LHC), the giant particle collider at CERN, the European physics laboratory. By pitting protons against lead nuclei, the collaboration found a new wrinkle in an unexplained correlation between charged particles that spew out of such collisions.

To analyze the results, Velicanu worked with Wei Li, a former MIT postdoctoral fellow who now is an assistant professor at Rice University. “Together Wei Li and I did the whole analysis completely independently to cross-check our results and help us find and fix bugs,” Velicanu says. “I also helped design, test and monitor some of the triggers used in this and upcoming analyses. Triggers are the algorithms and hardware/software we use to collect our data.”

Cosmologists and physicists agree that answers to their two fundamental questions must lie in quark-gluon plasma, or QGP, a species of matter a particle collider can create. QGP also existed during the universe’s first microseconds. In a QGP, quarks are freed from their partners but, still in the grip of the strong force, move around within a fireball no bigger than the original particles. The QGP winks out of existence in billionths of a second as the quarks recombine, forming everyday protons and neutrons.

However, as these pedestrian particles form, other particles escape from the fireball. By detecting these, scientists intend to find answers to both questions.

The CMS lies 100 meters underground and measures 21 meters long and 15 meters in diameter. With its array of instruments, it can detect not just the muons that inspired its name but also many other particle types. When CMS researchers sent protons into head-on collisions with lead nuclei, the instrument fielded particles from about 2 million collisions, detecting countless charged particles – mostly pions, the lightest union of quark and antiquark.

Velicanu and Li selected only collisions that shot at least 110 charged particles out to the sides – a sign of a direct hit between proton and nucleus –  giving them a comparatively small sample to analyze. They calculated the angle between every conceivable pair of charged particles. Then they searched for correlations among the angles.

“We primarily use distributed computing systems,” Velicanu says. “To run thousands of jobs in parallel, I use either Condor at the MIT cluster, which can run a given program with unique parameters on thousands of computers simultaneously, or CRAB, which I can submit to the entire LHC computing grid that’s spread out around the world, and simultaneously run a program on data held in different continents.”

“Outside work, I like to travel, go camping, hiking, skiing – basically see the world from all elevations, seasons and angles,” says the Department of Energy Computational Science Graduate Fellowship recipient at MIT.

What’s more, he’s fortunate that his advisor is Gunther Roland, a member of the CMS Collaboration at CERN’s Large Hadron Collider, or LHC. In Roland’s laboratory, Velicanu has joined an international research team that’s always looking at physics from new angles.

Velicanu’s impulse to examine landscapes and life from new points of view comes from his parents, who passed on their love of hiking and camping. They left their home in Romania and brought Dragos with them to Massachusetts when he was about to enter elementary school.

He had to adapt quickly. “There was a little bit of a language barrier,” he says. “I knew one word: apple.” But his English improved during summer camp, and by the end of first grade he had the language in hand.

Velicanu’s pursuit of physics seems to be a natural result of aptitude, parental encouragement and the influence of family friends who happened to be mathematicians and physicists. “I was always curious about how the universe works, especially in black holes and things like that,” he says. “In high school I had a very good physics teacher. He was very passionate, and that’s what convinced me to go into physics.”

Now a U.S. citizen like his parents, Velicanu looks forward to the coming decade. The LHC has been shut down for upgrades, but in two to three years it will be running with twice the energy and an increased collision rate. “That can open up some new physics,” Velicanu says, “and the field will become much more exciting.”

And Velicanu will be looking at it from as many angles as possible.

A sequence of false color images generated from a numerical simulation show a MagLIF liner as it is heated by a laser in preparation for an implosion.

On the last Friday before Christmas in 2010, fusion physicist Steve Slutz was in his office at Sandia National Laboratories’ Albuquerque, N.M, location, doing what he’s been doing for the past 34 years: crunching the numbers to get fusion to work on Earth.

This time around, he was calculating the potential fusion energy output from a large-scale version of a new fusion process he and his colleagues had proposed. He’d already shown that magnetized liner inertial fusion, or MagLIF, could theoretically produce enough fusion energy to make it useful for weapons-effects simulations.

Still, if he could show that MagLIF had potential as a large-scale fusion energy source, that could be the added push it needed to get the green light for experiments. That was the problem. Most earlier research by others had dismissed the possibility.

Slutz wanted to see for himself. For MagLIF to go big, he knew he needed a gain in energy of at least 50 times that needed to kick-start the reaction. Hunched over his keyboard, making the calculations, he realized it was possible – with the right technology. He went to a colleague in a neighboring office – one of the few people left in the building – to share this physics good news.

“Go home quick before you figure out what’s wrong with it,” the colleague joked.

Instead, Slutz returned to his calculations and found that, given the right circumstances, MagLIF could produce an energy gain of 1,000.

At which point there was no one at Sandia left with whom to share the news.

Today dozens of Sandia researchers are at work testing MagLIF, seeing if what Slutz’s computer simulations show is indeed a path to fusion that could one day actually power your Christmas lights.

“The message is that MagLIF is a really interesting path because in principle you could have really high gain,” Slutz says. “If this thing works at all.”

Fusion grand slam

Every day you’re bathed in the light and heat of fusion energy. The sun’s massive gravitational self-hug and the resulting sizzling temperatures fuse hydrogen nuclei, releasing vast amounts of energy in the process.

Yet achieving controlled fusion on Earth has proven to be a 60-years-and-counting grand challenge.

The trouble is that controlled fusion is the equivalent of the ultimate baseball bases-loaded grand slam. To make it work, researchers have to take a little bit of star fuel, then heat it, hold it and get it to heat itself for long enough to extract the fusion energy. All this with a fuel that’s heated to a plasma hotter than the interior of the sun – more than hot enough to melt any terrestrial material.

MagLIF (Magnetized Liner Inertial Fusion) envisions using Sandia’s Z machine as a massive magnetic vise to implode, and thus heat, a tiny cylinder full of deuterium to Sun-like temperatures, igniting a fusion reaction.

“I wanted an experimental platform that we could test and see if it works the way we think it will,” says Sandia fusion physicist Steve Slutz, who led the MagLIF computer simulations. “Now our approach is to put the simulations to the test one step at a time.”

The first experimental step was a successful one. In the first use of the Z machine as a nanosecond-fast magnetic boa constrictor, the pulse machine successfully crushed an empty, cylindrical beryllium fuel liner – one designed to hold deuterium-tritium fusion fuel.

“The experimental result – the degree to which the imploding liner maintained its cylindrical integrity throughout its implosion – were consistent with results from earlier computer simulations,” Sandia’s Ryan McBride, the lead researcher on these MagLIF experiments, said in a Sandia announcement last year when his team’s results had been accepted for publication in Physical Review Letters. The major concern in this experiment was whether the beryllium liner was thick enough to withstand the enormous electrical current passing along its surface – a current that gradually vaporizes the liner’s outer surface. “You might say the race is on,” McBride continued. “The question is, can we start off with a thick enough tube such that we can complete the implosion and burn the fusion fuel before the instability eats its way completely through the liner wall?”

In fact, the simulations appear to have found a liner-width sweet spot – X-ray diagnostic imaging of the implosion revealed that the liner walls maintained their integrity throughout the implosion.

“Our 2D simulations (of the liner walls) were too pessimistic,” Slutz says. “The instabilities didn’t grow as much in the experiments as in the simulations.”

After several more step-wise experiments, the MagLIF team hopes to perform a full-integrated experiment of the concept in late 2013.

Says Slutz, “If the fusion yield results are close to what we’re predicting, then we’ll know we’ve got the physics right in the simulations.”

A Reeb graph (above) provides an overview of particle flow during the simulation by summarizing all possible paths of particle flow through a two-dimensional space packed with uniform spheres (right). The technique allows investigators to avoid downloading much of the simulation data. (Lawrence Berkeley National Laboratory.)

Here’s a challenge: Try understanding the action in a movie that has 999 of a thousand frames missing. It might be possible to glean some idea of what’s happening, but the missing frames are likely to contain information crucial to comprehending the plot.

That’s the situation facing high-performance computing (HPC) as it moves to the petascale (quadrillions of calculations per second) and beyond. Many simulations already generate much more data than can be effectively stored and analyzed.

To cope with the situation, researchers sample regular time steps and likely miss interesting behavior. It’s akin to stepping out for popcorn just when Darth Vader tells Luke Skywalker he’s his father. Nothing that happens afterward makes much sense without the missing information.

This disconnect is inspiring Lawrence Berkeley National Laboratory’s visualization group to create a new generation of data analytics and visualization to meet that challenge.

As HPC reaches toward the exascale (a thousand times faster than petascale), a vastly increased number of essentially independent computing cores will make data movement a major bottleneck, says Gunther Weber, a research scientist in the Berkeley Lab visualization group. The current pattern of serially running a large simulation, dumping the data to disk and then doing post-processing analytics and visualization must change, he says; analytics will need to be built into simulations.

As a first step toward that goal, Weber and his research team are embedding data analytics into NYX, a simulation used to explore the behavior of matter in deep space. NYX, developed by Ann Almgren and her colleagues at the lab’s Center for Computational Sciences and Engineering, uses a technique called adaptive mesh refinement in a nested hierarchy of uniform grids that follows matter distribution in both space and time. The cosmology simulations, led by Peter Nugent, Berkeley Lab senior staff scientist, aim to pin down the balance of forces that lead to matter clustering that matches deep space observations. (See sidebar, “Going deep.”)

“These are large, large simulations,” Nugent says. “You need to have the analysis tools embedded in the simulation because you can’t dump out this data every single time step and look at things. Any time you move from memory to somewhere else, it’s painful.”

Type 1a supernova. (Supernova Cosmology Project, Lawrence Berkeley National Laboratory.)

The discovery of that our universe is expanding at an accelerating rate garnered a 2011 Nobel Prize for Saul Perlmutter of the Supernova Cosmology Project at Lawrence Berkeley National Laboratory, but the finding also opened up a plethora of new questions about what is happening in the far reaches of deep space.

There, researchers glimpse remnants of the early universe, shortly after the big bang, by combining computational simulations and data observations such as those from the recently operational Dark Energy Camera in Chile. Armed with this information, they can refine how the universe balances expansion and its countervailing force, gravity, which draws matter together to form stars and galaxies.

Measuring light from distant galaxies, the camera – the most powerful of its kind – will allow scientists to compare the expansion rate of near galaxies with far ones. The result should be a better understanding of the expanding universe, says Peter Nugent, a Berkeley Lab physicist.

“I would like to resolve clusters (of matter) of certain sizes,” he says.  Analysis software could “pick out those clusters. You have to go in knowing what a cluster starts to look like when it forms. That’s the type of thing one can imagine doing in the future when the analytics are fully embedded.”

The NDM-1 enzyme’s structure revealed a large cavity (dark gray) capable of binding a variety of known antibiotics (shown in different colors). Once bound, the enzyme can cut the carbapenem ring, destroying the compound’s antibiotic activity. Modeling the interactions computationally can allow researchers to design compounds that will readily adhere to NDM-1 and prevent it from binding with antibiotics. (Argonne National Laboratory.)

As word spread from India last year that common bacteria strains showed resistance to all known antibiotics, some officials feared that terrorists might find a way to weaponize those bacteria and trigger an epidemic. A later study revealed these bacteria were in virtually every modern country’s drinking water supply. Officials worried that if people picked up one of these bugs, became ill and tried taking antibiotics to recover, the bacteria could bypass any drug.

To produce the first publicly available model of a potent enzyme that was making the bacteria into superbugs, groups from across the Department of Energy’s Argonne National Laboratory and collaborators at Texas A&M University plied the computational muscle of Intrepid, the lab’s IBM Blue Gene/P.

Researchers focused on the bacterial enzyme NDM-1 (New Delhi Metallo-β-lactamase), which grabs a piece of the antibiotic beta-lactam ring, breaks it and renders the antibiotic useless. Analysis revealed the enzyme has an enormous and flexible active site, capable of gobbling the stout rings of existing antibiotics, including those of many powerful carbapenems, antibiotics of last resort. Decoding NDM-1 and quickly publishing their findings, the Argonne team documented the enzymes’ promiscuity and uncovered strategic flaws in existing antibiotic development strategies.

“From our simulations it appears that NDM-1 is capable of processing much larger molecules, voiding the easy path” of modifying existing molecules “to breathe new life into old drugs,” says Andrew Binkowski, assistant scientist in Argonne’s biosciences division and a fellow at the University of Chicago’s Computation Institute.

“The NDM-1 story is a really good test case of speed to solution,” says Binkowski, who collaborates with a team from Argonne, the University of Chicago and Northwestern University. “The time to the final solution is really important. If there is an outbreak and we want to quickly find viable drug leads to treat the infection, we can’t afford to wait hundreds of days. We want it in a matter of hours. Intrepid enables that possibility. We went in short order from headlines about a new national security threat to a crystallographic model of the protein used for designing new antibiotics, just months later.”

Argonne National Laboratory researcher Andrew Binkowski’s calculations of protein structure help find ligands – smaller molecules – that attach to them, to deliver drugs that stop dangerous infections.

But without supercomputers it could take months to model a single ligand, even using the most advanced algorithms available, Binkowski says.

On Intrepid, Argonne’s IBM Blue Gene/P supercomputer, it takes just hours to evaluate the same ligand. That’s important because Binkowski and his colleagues have a database of about 5 million compounds to sift in search of the right ligand. Starting next year, Mira, IBM’s next-generation Blue Gene Q supercomputer, will be able to do such calculations as much as 20 times faster and five times more efficiently.

“We need to look at how atoms can be affected by minute changes in position and find meaningful ways to measure them” in the atom protein and ligand complex, Binkowski says. A lack of computing power meant researchers had to be somewhat imprecise, setting artificial boundaries or ignoring some interactions. Intrepid lets researchers strip away those loose restrictions and run more complete models.

Binkowski’s team has an allocation of 10 million processor hours on Intrepid, awarded through DOE’s INCITE (Innovative and Novel Computational Impact on Theory and Experiment) program. The researchers are in the midst of the largest application of highly advanced and demanding FEP/MD-GCMC (free energy perturbation distributed molecular dynamics-grand canonical Monte Carlo) computations ever performed on protein-ligand interactions. And for the first time they are applying FEP/MD-GCMC as a screening tool. That lets researchers rank different compounds’ binding free energy – a factor that reveals just how effectively a potential ligand would stick to a piece of protein.

“This is a valuable, time-saving scale by which we can measure and compare different compounds to the same target,” Binkowski says. “Whichever ones have the most favorable free energy ranking are the ones we are most interested in.”

For years, the long time it takes to run the FEP/MD-GMGC calculations limited its use, “but now we’re using it to look at thousands of ligands, which has never been done before,” Binkowski says.  “It would have taken literally several lifetimes to do.”

Boron was one of several elements the computational scientists plugged into their model as they investigated ways to induce useful changes in nanotube structures. There were experiments to compare with the results of most of their calculations. There weren’t any to check against the boron-doped nanotube simulations.

“We didn’t think anything about boron, really,” says Bobby Sumpter, Chemical and Materials Sciences Group leader and director of ORNL’s Nanomaterials Theory Institute. “We thought it was interesting how it preferred negative curvature, and we kind of just left it at that.”

Then Humberto Terrones, an ORNL-affiliated researcher from Belgium’s Université Catholique de Louvain, came to visit last year. He and his brother, Mauricio, of Pennsylvania State University and Japan’s Shinshu University, were investigating new nanotube materials.

Humberto Terrones “was talking about how they’d observed these three-dimensional-looking structures when they doped boron in,” Sumpter recalls. “I said, ‘But, Humberto, remember our results? Where we found these interesting effects and we think we understand exactly what happens?’ I hadn’t realized they’d done experiments for boron and just learned about it over a casual discussion – which actually turn out to be usually the most productive scientific discussions – just a cup of coffee with a white board.”

Sumpter and Vincent Meunier of Rensselaer Polytechnic Institute recalculated the boron results and published them jointly with Rice University doctoral student Daniel Hashim’s discovery of three-dimensional, macro-scale nanosponges. “It’s always good to have experimental evidence that backs up theory or vice versa,” Hashim says. “In this case we made the theory and the experimental evidence together and it gave this paper a lot more impact.”

(Click image to enlarge.) Density functional theory calculations compare the energy necessary to substitute boron (red), nitrogen (blue) or sulfur (green) atoms for carbon atoms at 16 positions on a curved carbon nanotube (right). Where the curve is negative (K < 0), the carbon rings have seven atoms (heptagons) and it takes less energy for boron to substitute for carbon; boron selectively creates negative curves in the nanotubes and keeps the ends from closing, so they grow longer. Substitutional energy levels for nitrogen are lowest in pentagonal carbon atom arrangements, making nitrogen most likely to substitute for carbon at the positive (K > 0) positions. Researchers have found nitrogen gives rise to periodic raised rings along nanotubes, leading to bamboo-pole-like architectures. Sulfur can be accommodated in both pentagonal and heptagonal carbon rings and so promotes branched nanotubes.

In 2007, Bobby Sumpter and his Oak Ridge National Laboratory nanomaterials theory collaborators used computers to predict that adding a pinch of boron would produce kinks in carbon nanotubes. As an undergraduate student at Rensselaer Polytechnic Institute (RPI) the next year, Daniel Hashim guessed the curves might make these boron-doped nanotubes good energy-storing materials.

But no one expected what Hashim got when he actually used boron to contaminate the carbon in nanotubes – cylinders thousands of times thinner than a human hair.

“I didn’t know it was going to come out as a porous, three-dimensional solid framework” – a nanosponge big enough to hold in your hand, says Hashim, now a doctoral student at Rice University. “That was the big surprise. We didn’t have the insight at the time to understand that that’s what would happen.” Now researchers are touting these macro manifestations of nanotechnology as possible tools to clean oil spills, store energy, construct bones and meet other needs.

The Oak Ridge researchers used high-performance computing to help understand what’s happening at the atomic scale as these extraordinarily versatile black blocks grow. But it took an off-handed remark to link their work to the sponge-testing experimentalists. (See sidebar, “A spontaneous collaboration.”)

The scientists detail the material’s properties in a paper published online in Nature Scientific Reports:

• It repels water but sops up carbon-containing liquids, holding up to 100 times its weight in oil.

• The material is 99 percent air, so it’s remarkably light and absorbent.

• To get the oil out, just squeeze (tests show the sponges are still elastic after 10,000 compressions) or set on fire. The oil burns off, doing little harm to the reusable material.

• Because the material is made with an iron-based catalyst, magnetic fields can manipulate the sponges, making them easier to move or collect if they’re used for oil spill remediation or other purposes.

Boron, a common element that neighbors carbon on the periodic table and is found in laundry detergent, fiberglass insulation and other products, makes these high-tech sponges special. First, it “acts as a surfactant, which means it sits at the edge of the growth of carbon nanotubes and keeps them open,” says Sumpter, Chemical and Materials Sciences Group leader and director of Oak Ridge’s Nanomaterials Theory Institute. That helps the sponges grow to millimeters in length – extraordinarily large for such structures, says paper co-author Mauricio Terrones, a professor of physics, materials science and engineering at Pennsylvania State University and Japan’s Shinshu University.

A visualization of a Vlasov-Poisson simulation for a bump-on-tail instability problem, where a non-equilibrium distribution of electrons drives an electrostatic wave. The image shows particle density as a function of space and velocity. (Jeffrey Hittinger, Lawrence Livermore National Laboratory.)

Lawrence Livermore National Laboratory (LLNL) computational scientist Jeffrey Hittinger spends his life at extremes. On the job, he focuses on the physics of plasmas – searing clouds of speedy ions and electrons – for fusion energy. Outside work, he tends goal for a San Francisco Bay-area amateur hockey team. Instead of simulating flying particles, he’s blocking or catching flying pucks.

His two interests share a fast pace – extraordinarily fast for plasmas – and a challenging nature. “I’m attracted to difficult things,” Hittinger says, then laughs. “I’m a goalie, so maybe I’m interested in difficult, high-pressure things.”

Likewise, “we get to work on hard problems” at the lab’s Center for Advanced Scientific Computing (CASC), Hittinger says. And like pucks flying in from unexpected directions, “there are always new problems coming at you.”

Hittinger, a Department of Energy Computational Science Graduate Fellowship (DOE CSGF) recipient from 1996 to 2000, creates and tweaks computer algorithms that emulate and elucidate aspects of some of the world’s most complex experiments.

“It’s a mixture of my background and what I stumbled into when I came to the lab,” Hittinger says. As a graduate student, he used gas kinetics to model fluid mechanics. Lab personnel recruited him to improve fluid plasma models for laser-driven inertial confinement fusion (ICF), the goal of the National Ignition Facility (NIF).

In NIF’s stadium-sized building, powerful lasers shoot into a hohlraum, a thimble-sized container holding a BB-sized capsule of frozen hydrogen isotopes. The laser pulse generates powerful X-rays, imploding the pellet with tremendous pressure and heat. The hydrogen atoms fuse, releasing energy in a process similar to that powering the sun.

“For ICF to work, you have to get a nice, clean implosion,” Hittinger says. “To do that, you need all the energy you’re putting into the system to go where you want it to go.” Plasma, however, can interact with the lasers, scattering or reflecting them.

A still from an animation of methane, the blue and silver molecules, escaping from a methane hydrate, the red and silver molecules water molecules that form a cage around methane molecules. (Pacific Northwest National Laboratory.)

Hydrates are icy water-gas compounds abundant in ocean permafrost. They can hold numerous gases such as hydrogen, methane and nitrogen, a Pacific Northwest National Laboratory (PNNL) team’s supercomputer analysis of millions of molecular configurations reveals. Knowing the structure and molecular mechanisms of clathrates, the cage-like water molecules in compounds that host the gases, may help researchers find a feasible way to use hydrogen as an alternative fuel-storage system.

What’s more, the study suggests a mechanism governing how gas molecules hop between adjacent cages. A possible application for exploiting this feature: extracting methane from the hydrates and replacing it with carbon dioxide in a Davy Jones’ locker scenario to remove and safely store the potent greenhouse gas. The study took more than six months on Hopper, the Cray supercomputer at DOE’s National Energy Research Scientific Computing Center (NERSC).

Gas hydrates, long known to form at high pressures in deep ocean fossil fuel pipelines, were suspected as one culprit that complicated attempts to mend the Deepwater Horizon oil spill in 2010. At low pressure – Earth’s surface, say – methane hydrates are unstable and can be lit like an icy match, burning methane on the top as water drips at the bottom.

Researchers and entrepreneurs have begun to notice hydrates’ potential environmental applications.

“There is at least one large-scale effort by a company in Norway to take the methane out of these systems in the bottom of the ocean and try to pop carbon dioxide in there,” says PNNL Laboratory Fellow Sotiris Xantheas, who collaborated on the study with PNNL post-doctoral fellow Soohaeng Yoo Willow. “Take the gas out, use it as fuel for energy and fill the empty scaffold with CO2. You get double the benefit.”

Until Willow and Xantheas’ study, there had been scant information about how guest molecules like hydrogen interact inside clathrates’ hollow cages. Building on previous work, the team already had three-dimensional models of lattices made from cages of 20, 24 and 28 water molecules.

The model used for the study was made with 20 and 24 molecules. X-ray crystallography spelled out the oxygen atoms’ positions, and the team’s previous work suggested efficient ways to place hydrogen atoms in the lattice.

Imagine the clathrates’ cage of 20 water molecules as a soccer ball, Xantheas says. Because these are water molecules, 20 oxygen atoms on the surface means there must also be double that number – 40 – hydrogen atoms. “For the larger hollow cages comprised of 24 and 28 water molecules, there are millions of combinations you can do, and we figured out the most efficient ways and used numerous combinations in our simulations,” he says.

Time evolution of the primary convective activity (white) and lightning (red dots) for Hurricane Rita. (Image: Jon Reisner, Los Alamos National Laboratory.)

Jon Reisner is building the biggest, meanest hurricane model he can, one that grows over the sea from a relatively weak category 1 to a crushing, city-swamping category 5 by the time it makes landfall. He wants everything a hurricane has to offer, from violent, convective updrafts that turn vast quantities of tropical water vapor into ice, to the massive vortex’s swirling eye wall. Most of all, though, he wants lightning. Because it’s these flashes, atmospheric scientists believe, that could be sending an early-warning signal that a hurricane is turning from mean to a monster.

“You need computer modeling to do prediction,” says Reisner, a computational physicist at Los Alamos National Laboratory (LANL). “Right now we can predict a hurricane’s track, but our models have very little, if any, ability to predict hurricane intensification. If you’re going to evacuate Houston 36 hours in advance, you’re going to need a model that can do both.”

Since the devastation wrought by Hurricane Katrina in 2005 – the costliest and one of the deadliest natural disasters in United States history – Reisner is part of a groundswell of researchers using new observations and models to turn hurricane intensity forecasting into a solid predictive science.

Electric hurricanes

At Los Alamos, Reisner’s team led the creation of a three-dimensional model of hurricane lightning activity to test a hypothesis that many atmospheric scientists think holds the key to prediction: that the rate of a tropical storm’s electric flashes are a signal it’s growing stronger.

There’s good reason to think this might be the case. Lightning is a proxy for what we can’t see that’s at the heart of a hurricane’s power. Although Reisner cautions that there are “lots of theories” to describe exactly how and why lightning occurs in hurricanes, it all comes down to a violent mix of water and wind.

Tropical storms – including hurricanes and typhoons – are spawned when enormous volumes of warm water vapor rapidly rise from the ocean surface. This movement can occasionally create hot towers, enormous pillar-like clouds rotating around the vortex’s eye wall. As the warm water vapor is drawn upward it cools into a mixture of super-cooled water, ice crystals and hail-like graupel.

The energy water vapor releases as it cools and forms these particles fuels a hurricane’s ferocious convective winds – and lightning. When these various forms of water collide, there’s a tiny electrical charge transfer. And when they separate they fall at different rates, producing charge separation. With enough watery collisions, the charge separation builds until it produces a pulse of lighting. Thus the more energy release and convection, the more lightning, particularly intra-cloud lightning within the hurricane’s eye wall.

Data deluge

To test this theory, Reisner and colleagues at LANL spent months developing a lightning model that they’ve integrated into HIGRAD, one of the lab’s leading-edge fluid-dynamics models. Coupled to a cloud model, the simulation is as close as anyone’s gotten to mimicking lightning observed in a real hurricane.

Recently, Reisner released results of two intensive comparisons of the HIGRAD model with data on electrical activity from Hurricane Rita (2005) and dual-Doppler radar data from Hurricane Guillermo (1997). Researchers amassed some of the most detailed field data ever from both storms by using storm-chasing planes and ground-based lightning detection networks, including the Los Alamos Sferic Array.

University of Massachusetts Amherst researchers are using X-ray scans and computational models to learn the secrets of mantis shrimp, crustaceans who fire their appendages with amazing speed and force to ward off enemies and capture prey. On the left is a freeze frame from a high-speed video of an experiment in which a materials-testing machine compresses a mantis shrimp appendage to mimic the way the crustacean would prepare to strike. On the right is a finite element computer model of the appendage under similar loading conditions. Blue, or cold, regions represent areas with low calculated strain energy density. Red, or hot, regions have high calculated strain energy density. The comparisons show the model’s predicted behavior resembles the appendage’s physical behavior. (Images: Michael Rosario, University of Massachusetts Amherst. A video, “An inside look at the mantis shrimp’s punching mechanism,” is available in the Related Links box at right.)

An archery avocation got Michael Rosario thinking seriously about killer shrimp.

As an undergraduate integrative biology student at the University of California, Berkeley, Rosario co-founded an archery club. As he won competitions, he also studied with Sheila Patek, an evolutionary biology, biomechanics and animal behavior researcher. The intersection of the two interests has led Rosario, a Department of Energy Computational Science Graduate Fellowship (DOE CSGF) recipient, to apply computers to crustaceans and to a featured role on a National Geographic television program.

Rosario and Patek focus on the mantis shrimp, a group of around 400 species named for their resemblance to true shrimp and to the praying mantis, the garden insect hunter. These denizens of the ocean floor and coral reefs deliver powerful, lightning-fast blows. Their club-like appendages, which also can open to deploy needle-sharp claws, reach top speeds of around 50 miles per hour in less than 3 milliseconds. That’s so powerful the claws pull water molecules away from each other to form a cavitation bubble that delivers an additional shock, emits sound waves and a spark of light, and creates a temperature spike of up to 7,000 degrees centigrade – about as hot as the sun’s surface.

What’s more, a mantis shrimp appendage delivers a blow measured at up to 300 pounds, thousands of times the creature’s body weight. It’s one of the highest peak forces any animal produces.

“I kept looking at this animal and trying to figure out its relationship to how I knew bows and arrows work,” Rosario says. Shrimp claws and bows both store muscle power gradually and release it in a burst. But while each of a modern bow’s parts has a specific duty – acting as a brace or storing elastic energy – “when you look at a mantis shrimp appendage, it’s just the exoskeleton. It’s one structure that has to deal with these competing demands.”

As an undergraduate, Rosario largely focused on experiments testing the exoskeletons of mantis shrimp appendages. By using wire to replace the muscle that loads the appendage’s spring, he could measure the amount of force required as a function of displacement in the claw over time. Rosario found it took 40 to 50 Newtons to fully compress it – or the force necessary to lift between 9 and 11 pounds. “I couldn’t close the appendage with my hands.”

Rosario felt stifled by the experiment’s limitations. “What I was really interested in was elastic energy” – the potential energy stored in a crumpled or stretched or otherwise deformed bendable object, like a retracted bowstring. “There’s no good way to measure elastic energy in these systems.” He wanted to know how a mantis shrimp appendage, as a single structure, handled demands that usually require specialized structures.

Rosario joined Patek as a graduate student after she moved her lab to the University of Massachusetts Amherst. Computational models were the key, he decided, to discovering the mantis shrimp’s elastic energy secrets. In an intensive summer course he participated in Biomesh, a finite element analysis (FEA) workshop that Elizabeth Dumont, another UMass biology professor, had helped create. FEA, more typically used in engineering applications, decomposes a model into a series of simple digital bricks, Rosario says, then computes the physical properties and changes in each. Starting with computer tomography (CT) scans of mantis shrimp appendages, he created models to calculate what parts of the appendage bear the load when it’s compressed to strike – in essence, how and where it stores elastic energy.

Multiscale modeling of arterial blood flow can capture adhesion of red blood cells to the arterial wall, clot formation and other small-scale phenomena while capturing events at macro-scales – for instance, clot-induced changes in flow patterns. This is a still from an animation that illustrates a flow of healthy (red) and diseased (blue) blood cells using a method called Dissipative Particle Dynamics (DPD). Each blood cell is represented by a mesh made of 500 DPD-particles, and small spheres show a subset of the DPD particles representing the blood plasma; instantaneous streamlines and slices represent the ensemble average velocity. (Science: Leopold Grinberg and George Karniadakis, Brown University. Visualization: Joseph A. Insley and Michael E. Papka, Argonne National Laboratory.)

Despite gains in identifying and treating them, the cerebral blood vessel dilations known as aneurysms cause suffering and death for up to 5 percent of Americans. Aneurysms can rupture to start hemorrhages with often rapid catastrophic consequences. Clots formed at the site of an aneurysm may detach, block arteries and trigger a stroke.

Blood supply to the brain relies on a highly complex system where, at any point, an aneurysm may occur. Angiograms can show the presence of aneurysms and clots but don’t necessarily reveal their cause – the interactions among and between platelets and red and white blood cell and the endothelial cells that line blood vessels. When injured, endothelial cells trigger platelet aggregation, leading to a clot.

Conventional imaging also doesn’t show the big picture of blood flow throughout the brain – where blood is coming from and where it’s going. Precisely understanding these elements at the level of both gross blood flow and its microscopic properties would greatly improve neurosurgeons’ ability to predict when and where an aneurysm might rupture and when to operate.

High-performance computing (HPC) shows promise for simulating and visualizing brain blood flow at multiple scales. Paving the way is a team led by Leopold Grinberg, senior research associate in the Division of Applied Mathematics at Brown University. Other researchers include Brown’s George Em Karniadakis, Argonne National Laboratory’s (ANL) Joseph A. Insley, Vitali Morozov, Michael E. Papka and Kalyan Kumaran, and Dmitry Fedosov of the Institute of Complex Systems (ICS) in Jülich, Germany.

In 2006, Grinberg began developing arithmetical and software methods capable of simulating 3-D blood flow in complex arterial networks. “The methodology I started to build was based on functional decomposition, with many tasks assigned to different groups of processors, plus multilevel communicating interfaces capable of connecting the data computed by different tasks and synchronizing the processors assigned to different tasks,” Grinberg says.

Working on three fronts – parallel computing, mathematical algorithms and visualization tools –  the team from Brown and ANL received 50 million processor hours on Intrepid, Argonne’s IBM Blue Gene/P, HPC, and 23 million hours on Jaguar, the Cray XT system at Oak Ridge National Laboratory (ORNL). The allocations were granted through the Department of Energy’s Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program. The researchers also had access to Jugene, ICS’s Blue Gene/P, which runs at a peak speed of 1 petaflops – a quadrillion calculations per second – and is almost twice the size of Intrepid.

As they reported in November at the SC11 high-performance computing conference in Seattle, Grinberg and colleagues have created what they think is the first truly multiscale simulation and visualization of an actual biological system. The team’s ambitious target was brain blood flow, the most complex arteriovenous system in the human body. A paper describing the research was one of five finalists for the prestigious 2011 Gordon Bell Prize, which recognizes outstanding results in the application of parallel computing to practical scientific problems.

(a) Traditional approaches to address volume-change in battery materials use acetylene black as the conductive additive and PVDF polymer as the mechanical binder. (b) Conductive polymer with dual functionality, as a conductor and binder, could keep both the electric and mechanical integrity of the electrode during the battery cycles. (c) PF-type conductive polymers' molecular structure, with two key function groups in PFFOMB (carbonyl and methylbenzoic ester) tailor the conduction band and improve the mechanical binding force. (Click to enlarge schematic, courtesy of Lin-Wang Wang, Lawrence Berkeley National Laboratory.)

Electric cars will remain a tough sell until they can travel beyond 100 miles before needing to recharge their batteries. The battery-life bottleneck has driven the search for technologies that extend the energy storage capacity of lithium-ion batteries.

A collaboration, led by Gao Liu of Lawrence Berkeley National Laboratory’s Environmental Energy Technologies Division and Wanli Yang of Berkeley’s Advanced Light Source, has developed a next-generation battery that could fill the need.

Combining computational modeling and advanced materials synthesis, the Berkeley Lab scientists sought a low-cost anode to provide the needed battery boost. Their solution involves replacing inert graphite with silicon nanoparticles bound to a polymer that absorbs eight times the lithium and becomes electrically conductive as it does. The researchers hope the advance will help power the next generation of electronics and extend the range of electric vehicles.

They key to the advance was making silicon a practical anode material.

Scientists have known for decades that a silicon atom can absorb almost four lithium atoms. But in doing so, it balloons to three times its size. The shrinking and swelling during each round of discharging and charging have made silicon an impractical choice for battery anodes.

Materials scientists have tried a work-around by forming the silicon into nanoparticles and connecting them with an inert polymer binder and a graphite electrical conductor to improve performance. After a few charging and discharging cycles, however, the graphite tends to lose contact with the silicon nanoparticles, reducing its conductivity.

The new-generation polymer acts as both a mechanical binder and an electrical conductor. After testing several variations, the Berkeley Lab group designed a variant of a polyfluorine-based polymer hat worked much better than a previously tested conducting polymer. They hypothesized that a particular modification to the variant polymer – the addition of a carbonyl group (a double-bonded carbon-oxygen group) – contributed to its unique electronic properties. Indeed, the researchers used Berkley’s Advanced Light Source to produce an X-ray absorption spectrum that showed there is an additional peak below the electron conduction band. But that didn’t explain how the carbonyl group and an associated additional X-ray absorption peak were related to the enhanced battery performance.

An analysis of aerosol particle and cloud compositions at various altitudes demonstrates the power of so-called n-dimensional computation, which gives researchers information about the role of aerosols on global and local climate change. The parallel coordinate display allows users to view and filter the composition of the particles and clouds; the linked Google Earth display enables a spatial co-reference. Users also may select any sample point or group of points along the flight path to view compositions in the parallel coordinate display. In addition, users may enlist pie charts to summarize the data at that location or area. The dataset was acquired by a single particle mass spectrometer (SPLAT II) on Flight 26 (F26) on April 19-20, 2008 as part of the Indirect and Semi-Direct Aerosol Campaign (ISDAC), a month-long field campaign at the North Slope of Alaska. (Click on image for larger, detailed view.)

To give audiences a better feel for life on a fictional planet, movie director James Cameron made the film Avatar in 3-D and moviegoers donned 3-D glasses. For scientists trying to get a better grasp on what drives climate on Earth, 3-D isn’t good enough. Instead, they’re turning to new techniques to help them see the details and interconnections that ultimately shape the big picture.

“I’m interested in high-dimensional visualization, which is an extension from 3-D visualization into n-dimensional visualization,” says Klaus Mueller, director of the visual analytics and imaging lab in the Center for Visual Computing at Stony Brook University. “Three dimensions is a somewhat solved problem in computer visualizations; n-dimensions is the new frontier. There are all kinds of problems in terms of the user interface and how to make people understand what it means.”

A traditional graph with an x-and-y axis is a two-dimensional visualization, showing the relationship between two variables, or dimensions. An n-dimensional visualization involves showing the relationships between four or more variables.

Mueller is collaborating with the Brookhaven National Lab-led FASTER (FAst-physics System TEstbed and Research) project to develop novel multi-dimensional visualization techniques and tools to help the FASTER researchers see how fine-level, fast physics processes, such as local aerosols and raindrops, shape global climate patterns.

For Mueller, the bigger question that connects all his research is how to effectively convey multi-dimensional information. He can’t give researchers the equivalent of n-dimensional glasses, so instead turns to how we see and interpret information.

We’re familiar with seeing four dimensions – 3-D plus time – and how to visually represent this – motion blur, for example, to show movement. Five dimensions and beyond, however, is a poorly explored territory in visualization yet one that’s crucial for dealing with complex simulations, such as climate. For example, in the FASTER models there are a dozen variables, or dimensions, that are linked to changes in atmospheric pressure.

One thing that’s clear: There’s a learning process in moving to n-dimensional viewing, Mueller says. With two-dimensional imaging sensors – our retinas – we’re able to see in 3-D. In fact, we teach ourselves to develop this fine-tuned spatial sense. When babies reach out to touch objects around them, they’re calibrating their depth and 3-D geometry perception.

“People are familiar with bar charts and pie charts and scatter plots for the display of two- and three-dimensional information,” says Mueller, who’s also an adjunct scientist with BNL’s Computational Science Center. “Anything else is a challenge for many people. This is the challenge for n-dimensional information visualization – to bring this to the masses.”

FASTER and better

The FASTER project is an ideal test bed for these new high-dimensional visualization tools, Mueller says. FASTER is itself an effort to explore how the computational modeling of rapid, small-scale fast physics processes, such as precipitation and cloud dynamics, fits into and shapes global climate models. It’s believed that errors in modeling these fast physics processes are responsible for major uncertainties in global climate model predictions.

“Virtually all of the fast-physics processes interact,” says Yangang Liu, the Brookhaven atmospheric scientist leading FASTER, “so once we get a handle on the individual processes, we want to see how they interact and how to evaluate these interactions. Data integration and visualization are an essential part of this analysis.”

Working with FASTER scientists and Stony Brook doctoral student Zhiyuan Zhang, Mueller has created a unique visualization system that links sophisticated multidimensional information displays with geographical context.

At DOE’s Pacific Northwest National Laboratory, Zelenyuk specializes in using single-particle mass spectrometry to analyze the real-time transformations of nanoparticles. This includes atmospheric particles, such as aerosols, crucial to determining climate. Her experimental runs produce a jungle of spectral data in 450 dimensions for millions of particles.

Automated methods to analyze data with multiple variables often fail when the number of variables exceeds a dozen, Zelenyuk says. “So with 450-dimensional spectral data we needed new tools for visualizing and analyzing our data.”

Mueller, Zelenyuk and collaborators developed a two-part interactive data mining and visual analytics software package. SpectraMiner creates a unique hierarchical dynamical tree or cluster dendogram that can incorporate hundreds of clusters. Data then can be exported to ClusterSculptor so scientist can tune and explore parameters in search of important relationships.

“At each step the scientist is in control of the level of detail and the visualization format,” Mueller says, noting that the visualization tools are now used daily. “This allows them to refine, steer and control the data-mining process.”

A visualization of a lean hydrogen flame simulation shows three computed fields simultaneously. A bowl-shaped turbulent flame floats over the exit flow from a pipe that is swirling as it moves upward. The gray filaments at the bottom depict regions of high turbulence, the transparent red surface highlights the mixing region between the fuel from the pipe and the air outside, and the purple-to-red zone shows the concentration of nitrogen-based emissions from the flame.

With help from a Cray supercomputer, Lawrence Berkeley National Laboratory (LBNL) researchers John Bell and Marcus Day are studying the combustion properties of hydrogen – a potential fuel that science has been trying to tame for decades.

“Hydrogen-based fuels are going to play a key role in low-emissions energy sources of the future,” explains Day, a staff scientist in LBNL’s Center for Computational Sciences and Engineering. To design devices that can use hydrogen-based fuels, “we need to understand much more than we do now about how these flames respond to intense turbulence and high pressures. Both are fundamental features of practical power-generation combustors, but both have significant and complicated effects on the hydrogen flame.”

To understand those effects normally requires experimentation. But experiments aren’t sufficient here because flames in those conditions are extremely hard to measure accurately – and the generated data defies easy interpretation.

“Experiments with real flames are in a relatively hostile environment – it is very hot – and you need to not disturb the flame when you take measurements,” says Bell, an LBNL senior staff scientist. And “many of the things you want to look at are chemical species that live only for a short time in the actual flame zone. They can’t be measured directly.”

Although computer-simulated flames obviously don’t take place in a combustor and are not subject to Mother Nature’s unpredictability, they nevertheless are quite difficult to produce and study. Experimental-quality combustion simulations are complex and require loads of computer time. Bell and Day have access to 40 million processor hours on Jaguar, based at Oak Ridge National Laboratory, through the Department of Energy’s Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program.

Running the simulations is only part of the challenge. The huge volumes of data that result must be compared to experimental data and further explored to understand the new information that the data contain.

“Simulations don’t match experiments exactly,” Bell says, “but they do provide useful insights into how the flames are behaving.” The data help paint a more complete picture of the combustion system – a picture that is simply inaccessible by any other means. As Bell and Day figure out how to quantify uncertainties in the models they use to describe these systems, they can improve predictions and design more practical systems.

What’s more, Bell says, “we hope our work will open the door to an increasingly rich set of activities that couple computational, theoretical and experimental components of combustion science.”

Stabilizing the flame

Unlike fossil fuels that pump carbon dioxide and other pollutants into the air, hydrogen is virtually waste-free. It’s also a basic component of Earth’s atmosphere. Hydrogen is oxygen’s partner in forming water ­– and, it turns out, flames.

In an idealized setting oxygen and hydrogen burn together with near-zero emissions but only if the mix is sufficiently lean, meaning a low fuel-to-oxygen ratio. However, at these leaner mixtures, the flame becomes highly unstable and difficult to use in practical scenarios. High-pressure conditions tend to exacerbate the difficulties.

The same physics that makes hydrogen flames hard to study in the lab lead to difficulties on the computer. The flames blow off or flash back into the fuel supply. Strong turbulence can also have a huge impact – the flames will be twisted, violently shredded and possibly extinguished.

The tiny white yeast colonies in the right panel interspersed with larger normal colonies are cells that have had a synthetic chromosome inserted and their DNA shuffled by the lab-induced SCRaMbLE system, which introduces changes that slow cell growth. By comparison, all colonies on the left are grown from the standard lab yeast strain and appear uniform. (Click on image to enlarge.)

Last year the J. Craig Venter Institute made waves by creating the first fully synthetic bacterial genome. Now a group from Johns Hopkins University has extended that work to yeast, producing a built-from-scratch chromosome that works just like the natural chromosome it replaced.

The project, described today in the advance online issue of the journal Nature, is the first step in creating a modular, synthetic organism that its makers hope will act as a biological factory for churning out medicines or substances that break down toxic waste.

Led by biologists Joel Bader, Jef Boeke and Srinivasan Chandrasegaran, the team relied on the computational skills of Sarah Richardson, a graduate student in Bader’s lab and an alumna of the Department of Energy Computational Science Graduate Fellowship (DOE CSGF), to design the chromosome and to oversee its construction. Software assisted biologists in fashioning a chromosome containing millions of individual DNA bases and thousands of functioning genes in a highly structured, modular system.

“Probably the most computationally difficult algorithm is the segmentation of a chromosome into assemble-able bits,” Richardson says. To manage such a data-intensive project, Richardson searched for programs she could modify for her team’s task. She spoke to geneticists who wrote widely used gene annotation software but quickly discovered that the tools, which ensure genes are correctly sequenced and labeled, fell flat at breaking the chromosome down and moving genes around.

“The biggest problem was that there are not (publicly available) algorithms to edit chromosomes or genomes,” Richardson says. “So I set out to write those algorithms and create that framework for editing sequence on a large scale.”

The result was a software suite called BioStudio and an associated program called GeneDesign. Together, the software assists in designing genetic constructs and tracking the progress of synthesis and assembly. Richardson specifically designed the programs to be as generic and user friendly as possible. She wove in touches adapted from open-source packages, such as a collaborative wiki-like interface with revision-control systems and color-coding graphics to assist editing tasks.

In yeast, all the essential genes – ones the organism can’t survive without – are known. With that information in place, the visualization software colors all the essential genes red. “The red flag on essential genes,” she says, “really lets you know if you are editing a particular gene, you are potentially affecting the fitness of the yeast.” All genes with known functions follow the color-coding system, enabling the designers to monitor changes they are making.

Although the computer can automate many tasks, deciding which genes to move around requires a scientist’s experienced eye for subtle detail.

“It turns out it is pretty hard for the computer to decide what stays and what goes” in the genome design, she says. “First you need to know what you want. Then you can apply the algorithms.”

As part of her Department of Energy Computational Science Graduate Fellowship, Richardson served a 2009 practicum under Adam Arkin, director of Lawrence Berkeley National Laboratory’s Physical Biosciences Division. She made important contributions to Arkin’s research into an RNA-based transcription attenuator found in Staphylococcus aureus.

Attenuators are regulatory sequences that halt gene transcription in bacteria and other prokaryotes. The researchers want to make it a standardized tool for fine-grained gene expression control.

“This RNA-regulated attenuator provides an opportunity to, for one, engineer it for better function so it really is an off switch and has a large dynamic range,” Arkin says. Because it blocks transcription, an engineered attenuator might be built into a family of parts operating similarly but orthogonally – without interfering with other functions – in cells, Arkin says.

Attenuators have two pieces, like a lock and key, Richardson says. Arkin’s group wants to modify them so a specific lock works only with a particular key. Two or more could be inserted in the same transcript, working together much like a logic circuit on a computer chip.

Richardson’s practicum focused on improving and augmenting existing computer code to generate new attenuators orthogonal to wild type S. aureus. She learned the Python programming language and wrote an interface between the group’s existing code and the Vienna RNA Package – software designed to predict and compare RNA secondary structures. The bridge made it possible to interchange Vienna with mFold, another RNA secondary structure prediction package, and to compare the two programs’ predictions.

The group’s code starts with the wild-type lock and key and then mutates it. Richardson devised an algorithm to choose a large set of mutually orthogonal or nonorthogonal attenuators from the mutants. She then worked on a cloning technique and lab protocol to synthesize and evaluate proposed attenuators.

Months after Richardson finished her practicum, Arkin’s group was still using software she helped create. The group also improved and automated her lab protocol.

A frame from an animation showing the possible route into the Atlantic Ocean of oil and dispersant from the spot of the Deepwater Horizon spill in the Gulf of Mexico.

Within days of the Deepwater Horizon oil well blowout on April 20, 2010, Los Alamos National Laboratory (LANL) oceanographer Mathew Maltrud, working at the New Mexico Supercomputing Applications Center, ran a simulation of the spilled oil and sent a visualization to collaborator Synte Peacock of the National Center for Atmospheric Research (NCAR).

Maltrud and Peacock had been studying – and continue to study – ocean dynamics and the relationship between the ocean, climate, the environment and Earth’s atmosphere. All of the gushing oil was at once fascinating and fearful, and it piqued their curiosity.

“Even very experienced ocean modelers looked at the speed of the ocean currents in the simulations and thought, ‘Man, we forget how fast the ocean can move sometimes.’” says Maltrud. “Most of us are working on climate time scales; we’re not used to thinking in terms of weeks to months. It was very impressive in terms of speed.”

About the same time, the two heard from Peacock’s former Ph.D. adviser and occasional collaborator Martin Visbeck, who lives in Germany. Visbeck mentioned that a low-grade panic was sweeping parts of Europe, where citizens began to fear the oil might someday pollute their coasts. With visions of the Alaskan Exxon Valdez spill still fresh after 20 years, many were wondering when Gulf oil might reach their continent and how diluted might it be. Visbeck told his German colleagues of his friends who had a global model that might provide some answers. Suddenly, real-time events were driving science at a frantic pace.

The U.S. Department of Energy allotted Maltrud generous computing hours on the Jaguar Cray XT system at Oak Ridge National Laboratory’s Leadership Computing Facility. He began exhaustive weeks programming a series of ensembles, each a varied group of simulations in which a certain parameter is changed. In this case, the only variant was  the ocean’s initial state. The researchers sought to answer a riveting question: What were the chances that oil, exuding at a rate ranging from an estimated 1,000 barrels a day in April to upward of 62,000 barrels a day in August, would escape the Gulf and lurch up the Atlantic Seaboard and beyond to the coasts of Europe?

Meanwhile, other models were capturing world attention because they used real-time data and monitored the situation the way weather is forecast. Businesses, tourists, governments, environmentalists – all wanted to know the spill’s effect on local beaches, on ocean flora and fauna, and on seafood safety.

“We sought a different approach,” Maltrud says. “We didn’t try to make predictions, but tried instead to understand statistical distributions of what’s possible because that’s the kind of thing that our model can do. Those other models tried to represent exactly what was going on in the ocean at a given time, so we figured that we would try to shed some light on how long it might take oil to leave the Gulf, where it might go, and in what relative concentrations in relation to what was being released at the spill site.

The model Maltrud ran is called POP (Parallel Ocean Program). He made major contributions to its development at LANL in the ’90s; it is the ocean component of NCAR’s Community Climate System Model.

The crux of the modeling problem was a Gulf feature known as the Loop Current, a complex “clockwise surface circulation” entering the Gulf through the Yucatan Channel, and exiting in the Florida Current through the Florida Straits. Maltrud calls the Loop Current the “big meander” because its behavior is akin to a river on a flood plain: now and then changing course, creating oxbow lakes and mini-currents. In the Gulf, these small currents are eddies, subsets of the major loop circulation. The eddies are complex because there may be two or three going at any given time, each broken off from the Loop Current at its own interval and moving in its own pattern.

These tracers include carbon and radiocarbon isotopes, paleotracers (fossils from the sea, in sediments and shells) and theoretical tracers called transit time distributions, which get at how long it’s been since water at the ocean’s interior was lost to the surface – useful for tracking the behavior of greenhouse gases carbon dioxide and methane and the long-banned chlorofluorcarbons (CFCs).

Add passive dye to that list of tracers. In fact, a key reason she and her Los Alamos National Laboratory collaborator Mathew Maltrud could start simulations so quickly during the Deepwater Horizon incident was that passive dye already was part of their modeling repertoire and the code had been worked out for previous simulations.

“It was easy to set this model up,” Peacock says, “like flipping a switch and letting the model go. You specify the latitude and longitude for where you want to inject (the dye), the depth, and how long you want to inject for.”

Earlier in 2010, Peacock and Maltrud had reported on their simulation of CFCs and numerous other tracers in a 100-year ocean circulation model. The model’s extreme high resolution of .1 degree let the researchers follow the movements of eddies, which previous CFC distribution studies characterized poorly because their resolutions were coarse in comparison. Though the eddying model did not directly lead to the Deepwater Horizon incident study, it laid the groundwork for the researchers to act and implement similar theory.

The model the researchers ran is considered to be one of the most realistic global fine-resolution eddying models, and the only one to simulate such a large set of tracer distributions, thanks to the power of the Jaguar supercomputer at Oak Ridge National Laboratory.

CFC modeling is vital in understanding how the ocean can store chemicals for up to thousands of years and globally circulate them at various depths from the surface to hundreds of meters down, a process called ventilation that has multiple effects on climate.

Maltrud wants “to know how gases get transferred from the surface of the ocean down into the depths, and CFCs are a really good way to do this.”

Adds Peacock: “It was the first time that CFCs had been carried for many decades in a model of one-tenth degree resolution. So, the grid was small enough that we could resolve these eddies.”

Peacock refers to eddies as “the weather of the ocean. They happen on small spatial and time scales, are about 1 to 10 kilometers in diameter, with circular features that rotate. You get very large temperature gradients in relationship with surrounding water and very different properties within the core of an eddy. They’re important in transporting, for example, heat and mixing properties in the ocean. One of the reasons we run the high resolution models is to see how accurate we are in parameterizing them.”

CFCs are directly measured in real-time on ocean-going ships with devices called CTD sensors (for conductivity, temperature, depth) from water samples captured in bottles dunked to various depths. Peacock says many measurements have been made over the past 20 years. She and Maltrud used the real-time measurements to test their simulated CFC distributions.

“We didn’t use the real-time observations to push the model in any way, just for the validation,” Peacock says. “This was a very powerful tool that gives us a snapshot of what is going on and clues to how the ocean is being ventilated over the past couple of decades.”

Describing all of the nuclei and the reactions between them, however, demands powerful algorithms running on high-performance computers.

The Universal Nuclear Energy Density Functional (UNEDF) collaboration, which was created by the Department of Energy’s Scientific Discovery through Advanced Computing (SciDAC) program, focuses on developing such descriptions.

An optimized sequence of parameter values in nuclear simulations. (Image courtesy of Stefan Wild.)

The UNEDF collaboration includes researchers from seven national laboratories – Ames, Argonne, Lawrence Berkeley, Lawrence Livermore, Los Alamos, Oak Ridge, and Pacific Northwest – and nine universities: Central Michigan, Iowa State, Michigan State, Ohio State, San Diego State, North Carolina at Chapel Hill, Tennessee-Knoxville, Texas A&M in Commerce and University of Washington. Recently, researchers in this collaboration made a significant advance through the use of density functional theory (DFT).

On Earth, only about 300 kinds of nuclei – specific combinations of protons and neutrons – exist. In accelerators and stars, the number of known nuclei grows to about 3,000, and it could eventually expand to around 6,000. Many of these tiny systems prove extremely difficult to study, largely because they live such short lives before decaying.

Consequently, researchers need ways to accurately simulate these elusive species. Other applications also require extremely precise simulations of interacting nuclei. For example, the National Nuclear Security Administration (NNSA) Stockpile Stewardship Program requires such simulations to assess the safety and functionality of the weapons in the U.S. nuclear stockpile.

Witold Nazarewicz, professor of physics at the University of Tennessee and co-director of UNEDF, describes the basic structure: “A nucleus resembles a droplet of liquid, where there’s a high density inside and a surface area where it drops, and there’s little outside.” Moreover, the quantum behavior of the protons and neutrons at that surface determines the energy of the nucleus and how it interacts with other nuclei. “We need to know how the nuclear energy is generated in a nucleus to use it.”

Talented teamwork

DFT provides an extremely useful, but not necessarily easy, approach to modeling nuclei. For one thing, DFT includes many parameters that must be determined. As Stefan Wild, assistant computational mathematician in the Laboratory for Advanced Numerical Simulations at  Argonne and a fellow in the Computation Institute at the University of Chicago, asks, “What are the best parameters to calibrate these new models to experimental data?”

In the past, scientists searched for the best parameters with what Wild, an alumnus of DOE’s Computational Science Graduate Fellowship, calls “a lot of hand-tuning. They used intuition to pick the values of parameters, ran a simulation, saw how close the answer came to observed data, made small adjustments and ran the simulation again.”

Given the increasing complexity of nuclear simulations, however, “hand-tuning was like looking for a needle in a haystack and far too time consuming to do anything rigorous or thorough.”

As high-performance computing grew more powerful, though, Wild says that “people started thinking about doing something more mathematical” with DFT. For example, Wild and Jorge Moré, an Argonne Distinguished Fellow and director of Argonne’s Laboratory for Advanced Numerical Simulations, developed an algorithm and computer code called POUNDERS (for “practical optimization using no derivatives for sums of squares”).

“If we can determine those densities precisely,” says Witold Nazarewicz, professor of physics at the University of Tennessee, “we can determine the binding energy – the energy stored in the nucleus.”

The energy density functional (EDF) in DFT is an integral of a function of those particle densities. The corresponding energy density is composed of proton and neutron densities, spins, momentum and more. Such an EDF includes variables, or coupling constants, that must be adjusted. The goal of the Universal Nuclear Energy Density Functional (UNEDF) collaboration is to find a universal EDF that works across the entire nuclear landscape, or chart of nuclides, which is a two-dimensional table that collects all nuclei represented by their proton and neutron numbers. Building that requires simulating the properties of thousands of nuclei and doing so repeatedly.

The first step in using DFT to model a nucleus is finding the densities that minimize the energy in a nucleus. This creates an optimization problem that can be tackled by solving the so-called Hartree–Fock–Bogoliubov (HFB) equations of the nuclear DFT.

For the coupling constants, says Nazarewicz, “some of those – but very few – are basically given by theory. Most of them have to be found by comparing DFT calculations with experiments.”

The results of the optimization experiments and starting values for coupling constants can be used to generate nuclear features, such as binding energies, radii, shape deformations and others.

The UNEDF team started with about a dozen coupling constants and used them in HFB equations to calculate more than 100 features of nuclei.

“We compare those observables with experiment and design a chi-square to see if the results are good or not,” Nazarewicz says. “Then we try to adjust the parameters so the chi-square becomes minimum. So this constitutes a huge optimization-minimization problem.”

Fortunately, some of the parameters cannot be varied broadly, because physical bounds limit some of them. That provides some small simplifications to this process.

To make this technique work, Nazarewicz and his colleagues came up with the form of the functional and selected the experimental fit-observables to use.

“We provided our codes to the Argonne group,” Nazarewicz says, “and they designed an optimized procedure that minimized the chi-square. That is POUNDERS (for “practical optimization using no derivatives for sums of squares”), which is a tremendous algorithm because it saved orders of magnitude of time.”

Without POUNDERS, the chi-square converged on a minimum so slowly that it would have taken hundreds of iterations. That takes too long when the minimization involves what Nazarewicz describes as “a problem that is highly nonlinear for more than 100 observables and a dozen or so coupling constants.”

With POUNDERS, DFT can be applied to more nuclei and other applications of this technique will surely keep emerging.