National Center for Computational Sciences

Modeling Volcanic Eruptions Mimics a Stressed Climate

Scientists simulate climate response to volcanic gas emissions to test model’s accuracy

The 1991 eruption of Mount Pinatubo in the Philippines spewed 20 million tons of sulfur dioxide gas into the atmosphere, the bulk of it in only 9 hours. The sulfur dioxide circulated around the globe in about 2 weeks, interacting with oxygen molecules along the way to become a sulfate aerosol that would remain in the atmosphere for several years.

Scientists at Oak Ridge National Laboratory (ORNL) and the National Center for Atmospheric Research (NCAR) are using the Cray XT5 supercomputer, Jaguar, at the Oak Ridge Leadership Computing Facility (OLCF) to simulate how the climate system reacts to the atmospheric increase in aerosols from volcanic eruptions.

Eruption aerosols circulate higher in the atmosphere than gases released by power production and other human activities. Solar radiation bounces off the aerosol particles, directing some of the radiation away from the earth and causing short-term cooling. In contrast, anthropogenic, or human-produced, gases in the lower atmosphere trap solar radiation, warming the planet’s surface over longer periods of time. Volcanic emissions can mask the long-term warming caused by human activity.

Using Jaguar, the fastest supercomputer for open science research, the team will be able to follow the impact of aerosols on smaller scales than was possible in the past. It will monitor timescales in decades, not centuries, and space scales in geographical regions, not the entire globe.

“For the first time we’ll actually be able to include more detailed information about changes in atmospheric aerosols and run the atmospheric model at a 40-kilometer resolution to assess the impact,” said Kate Evans, principal investigator on this project to predict climate change on decadal timescales. The project began running on Jaguar in January as one of 21 petascale early science projects, or large-scale endeavors given early access to use most of the supercomputer’s processing cores.

The sudden influx of aerosols into the atmosphere following a volcanic eruption makes a bold statement relative to other factors that influence climate.

“It’s analogous to plucking a guitar string against a background of silence,” Evans said. “The volcano is perturbing the system on a new frequency.”

Aerosols from volcanic eruptions typically clear away within 2 to 3 years as gravity pulls the aerosol particles back down. Conversely, anthropogenic emissions are ongoing and may alter climate long into the future.

Although several types of gases, including carbon dioxide, are emitted during a volcanic eruption, the release of sulfur dioxide and the resulting sulfate aerosol are especially important because the sulfate particles scatter incoming solar radiation. With less energy reaching the ground, the average global temperature can decrease up to a degree Fahrenheit. This short-term cooling contrasts with long-term global warming, predicted to be 3 to 10 degrees Fahrenheit over the next century.

Global models cannot yet predict climate change based on escalating greenhouse gases for regional scales. Scientists confidently use current climate models to predict climate change as a global average, but they are still exploring whether the models can accurately predict the unique impacts of climate change from region to region. Where one region may experience drought, another may experience rising water levels, depending on many interconnected variables of climate change.

Looking back to leap forward

Evans and her team are using the OLCF supercomputer to simulate more than 30 years of global climate, from 1978 to the present. The team includes five other researchers at ORNL—James Hack, leader of ORNL’s Climate Change Initiative; John Drake, leader of ORNL’s contribution to the Climate Science Computational End Station, which will play a role in the upcoming Intergovernmental Panel on Climate Change’s fifth assessment report; James Rosinski, a computational scientist who designed the model runs; Pat Worley, a computer scientist who will help the team scale the climate modeling codes to make the best use of the supercomputer; and Lianhong Gu, a climate scientist in the Environmental Sciences Division who will study atmosphere interactions with land and ecosystems. The team also includes three researchers at NCAR—climate scientist Julie Caron, computational scientist John Truesdale, and atmospheric chemist Jean-Francois Lamarque.

Simulating global climate requires enormous processing power to calculate temperature, pressure, and other variables for millions of points on the globe every 150 seconds for more than a quarter century. The Jaguar XT5’s 1.4-petaflop processing speed allows more than a quadrillion calculations per second, enabling the project’s researchers to achieve the planned level of detail.

The Community Climate System Model considers four key components of the earth’s climate system: atmosphere, ocean, land surface, and ice. Using a version of the Community Atmosphere Model computational code developed at NCAR and run on Jaguar, Evan’s team is configuring the atmosphere and land components to run at a resolution ten to 40 times the spatial degrees of freedom typical of most global climate models.

The team must test the model for accuracy at the finer resolution by running it through a period of recent history. Observational data from weather balloons and satellites are compared to data generated by the model. When observations can confirm the results of computer modeling, scientists can better determine the model’s ability to predict climate response.

The project is slated to use more than 8,000 of the supercomputer’s processors for a single run, Rosinski said. Four times that number may be required as multiple runs are carried out. The goal of running the model several times is to build a collection of datasets of slightly different results that represent climate variability. Once averaged, the collection of data will yield a more accurate prediction.

“If the model has the ability to predict the system response to changes in the aerosols,” Hack said, ”then we will have increased our confidence in being able to detect the response to longer-term anthropogenic changes.”

Sequestration Puts Carbon Dioxide Underground

Supercomputer simulations explore possibility of storage in saline aquifers

Supercomputing meets sequestration in an experiment that could determine if carbon sequestration, removing carbon dioxide from the air and storing it underground, is an effective way to keep this greenhouse gas out of the atmosphere.

A team led by Peter Lichtner of Los Alamos National Laboratory will attempt to understand long-term carbon sequestration using the National Center for Computational Science’s Cray supercomputer, Jaguar, at ORNL. The 12 million processor hours allotted by the Department of Energy for Lichtner’s project in 2009 will help shed light on the risks and rewards of carbon sequestration.

“We are looking at disposing of carbon dioxide deep underground and how it will change over time,” Lichtner said. “By modeling the underground chamber and the surrounding rock, including abandoned wells, we can identify where any problems or leaks will be.”

Out of sight deep underground

Coal-burning power plants are the main source of carbon emissions in the United States, releasing more than 2 billion tons into the atmosphere every year, where they remain for up to 200 years. However, we can’t just quit using coal. It supplies more than 50 percent of the nation’s electricity.

Carbon sequestration could capture the carbon emissions of these power plants and put them underground so they never reach the air we breathe. Carbon dioxide is captured from coal-burning plants by placing a scrubber on the flue from which the gases would be released. The scrubber absorbs the carbon dioxide, which is then transported in pipes to carefully chosen—ideally nearby—sites and injected into storage containers.

Supercritical carbon acts as a gas, expanding to fill its container. But it also has the density of a liquid. In this supercritical phase, carbon dioxide is injected into the ground much like oil is pumped out of the ground, using wells drilled to great depths.

Lichtner’s project will focus on a saline aquifer as the storage container for carbon dioxide. Aquifers are underground water reservoirs and ideal for this sequestration because they commonly occur in nature, can hold a large volume of liquid, and have a layer of mineralized brine. Once injected into the saline aquifer, the supercritical carbon dioxide begins to absorb the brine and sink to the bottom of the chamber and away from the most likely leak points, which are the surface and nearby abandoned wells.

As the brine absorbs the supercritical carbon dioxide, the brine becomes heavier and more acidic and starts to sink, resulting in a convection current with “fingers” of sinking brine. In the simulation Lichtner will explore this poorly understood event, which is thought to speed up the dissolution of carbon dioxide and play a role in its ultimate fate.

Understanding the dissolution speed is important to estimating the risk of leakage, as well as how the increasing acidity of the brine affects the walls of the aquifer. The faster the carbon dioxide leaves the supercritical phase (dissolves) and sinks, the less chance of a dangerous leak that could pollute the groundwater flowing around the aquifer or escape into the atmosphere.

The magnitude of this study is such that researchers have had to wait for supercomputing to reach the power needed for their simulations. Looking at square-kilometer chunks of land is no small feat. Now the Cray XT Jaguar, the fastest supercomputer for open scientific research, can run 1.64 quadrillion calculations per second, enabling the research team, which consists of Lichtner, Glenn Hammond of Pacific Northwest National Laboratory, and Richard Mills of ORNL, to study carbon storage and capture in greater detail than ever.

Out of mind for a few millennia

Carbon sequestration is a new field of research, and no computer code exists that can predict exactly what will happen when carbon dioxide is injected into the ground. The encompassing goal of this project is to improve the PFLOTRAN code, which will eventually reveal the behavior of underground flow processes.

Until now the vast range of scales for both length and time has limited computer simulations of subsurface processes. Now, with more advance computer technology and PFLOTRAN, it is possible to explore lengths ranging from chemical reactions inside the aquifer that are less than one centimeter long up to the size of the aquifer itself—on the order of kilometers. Times range from relatively fast chemical reactions that take days up to thousands of years.

Lichtner’s team will focus its calculations on the Scurry Area Canyon Reef Operators Committee (SACROC) unit of the Permian Basin in western Texas for its calculations. This 8,000-cubic-kilometer aquifer is six to seven thousand feet below the earth’s surface and contains more than 200 oil wells. Carbon dioxide flooding operations began here in 1972, making SACROC the oldest demonstration of carbon sequestration in the United States. Since then 55 million tons of carbon dioxide have been sequestered.

Using the PFLOTRAN code, the team can simulate this huge geologic formation and how the carbon dioxide has changed since it was injected. The computer power required for this project is enormous given the range of scales and complex processes. And Jaguar now offers the power to model subsurface processes with unprecedented detail, enabling researchers to improve the PFLOTRAN code for future experiments.

This research brings the idea of carbon sequestration from strictly scientific study to near-future application as a means of keeping carbon dioxide out of the atmosphere. Trapped kilometers underground, carbon dioxide will do far less harm to the planet than if it was released to the atmosphere.

“Once we understand how supercritical carbon dioxide behaves underground, sequestration can definitely be a useful method for reducing greenhouse gases in the atmosphere,” Lichtner said.

CUG 09 Stresses Future Science on Giant Petascale Platform

NCCS hosts, provides speakers, unveils Jaguar

More than 150 supercomputer users attended “Compute the Future,” the 51st Cray User Group meeting, an international body of member organizations that own, operate and use Cray HPC systems, and support the use of Cray resources to achieve research goals. CUG ’09 was held May 4-7 in Atlanta, GA, and was hosted this year by the National Center for Computational Sciences (NCCS) and the National Institute for Computational Sciences (NICS) at ORNL.

The theme—“Compute the Future”—underlines that high performance computing is not an end in itself, but a very powerful tool to solve the challenging science questions of the day. With the launch last November of the DOE’s Cray XT4/XT5 1.64 petaflop supercomputer Jaguar at ORNL, followed closely by the National Science Foundation’s Kraken XT5 at NICS, scientists now have computer platforms of unprecedented size to answer many of their most enduring and complex research questions.

James Hack, director of the National Center for Computational Sciences and an atmospheric scientist who leads ORNL’s Climate Change Initiative, gave the keynote address. Dr. Hack explained why computational innovation is so important to accurately predicting the coupled chemical, biogeological, and physical evolution of today’s global climate system. “Meeting the challenges that climate change pose will require qualitatively different levels of scientific understanding, numerical modeling capabilities, and computational infrastructure, than have been historically available to the climate community,” he said.

Arthur (Buddy) Bland, leader of the OLCF, who supervised the installation of the Cray XT4/XT5 1.64 petaflop Jaguar supercomputer at ORNL, introduced the world’s most powerful computer for open science to the assembly, and described several applications now being used with speeds exceeding the 1 petaflop mark. “Today Jaguar is running a broad range of time-critical applications of national importance in such fields as energy assurance, climate modeling, superconducting materials, bio-energy, chemistry, combustion, and astrophysics,” Bland said.

In a second keynote address, Thomas Schulthess, newly-appointed director of the National Supercomputing Center of Switzerland, described algorithmic and computational advances on the new Cray XT5 Jaguar at ORNL. Schulthess, who studies superconductivity in nanophase materials, described applications which have made it possible to use 150,000 of Jaguar’s 180,000 core processors, at a sustained performance of 1.35 petaflops—an achievement which won him and his collaborators the Gordon Bell prize for best application performance at Supercomputing 08.

Cray Inc. chief executive and president Peter Ungaro held a ‘Hundred on One session,’ at which users could share questions and concerns in the absence of Cray management and staffers. Talks at CUG ’09 encompassed all aspects of the Cray system, from keeping it running effectively to taking full advantage of the giant architecture, to advanced software innovations. Researchers from ORNL gave a total of 21 presentations at the conference, many of them to share early research experiences on the new machines. Richard Mills of ORNL and his collaborators described the experience and the challenges of scaling PFLOTRAN on Jaguar at NCCS. PFLOTRAN is a PETSc-based code for modeling the flow of reactive contaminants deep in mineral beds. Richard Loft and climate colleagues from NCAR presented performance and scaling data for climate simulations on Kraken’s Cray XT4 and XT5 at NICS.

“Part of the program was Cray presenting its roadmap for future systems they are going to offer, and discussing the current software and hardware,” said James B. (Trey) White III, leader of the User Assistance Group at NCCS and local arrangements chair, who with Sherry Hempfling of NCCS organized the conference. “Part of the draw of the meeting is for the technical people at Cray Inc to meet with the technical users and the people who run their systems. It’s a mix of people from the computing centers, the users, Cray employees and the vendors who are associated with Cray supercomputers, like AMD Processors and others who do software for Crays.”

Student volunteers from Morehouse College, an all-male, black college in Atlanta, assisted with registration and were invited to attend several conference sessions.

Award Allows Researcher Access to ORNL Supercomputer

World’s fastest computer for open science to explore nanodevice technology

A researcher at Southern Illinois University Carbondale will have access to some of the most powerful computers in the world as he joins a team of scientists working on a variety of issues. Shaikh Ahmed, an assistant professor in the Department of Computer and Electrical Engineering in the College of Engineering, is one of just four researchers nationwide included in Oak Ridge National Laboratory’s first High-Performance Computing Grants Competition. As part of his successful application for the program, which was open to universities associated with Oak Ridge, Ahmed will use the ORNL’s Jaguar supercomputer housed at the site.

Ahmed will use the massive computing power to conduct harsh-environment nanodevice technology research. The $25,000 grant covers one year of study and holds the possibility of up to two more years of funding for a total of $75,000 provided by the Oak Ridge Associated Universities group. The association is a consortium aimed at pooling resources of 100 major research institutions to advance science and education in partnership with national laboratories