National Center for Computational Sciences

Researchers Seek lingua franca So Fusion Codes Can Converse In Coupled Model

Projects explore radio frequency waves and magnetohydrodynamics in fusion plasmas

In the Bible, all people spoke the same language until God felled the Tower of Babel and produced diverse languages that prevented people from understanding each other. Today physicist Don Batchelor at Oak Ridge National Laboratory (ORNL) is trying to solve a Babel-esque problem by creating a common language and a computational framework that will allow diverse software codes to communicate with each other in simulations of plasma—the hot, ionized gas that fuels nuclear fusion reactors.

Right now only some codes can share data. Different data formats and data names confound communication between many codes. Also challenging is getting codes to couple, or provide information to other codes at specific times to solve equations about the evolving state of plasma. Coupling is a tough task because a factor described by one code depends on another factor described by some other code.

“We’ve developed basically a lingua franca whereby different physics codes can talk to each other,” Batchelor says. In 2007 he and a team of more than two dozen researchers at 10 institutions used resources at the National Center for Computational Sciences (NCCS) to make progress toward developing an integrated plasma simulator. The work was made possible with an allocation of 1.7 million processor hours on the center’s Cray Jaguar XT supercomputer through a Department of Energy (DOE) Office of Science program called INCITE (for Innovative and Novel Computational Impact on Theory and Experiment).

Controlling plasma is the key to getting cheap, clean energy from future commercial reactors. Construction is under way for ITER, an experimental fusion reactor expected to begin operations in Europe in 2016. The $13 billion, 30-year multinational megaproject will build and operate a full-scale experimental device to demonstrate the technical feasibility of fusion energy. Following ITER would be a full-scale commercial fusion plant by 2050 and large-scale fusion-power adoption over the ensuing 30 years.

Improved simulation capability is urgently needed to support ITER, Batchelor says. Simulations are critical to supporting both theory and experiment. Theorists use simulations to gain insight into complex equations that correctly describe the state of plasma but that are difficult to intuit without solving the equations and visualizing the data. Experimentalists use simulations to evaluate if the desired trial conditions are achievable with the available equipment, guide operation of equipment, and interpret measurements from experiments.

“Once you have a simulation which is validated against the experiment—that you have confidence in—then you can use it to predict the next device,” Batchelor says.

Two of the most important fusion codes simulate radio frequency (RF) waves, which heat and control plasma, and magnetohydrodynamics (MHD), the behavior of “fluid” that has a magnetic field and carries current. The goals of Batchelor’s project are to use coupled codes to improve understanding of RF and MHD interactions in plasma, develop an integrated computational system for treating many physics phenomena, and serve as a prototype for the Fusion Simulation Project (FSP). The FSP aims to simulate the behavior of toroidal magnetic fusion devices called tokamaks on all important time and space scales and to account for the interactions of all relevant processes. It can contribute to ITER’s experimental planning and performance optimization and design of devices beyond ITER.

If today’s simulations support experiments, tomorrow’s experiments may support simulations. “Looking ahead, you could consider that the purpose of the ITER device is to provide data to validate the FSP,” Batchelor says. “If the purpose of developing all this simulation is to take the next step—have a validated simulation that enables you to build a demo reactor or a commercial reactor—then all these experiments are to build a scientific basis for that, and a major factor for taking the next step beyond ITER is going to be simulation. On the one hand FSP will support ITER by helping it do its experiments. But the real value of ITER is going to be to validate the simulations so we can extrapolate with confidence to the next step.”

UCI Researchers, NCCS Staff Conduct Breakthrough Fusion Simulation

GTC consumes 93 percent of 263TF Jaguar

A team of researchers from the University of California-Irvine (UCI), in conjunction with staff at the NCCS, has just completed what it says is the largest run in fusion simulation history.

The team, led by Yong Xiao and Zhihong Lin of UCI, used 93 percent of the NCCS’s flagship supercomputer Jaguar, a Cray XT4, with the classic fusion code GTC (Gyrokinetic Toroidal Code), the key production code of two fusion SciDAC projects (GPS-TTBP and GSEP).

The simulation primarily studied electron transport in ITER, a prototype fusion reactor now in development that is meant to test fusion’s feasibility for commercial power production (because in ITER the fusion process will primarily heat electrons, electron transport will be more important compared to existing fusion devices). Fusion energy could one day provide the world with a cleaner, more abundant energy source with far fewer harmful emissions than fossil-fuel-burning power plants and fewer waste issues than current nuclear power production.

To successfully produce a fusion reaction, an extremely hot ionized gas known as a plasma must be confined magnetically for a sufficient period of time. Previous research has shown that heat transport for both the ions and the electrons in the plasma is far greater than theory predicts (the electron heat transport can be two orders of magnitude greater). This larger-than-expected heat transport could lead to confinement failure within one second, quickly rendering energy production impossible if reactor designs are not modified to accommodate it.

The researchers discovered, among other things, that for a device the size of ITER, the containment vessel will demonstrate GyroBohm scaling, meaning that the heat transport level is inversely proportional to the device size. In other words, the simulation supports the ITER design: a larger device will lead to more efficient confinement.

“The success of fusion research depends on good confinement of the burning plasma,” said Xiao. “This simulation size is the one closest to ITER in terms of practical parameters and proper electron physics.”

Because fusion simulation is such a complex process, today’s leadership-class supercomputers such as Jaguar, with their fast CPU speeds and large memory, are an absolute necessity. GTC can effectively utilize large numbers of Jaguar’s cores thanks to an efficient three-level parallelism (using both MPI and OpenMP) designed by Stephane Ethier (Princeton Plasma Physics Laboratory) and Zhihong Lin.

However, the huge amounts of data produced by fusion simulations can create I/O nightmares: in one GTC run, the team can produce terabytes of data (in this case 60TB). To address this potential bottleneck, the team used ADIOS, a set of library files that allows for easy and fast file input and output, developed mainly by the NCCS’s Scott Klasky and Chen Jin and Georgia Tech’s Jay Lofstead and Karsten Schwan.

“This huge amount of data needs fast and smooth file writing and reading,” said Xiao. “With poor I/O, the file writing takes up precious computer time and the parallel file system on machines such as Jaguar can choke. With ADIOS, the I/O was vastly improved, consuming less than 3 percent of run time and allowing the researchers to write tens of terabytes of data smoothly without file system failure.”

ADIOS is a good example of the symbiotic relationship between researchers and NCCS staff. “My experience with Cray and the NCCS has been very good,” said Xiao. “The account and operation staffs are very accessible and responsible, which enabled us to run the GTC code with a total of 28,000 cores smoothly for two days. [The NCCS’s] Scott Klasky provided an effective channel for technical communication between the science application team and the computational support team.”

Despite the success of the recent simulation, there is still much work to be done. “Plasma turbulent transport is still an ongoing research area,” said Xiao. However, each new simulation brings new data, which bring a revolutionary energy source one step closer to reality.

ORNL Alive and Well at SciDAC 08

Seattle sets stage for laboratory’s HPC achievements

The 2008 SciDAC (Scientific Discovery through Advanced Computing) symposium once again showcased the numerous research achievements taking place at ORNL.

The meeting, held in Seattle from July 13-17, hosted over 300 scientists discussing the latest research breakthroughs in scientific computing. Three representatives from ORNL served as session chairs and several ORNL researchers presented talks, posters, and tutorials to the annual gathering of computational scientists.

A few of ORNL’s contributions included Bronson Messer’s discussion of the groundbreaking astrophysics simulations taking place at the lab titled “Multidimensional, Multiphysics Simulations of Core-Collapse Supernovae;” Rebecca Hartman-Baker’s introduction to the Cray XT4; an ORNL team’s (led by Scott Klasky) tutorial on ADIOS, an adaptable file input/output system that is the product of a collaboration between researchers at Georgia Tech University and ORNL; and Richard Barrett’s forward-looking talk titled “New Languages for Enabling Exascale Science”.

SciDAC, a program in the Department of Energy’s Office of Science, brings together the best and brightest from multiple disciplines to address research areas that prove difficult for theory and experiment alone. These collaborations have led to significant advances, via simulation, in understanding some of science’s most challenging questions, including energy alternatives and climate change.