National Center for Computational Sciences

2009 DOE INCITE Projects Allocated at ORNL

Jaguar to host more than half of allocated hours

In 2009, Oak Ridge National Laboratory (ORNL) will make nearly 470 million processor hours available on Jaguar, its Cray XT supercomputer, under the Department of Energy’s Innovative and Novel Computational Impact on Theory and Experiment, or INCITE, program.

Thirty eight separate projects will advance breakthrough research in critical areas such as climate studies, energy assurance, materials, and other areas of fundamental science. All told, the DOE program is allocating 889 million processor hours at national laboratories in Illinois, California and Washington as well as ORNL.

“From understanding the makeup of our universe to protecting the quality of life here on earth, the computational science now possible using DOE’s supercomputers touches all of our lives,” said DOE Under Secretary for Science Raymond Orbach, who launched INCITE in 2003. “By dedicating time on these supercomputers to carefully selected projects, we are advancing scientific research in ways we could barely envision 10 years ago, improving our national competitiveness.”

Since a processor hour is equivalent to one hour on one processing core, a single hour using all of Jaguar’s 181,000 processing cores is equivalent to 181,000 processor hours. Located within ORNL’s Leadership Computing Facility (LCF) in the National Center for Computational Sciences (NCCS), Jaguar is the world’s most powerful supercomputer for open scientific research, with a peak performance of 1.64 quadrillion calculations a second, or 1.64 petaflops.

The 2009 INCITE program reflects several recent upgrades to the system. The LCF’s allocation in the coming year is more than three times the 145 million processor hours allocated in 2008 and more than six times the 75 million processor hours allocated in 2007.

Researchers will use Jaguar to continue their record of scientific achievement. In the past year, INCITE projects at the LCF have conducted the largest, most detailed simulations ever attempted in a wide range of scientific explorations, including the physics of electron transport in the prototype ITER fusion power reactor, the behavior of carbon dioxide sequestered from power plant emissions and pumped deep underground, the workings of a molecular machine known as the voltage-gated potassium channel, and the evolution of dark matter in our Milky Way galaxy. These achievements pave the way for unprecedented advances in alternative energy technology, climate change mitigation, nanotechnology, and our understanding of the universe.

The 2009 allocations reflect ORNL’s leadership role in pushing the boundaries of climate science. Researchers from the National Center for Atmospheric Research, several national laboratories (Oak Ridge, Pacific Northwest, Los Alamos, Argonne and Lawrence Livermore), NASA, and Georgia Institute of Technology received 30 million processor hours for a climate-science computational end station, part of which is dedicated to advancing the Community Climate System Model (CCSM), one of the world’s leading models for predicting and analyzing climate change.

“The extremely high quality and increased demand of this year’s INCITE awards is testimony to the role of leadership computing in facilitating breakthrough science,” said ORNL Director Thom Mason.

A team led by Venkatramani Balaji of the National Oceanic and Atmospheric Administration (NOAA) was allocated 24 million processor hours to advance an ambitious climate simulation project known as Coupled High-Resolution Modeling of the Earth System. A team led by Zhengyu Liu of the University of Wisconsin–Madison received 4 million processor hours for a project using CCSM to study abrupt climate change. And a team led by Gilbert Compo of the University of Colorado and NOAA received more than 1 million hours to produce the first-ever dataset of global weather maps for the period 1850–2011.

“It’s extremely gratifying to know that we will be working with such a strong collection of scientific partners, particularly in areas of such critical importance to the nation,” said NCCS Director James Hack. “The demand for the unique capabilities available through the LCF continues to grow and reflects the rapidly growing complexity of the most computationally demanding of scientific problems.”

Energy assurance projects will also play a prominent role. Fusion energy simulations will include teams led by Patrick Diamond of the University of California–San Diego, which received 30 million processor hours, William Nevins of Lawrence Livermore National Laboratory and C. S. Chang of New York University, which received 20 million processor hours each, and Jeff Candy of General Atomics and Fred Jaeger of ORNL, which received 2 million processor hours each.

Teams led by Jacqueline Chen of Sandia National Laboratories and Madhava Syamlal will help make combustion energy cleaner and more efficient with allocations of 30 million hours and 13 million hours, respectively. And a team led by Jeremy Smith of ORNL will use biological simulations to improve the production of cellulosic ethanol with an allocation of 6 million hours.

General Atomics is not the only industrial partner that will be using Jaguar under the INCITE program. The 2009 roster reflects ORNL’s ongoing relationship with partners throughout private industry. A team led by Jihui Yang of General Motors is exploring materials that will be able to convert a vehicle’s waste heat into electricity. Cleaner aircraft gas turbine engines will be the focus of a project led by Robert Malecki of Pratt & Whitney. And Moeljo Hong of the Boeing Co. and colleagues are developing advanced tools for aircraft design.

More information about the LCF and its contribution to the 2009 INCITE program can be found at www.nccs.gov. Further details about the DOE INCITE program can be found at www.sc.doe.gov/ascr/incite/.

Oak Ridge Supercomputer Wins Big at HPC Challenge

“Jaguar” demonstrates its power and ease of use.

A Cray XT5 supercomputer named Jaguar that runs scientific applications at the U.S. Department of Energy’s (DOE’s) Oak Ridge National Laboratory (ORNL) placed in three out of four categories at the High-Performance Computing (HPC) Challenge awards, winning two “gold medals” and one “bronze” in this head-to-head competition. Results of the challenge, which measures excellence at handling computing workloads, were announced November 18 in Austin at SC08, an international gathering of supercomputing professionals.

Jaguar won first place for both speed in solving a dense matrix of linear algebra equations (running a software code called High-Performance Linpack, or HPL) and sustainable memory bandwidth—or how many gigabytes per second a node can fetch and store (running the STREAM code). It won third place for speed in executing the Global-Fast Fourier Transformation, a common algorithm used in many scientific applications.

“The Cray Jaguar at ORNL winning two of the HPC Challenge benchmarks shows the power and potential of the computer system for handling some of the most challenging computational science problems,” said Jack Dongarra of the University of Tennessee–Knoxville and ORNL. “It was able to produce an impressive 902 teraflops [trillion floating point operations per second] on HPL and 330 TB/s [terabytes per second] on STREAMS. Both results leave the second-place IBM Blue Gene/L at Lawrence Livermore National Laboratory far behind and demonstrate the balance between computing and communication bandwidth.”

ORNL, along with Cray’s Chapel team led by Brad Chamberlain, shared another award for the most elegant implementation of the HPC Challenge benchmark applications in Cray’s Chapel computer language.

John Levesque, director of the Cray Supercomputing Center of Excellence at ORNL, said the HPC Challenge, sponsored by the Defense Advanced Research Projects Agency High Productivity Computing Systems Program, supports hardware and software development needed to effectively use petascale computers, which can execute quadrillions of calculations each second. Similar to the way a decathlon measures performance in ten track and field events, the HPC Challenge measures a computer’s ability to excel in execution of a wide variety of components important to running scientific applications.

All of the benchmarks were run in two modes: baseline (no source-code modifications) and optimized (significant source-code modifications). Baselines demonstrate a machine’s overall performance and ease of use, whereas optimizations boost performance on one specific aspect of computation. ORNL submitted baselines for Jaguar reflecting the ease of use of the system. In the list of the baseline results, Jaguar ranked as the most powerful machine in three of the four categories and ranked second in the final category, according to the posted results.

“The fact that Jaguar won these awards and placed so highly on the four major benchmarks with the baseline run attests to the superior performance and balance of the system,” said Buddy Bland, project director for ORNL’s Leadership Computing Facility, which hosts Jaguar. “This is truly a remarkable machine. It is exceptionally powerful in every measure that is important to the scientists who use it. Because it is a general-purpose computer that is easy to use, the scientists using this machine have been able to set new performance records on a wide range of science problems in just its first week of availability.”

NCCS Experts Share Their Views with the World

Director Hack, Science Director Kothe interviewed for HPCwire

National Center for Computational Sciences (NCCS) Director Jim Hack and Science Director Doug Kothe recently shared their perspectives with the computational science community through interviews published in the online magazine HPCwire.

The November question-and-answer (Q&A) stories looked to the future. Hack outlined the future of computational climate science and Oak Ridge National Laboratory’s (ORNL’s) climate science initiative in particular, and Kothe reviewed the range of breakthrough research now possible on petascale computing systems such as the lab’s Jaguar supercomputer.

During the course of his interview, Hack noted that while climate science has so far been driven by the curiosity of researchers, in the future it will be dominated by the needs of resource managers who will need to know, for instance, if specific regions will be subject to increasing droughts or increasingly severe weather.

“For example, we know from observations over the last 50 years that the snowpack in the Pacific Northwest has been decreasing,” he noted. “At the same time, temperature in the same region has been increasing. If that trend continues, it raises lots of concerns for water resource managers who have counted on storing their water in the form of snow until a certain time of year when it starts melting.”

The whole Q&A can be found here.

Kothe discussed the exciting scientific results he expects to see as users take advantage of Jaguar, which is, at a peak performance of 1.64 quadrillion calculations a second (1.64 petaflops), the world’s most powerful system for open scientific research.

“The breadth and depth of critical science potentially solvable on this system are daunting,” he said, “with domains including fusion, biology, atomic physics, chemistry, nuclear energy, materials and nanoscience, climate and geosciences, astrophysics, high-energy physics, turbulence, and combustion. And this is not an exhaustive list.”

Kothe’s Q&A can be found at hpcwire.com.

Kendall Cochairs Cluster Challenge to Involve Undergraduates In Supercomputing

Team from Indiana University and Technische University Dresden triumphs at SC08

Ricky Kendall leads the Scientific Computing Group at the National Center for Computational Sciences, the Oak Ridge National Laboratory (ORNL) facility that fields the world’s fastest supercomputer for unclassified research. His team—with expertise in high-performance computing (HPC); fundamental scientific fields; and data management, analysis, and visualization—shows top researchers how to maximize the use of this unprecedented system. At SC08, a supercomputing conference held in Austin, Texas, November 15–21, Kendall labored behind the scenes with Brent Gorda of Lawrence Livermore National Laboratory and other HPC experts to organize a contest to excite and educate the next generation of HPC professionals. The Cluster Challenge pitted undergraduates from around the world against each other to build a computer cluster that would fit on a single rack, run a scientific application using 26 amps of power or less, and arrive at an answer first.

“The goal is to understand high-performance computing from a small-cluster perspective,” said Kendall, who put together a committee to review proposals in which students explained what hardware and software they would use, why their entry was strong enough to win, and how the event would improve the core curriculum at their universities. “The event is a backdoor way to get HPC into the undergraduate core curriculum, thus broadening the interest and scope of people involved in HPC,” said Kendall, who obtained scientific-application software from owners and made sure students and judges had what they needed for the race. He also ensured proper representation of sponsors (WesternGeCo, Morgan Stanley, and Chevron were the main backers) and kept the pizza and Mountain Dew flowing as shifts of students toiled around the clock from the SC08 exhibit floor, working with academic advisors and hardware vendors from SciCortex, SGI, IBM, HP, Cray, Microsoft, Aspen Systems, and Dell.

Students ran the same benchmark software that professionals ran in their own “Olympics” that week (the HPC Challenge, in which ORNL’s Cray XT machine prevailed). They also coordinated hardware and software for optimal performance and completed an interview with HPC luminaries led by Jack Dongarra, a University of Tennessee–Knoxville professor who cofounded the TOP500 list ranking the world’s fastest supercomputers. Students worked with resources that a decade ago would have been at or near the top of the TOP500 list, Kendall noted.

When the points were all tallied, a joint team from Indiana University and Technische University Dresden triumphed. A Purdue University team reigned for “most power-efficient cluster,” and University of Colorado students won “best Cluster Challenge spirit” for their efforts despite hardware obstacles. The other teams represented the Massachusetts Institute of Technology, Arizona State University, National Tsing-Hua University, and the University of Alberta.

Throughout the week, organizers had used bicycle racing as a theme to publicize the sprint to build a productive, efficient cluster. The Indiana/Dresden winners had a bike in their booth to power a laptop, and the Purdue team on the final day of the event hooked their energy-efficient SciCortex cluster to a generator powered by eight bicyclists—and rode into Cluster Challenge history.