National Center for Computational Sciences

ORNL’s Petaflop Jaguar Completes Acceptance Testing

World’s most powerful scientific computer already tackling critical challenges

The petascale upgrade to Oak Ridge National Laboratory’s (ORNL’s) Jaguar supercomputer has completed acceptance testing, verifying the power and stability of the most powerful system ever built for open scientific research.

Conducted by staff from the Oak Ridge Leadership Computing Facility (LCF) at the National Center for Computational Sciences (NCCS) and the system’s manufacturer, Cray, Inc., the testing demonstrated that Jaguar is ready to tackle the world’s most pressing challenges. The process concluded with a weeklong marathon in which the system demonstrated its stability by running a suite of applications from the climate, fusion, materials research, combustion science, chemistry, and astrophysics communities.

In the coming months, Jaguar will host a series of mammoth simulations featuring some of the world’s most demanding scientific computing applications. According to NCCS Director of Science Douglas Kothe, several of those applications were also used in Jaguar’s acceptance test.

“The point,” he said, “is that on Day 1 we already know we can do work—we can generate the science—because these codes have been there and helped us determine that the system is ready.”

Jaguar’s latest upgrade brought the system to a peak performance of 1.64 thousand trillion calculations a second—or 1.64 petaflops—making it the first petaflop system dedicated to open scientific research and the second such system ever built. Jaguar uses more than 45,000 quad-core AMD Opteron processors. In addition, it has 362 terabytes of memory (more than three times that of any other system in existence), a 10-petabyte file system, 578 terabytes per second of memory bandwidth, and an unprecedented input/output bandwidth of 284 gigabytes per second.

Jaguar had demonstrated why it is the world’s most valuable scientific computing system weeks before acceptance was completed. Teams led by Thomas Schulthess of ORNL used the system to run the first two scientific applications ever to top 1 petaflop sustained performance. One of these brought in the prestigious Gordon Bell Prize, an award administered by the Association for Computing Machinery that each year recognizes the world’s fastest performance on a scientific computing application. In addition, Jaguar placed in three of four categories at the recent High-Performance Computing Challenge awards, winning two “gold medals” and one “bronze” in a head-to-head competition of the world’s fastest supercomputers.

Jaguar’s steady performance is a reflection of its history. The current system is the culmination of four years of steady upgrades guided by the center and its user community. The acceptance tests, in fact, included specific performance goals for each of the applications included. As a result, applications can make use of the system’s unparalleled performance with a minimum of growing pains.

“It’s very difficult to go from 100 processors to 100,000 processors in one step,” noted Buddy Bland, director of the Oak Ridge LCF. “When the machine grows over time, you get a chance to take it from a few hundred processors, to a few thousand, to tens of thousands, to a hundred thousand. You need that time to work through the problems you find at each of those steps in order to get the applications to scale to these sizes. So we’ve been fortunate—and this is part of the design from the beginning of this series of machines—that the process gives us this scalability.”

In order to pass, the system was first put through a hardware acceptance process that verified that equipment such as the disks, interconnects, memory, and network was installed correctly and working. In the next phase, software acceptance, the suite of scientific applications ran for a solid week. The system could have no more than three reboots and 10 percent downtime to pass the test. In addition, 95 percent of the application runs had to complete successfully.

“The acceptance test that we’ve developed is an extremely strenuous test of the functionality, performance, and stability of the system,” Bland noted. “When we finish the acceptance test, we know that Jaguar is a very stable, reliable, high-performing system that will meet the needs of the scientists.”

NCCS Announces Early Petascale Applications

Pioneering science will put Jaguar to the test

With the upgrade of the Oak Ridge Leadership Computing Facility’s (LCF’s) Cray XT supercomputer known as Jaguar (http://www.nccs.gov/jaguar), high-performance computing (HPC) has entered the petascale. With a peak performance of 1.6 petaflops, Jaguar is now the most powerful computer in the world for open science and is the first to enter this new era, one that will certainly redefine the potential of HPC research.

To facilitate the most able codes and to test the mettle of the new system, the Oak Ridge National Laboratory’s National Center for Computational Sciences (NCCS), which manages the machine, has granted early access to a number of projects that can utilize a majority of the machine and take it, and science, to their respective limits.

“The current plan is for the system to be used during the next several months for specific high-impact projects of national importance,” said NCCS Director of Science Doug Kothe in a recent interview featured on HPCwire. “We have three principal goals during the system’s early phase: deliver important, high-impact science results and advancements; harden the system for production; and embrace a broad user community capable of and prepared for using the system.”

This priority “Petascale Early Science” period will run approximately 6 months and consist initially of 20 projects, said Kothe, adding that a broad range of science, including climate, chemistry, biology, combustion, and materials science, will be explored.

The climate projects will incorporate atmospheric and oceanic elements and will produce results at higher resolutions, bringing researchers one step closer to understanding the dynamics of Earth’s complex climate system. Ultimately, the data gleaned from these allocations will provide policymakers with information with which to better plan for predicted future climate fluctuations, such as regional climate change on decadal time scales.

The featured chemistry proposal will focus on enhanced energy storage in nanostructured systems, which has the potential to revolutionize battery and related technologies; the biology proposal will explore a more efficient means of converting cellulose to ethanol, a process that could one day make economically feasible biofuels a reality; combustion research will dissect the properties of ignition and flame dynamics, paving the way for more efficient future engines; and the materials science effort will use the classic Hubbard model for the design of high-temperature superconductors, a technology that could increase the efficiency of transmitting electricity several times over.

Fusion, nuclear energy, materials science, nuclear physics, astrophysics, and geosciences (exploring carbon sequestration) will also be explored.

“We expect these projects to deliver important results. Since they will be led by the community’s most sophisticated users and prominent scientists, early simulations on Jaguar will also help us harden the system for a broader collection of projects later in the year,” said Kothe, referring to the upcoming Innovative and Novel Computational Impact on Theory and Experiment (INCITE) projects hosted annually by the Oak Ridge LCF.

These early pioneering research applications will push computational science to new limits, giving researchers a peek at the possibilities of petascale simulation and prepping Jaguar for the nearly 40 INCITE projects that will explore domains from biology to fusion energy in unprecedented detail. Now that the petascale era has finally arrived, the work begins.

Tracking CFCs in a Global Eddying Ocean Model

Ocean ventilation provides clues to climate change

“The entire ocean is affected by a pebble,” said Blaise Pascal, the 17th century scientist. “The least movement is of importance to all nature.” Pascal, a mystic at heart, may have overstated the case. However, researchers today increasingly recognize the important, but largely unknown, influence of oceanic activity on climate change. They want to know how the ocean is coping with vast deposits of chemical pollutants, how it moves them about, stores them over long periods of time, and ultimately exchanges them at the surface with the air. Using the powerful computers now available, they are building simulations to assess with greater precision the long-term effect of this oceanic housekeeping on global climate.

Oceans play a critical role in the earth’s balance of heat, water, and chemicals such as carbon dioxide (CO2) and chlorofluorocarbons (CFCs). After absorbing chemicals from the atmosphere at the surface, the ocean can store substances for hundreds to thousands of years, circulating them through the 319 million cubic miles of water around the globe. This process, called ventilation, influences climate in multiple, still-to-be-determined ways. It is very difficult to measure directly, but it can be inferred from observations of dissolved chemical compounds, or tracers. One particularly useful class of chemical tracers for seeing how chemicals are moved through the ocean is CFCs, which human activity has introduced into the atmosphere.

Using Jaguar, the Cray XT system at the Oak Ridge Leadership Computing Facility (LCF), Synte Peacock and Frank Bryan at the National Center for Atmospheric Research (NCAR) and Mathew Maltrud at Los Alamos National Laboratory (LANL) have for the first time carried out a global eddy resolving model that has run a 100-year simulation. The model carried not only CFCs but also a host of other tracers that yielded valuable information about ocean ventilation pathways and timescales. To date, this team has been able to refine and successfully reassess earlier estimates linking changes in pollutant concentrations to climate change.

“This will help researchers better understand the role of the ocean in uptake and redistribution of gases such as anthropogenic (man-made) CO2, which will ultimately increase understanding of the role that oceanic activity plays in climate change,” said Peacock.

Because of the limits of computational power, most previous studies of CFC distributions using ocean models have been done using fairly coarse resolutions (grid spacing greater than 100 kilometers) for which some important transport activities are either poorly resolved or poorly estimated. To begin to resolve features such as narrow currents and mesoscale eddies (circular loop-like features with a diameter of less than 200 kilometers), researchers need a model with a finer grid resolution—one of kilometers to tens of kilometers.

Thanks to powerful supercomputers such as Jaguar, it has been possible to perform studies of the ocean uptake of CFCs and other trace gases using global fine-resolution (eddying) models. The NCAR/LANL model is one of the most realistic global eddying models ever run, Maltrud said, and the only one to simulate such a large set of tracer distributions. A standard way to assess the accuracy of the model’s eddy strength is to compare model sea-surface height changes with measurements from satellite altimeters (signals bounced off the sea surface to detect local changes in the height of the water). The close agreement between altimeter readings and this model on the size of the eddies and their patterns is unprecedented in this type of ocean model.

Earlier studies have noted apparent changes in the CFC concentrations in various parts of the world ocean and attributed these to changing ocean circulation driven by changing climate. But do these conclusions hold water? The NCAR and LANL researchers were able to investigate this question with five CFC simulations, each one giving a sample of the possible oceanic CFC distribution. Using their eddy-resolving CFC simulation to quantify measurement uncertainty, the researchers found that in a number of places where such studies had taken measurements, the observed CFC age differences were only marginally significant, and most of the change was attributable to eddy variability deep beneath the surface. These results, the first of their kind, will be very useful in the future for both comparing model output with actual measurements and helping scientists interpret the observational record.

ORNL Hosts Online HPC Classes

Lab begins relationship with four new institutions

Oak Ridge National Laboratory (ORNL) is offering a high-performance scientific computing class to four historically black colleges and universities.

“This opportunity allows leveraging previously untapped student talent with advanced research capabilities,” said Robert Franklin, president of Morehouse College in Atlanta, Georgia. “It also prepares the students for expanded career opportunities in the future. We have confidence that the course will give rise to many new ideas and other initiatives that we can’t even dream of at this time.”

Also attending the classes are students and faculty from Knoxville College in Knoxville, Tennessee; Claflin University in Orangeburg, South Carolina; and Jackson State University in Jackson, Mississippi.

The survey course, featuring about 25 students and faculty from the four colleges, will meet twice a week via satellite at ORNL. There they will learn the basics of using the systems, parallel programming, model design, and visualization.

“It’s our expectation that this will give us the sort of scientific workforce we need to meet the challenges of the 21st century,” said Thom Mason, Laboratory Director of ORNL. “This connects our unparalleled resources for scientific computing with Morehouse’s computer science department. That gives a unique learning opportunity for students at Morehouse and also for the lab.”