From DOE’s Oak Ridge National Laboratory (US): “Updated exascale system for Earth simulations delivers twice the speed”

From DOE’s Oak Ridge National Laboratory (US)

January 20, 2022

Kimberly A Askey

The Energy Exascale Earth System Model project reliably simulates aspects of earth system variability and projects decadal changes that will critically impact the U.S. energy sector in the future. A new version of the model delivers twice the performance of its predecessor. Credit: E3SM, Dept. of Energy.

A new version of the Energy Exascale Earth System Model, or E3SM, is two times faster than an earlier version released in 2018.

Earth system models have weather-scale resolution and use advanced computers to simulate aspects of Earth’s variability and anticipate decadal changes that will critically impact the U.S. energy sector in coming years.

Scientists at the Department of Energy’s Oak Ridge National Laboratory are part of the team that developed version 2 of the model — E3SMv2 — which was released to the scientific community in September 2021.

“E3SMv2 delivered twice the performance over E3SMv1 when using identical computational resources,” said ORNL computational scientist Sarat Sreepathi, who co-leads the E3SM Performance Group. “This is a significant achievement as the performance boost is reflected while running the fully integrated Earth system model and not just confined to smaller model components.”

The Earth, with its myriad interactions of atmosphere, oceans, land and ice components, presents an extraordinarily complex system for investigation. Earth system simulation involves solving approximations of physical, chemical and biological governing equations on spatial grids at resolutions that are as fine in scale as computing resources will allow.

“Even with the addition of new features in E3SMv2 to the atmosphere model and how it represents precipitation and clouds, we still doubled the model throughput,” Sreepathi said. “To put it another way, we cut the computational run time or time-to-solution in half.”

“E3SMv2 allows us to more realistically simulate the present, which gives us more confidence to simulate the future,” said David Bader, Lawrence Livermore National Laboratory scientist and lead of the E3SM project. “The increase in computing power allows us to add more detail to processes and interactions that results in more accurate and useful simulations than the previous version.”

Achieving these improvements required collaboration across the national laboratory system. Sreepathi, along with ORNL’s Gaurab KC and Youngsung Kim, accelerated the effort by creating a comprehensive monitoring framework called PACE, or Performance Analytics for Computational Experiments. The PACE web portal provided both an automatic data collection system and a streamlined interface for scientists to evaluate the performance of E3SM experiments executed on DOE supercomputers. These data facilitated feedback-driven E3SMv2 model development and allowed researchers to optimize their experiments.

“Using the PACE web portal helped the multi-laboratory team understand how new model features were impacting computational performance,” said Sreepathi. “We were able to accurately track the evolution of the model’s performance.”

The E3SM project reliably simulates aspects of Earth system variability, including regional air and water temperatures, which can strain energy grids; water availability, which affects power plant operations; extreme water-cycle events, such as floods and droughts, which impact infrastructure and bioenergy resources; and sea-level rise and coastal flooding, which threaten coastal infrastructure.

In addition, the resolution has been refined due to more powerful computers. There are now two fully coupled configurations: 100-kilometer, or km, globally uniform resolution atmosphere model and a regionally refined model, or RRM, with a resolution with 25 km over North America and 100 km elsewhere. The refined mesh configuration is particularly well suited for DOE applications.

“Thanks to the performance improvements, the RRM configuration of E3SMv2 runs as fast as E3SMv1 did in its standard resolution configuration (100 km) a few years ago. We are essentially getting the much higher resolution for ‘free,’” said LLNL atmospheric scientist Chris Golaz.

The team is now conducting the simulation campaign with E3SMv2. Team members have already simulated several thousand years, and are planning to run several thousand more.

The project includes more than 100 scientists and software engineers at multiple DOE laboratories as well as several universities; the DOE laboratories include DOE’s Argonne National Laboratory(US), DOE’s Brookhaven National Laboratory(US), DOE’s Lawrence Livermore National Laboratory(US), DOE’s Lawrence Berkeley National Laboratory (US), DOE’s Los Alamos National Lab (US), Oak Ridge, DOE’s Pacific Northwest National Laboratory (US) and DOE’s Sandia National Laboratory (US). In recognition of unifying the DOE earth system modeling community to perform high-resolution coupled simulations, the E3SM executive committee was awarded the Secretary of Energy’s Achievement Award in 2015.

In addition, the E3SM project benefits from DOE programmatic collaborations, including The Exascale Computing Project and research efforts in Scientific Discovery Through Advanced Computing, Climate Model Development and Validation, Atmospheric Radiation Measurement, Program for Climate Model Diagnosis and Intercomparison, International Land Model Benchmarking Project, Community Earth System Model and Next Generation Ecosystem Experiments for the Arctic and the Tropics.

The E3SM project is supported by the Biological and Environmental Research program in DOE’s Office of Science.

See the full article here .


Please help promote STEM in your local schools.

Stem Education Coalition

Established in 1942, DOE’s Oak Ridge National Laboratory (US) is the largest science and energy national laboratory in the Department of Energy system (by size) and third largest by annual budget. It is located in the Roane County section of Oak Ridge, Tennessee. Its scientific programs focus on materials, neutron science, energy, high-performance computing, systems biology and national security, sometimes in partnership with the state of Tennessee, universities and other industries.

ORNL has several of the world’s top supercomputers, including Summit, ranked by the TOP500 as Earth’s second-most powerful.

ORNL OLCF IBM AC922 SUMMIT supercomputer, was No.1 on the TOP500..

The lab is a leading neutron and nuclear power research facility that includes the Spallation Neutron Source and High Flux Isotope Reactor.

It hosts the Center for Nanophase Materials Sciences, the BioEnergy Science Center, and the Consortium for Advanced Simulation of Light Water Nuclear Reactors.

ORNL is managed by UT-Battelle for the Department of Energy’s Office of Science. DOE’s Office of Science is the single largest supporter of basic research in the physical sciences in the United States, and is working to address some of the most pressing challenges of our time.

Areas of research

ORNL conducts research and development activities that span a wide range of scientific disciplines. Many research areas have a significant overlap with each other; researchers often work in two or more of the fields listed here. The laboratory’s major research areas are described briefly below.

Chemical sciences – ORNL conducts both fundamental and applied research in a number of areas, including catalysis, surface science and interfacial chemistry; molecular transformations and fuel chemistry; heavy element chemistry and radioactive materials characterization; aqueous solution chemistry and geochemistry; mass spectrometry and laser spectroscopy; separations chemistry; materials chemistry including synthesis and characterization of polymers and other soft materials; chemical biosciences; and neutron science.
Electron microscopy – ORNL’s electron microscopy program investigates key issues in condensed matter, materials, chemical and nanosciences.
Nuclear medicine – The laboratory’s nuclear medicine research is focused on the development of improved reactor production and processing methods to provide medical radioisotopes, the development of new radionuclide generator systems, the design and evaluation of new radiopharmaceuticals for applications in nuclear medicine and oncology.
Physics – Physics research at ORNL is focused primarily on studies of the fundamental properties of matter at the atomic, nuclear, and subnuclear levels and the development of experimental devices in support of these studies.
Population – ORNL provides federal, state and international organizations with a gridded population database, called Landscan, for estimating ambient population. LandScan is a raster image, or grid, of population counts, which provides human population estimates every 30 x 30 arc seconds, which translates roughly to population estimates for 1 kilometer square windows or grid cells at the equator, with cell width decreasing at higher latitudes. Though many population datasets exist, LandScan is the best spatial population dataset, which also covers the globe. Updated annually (although data releases are generally one year behind the current year) offers continuous, updated values of population, based on the most recent information. Landscan data are accessible through GIS applications and a USAID public domain application called Population Explorer.