From The DOE’s Thomas Jefferson National Accelerator Facility : “Nuclear Physics Gets a Boost for High-Performance Computing” 

From The DOE’s Thomas Jefferson National Accelerator Facility

Kandice Carter
Jefferson Lab Communications Office

Jefferson Lab’s Data Center, JLab photo: Aileen Devlin

The Frontier Supercomputer, OLCF at The DOE’s Oak Ridge National Lab photo.

Efforts to harness the power of supercomputers to better understand the hidden worlds inside the nucleus of the atom recently received a big boost. A project led by the DOE’s Thomas Jefferson National Accelerator Facility is one of three to split $35 million in grants from the DOE via a partnership program of DOE’s Scientific Discovery through Advanced Computing (SciDAC).

Each of the projects receiving the grants are joint projects between DOE’s Nuclear Physics (NP) and Advanced Scientific Computing Research (ASCR) programs via a partnership program of SciDAC.

Making the Most of Advanced Computational Resources

As supercomputers become ever-more powerful, scientists need advanced tools to take full advantage of their capabilities. For example, the Oak Ridge Leadership Computing Facility (OLCF) at DOE’s Oak Ridge National Lab now hosts the world’s first public exascale supercomputer. Its Frontier supercomputer has achieved 1 exaFLOPS in capability by demonstrating it can perform one billion-billion calculations per second.

“Nuclear physics is a rich, diverse and exciting area of research explaining the origins of visible matter. And in nuclear physics, high-performance computing is a critically important tool in our efforts to unravel the origins of nuclear matter in our universe,” said Robert Edwards, a senior staff scientist and deputy group leader of Jefferson Lab’s Center for Theoretical and Computational Physics.

Edwards is the principal investigator for one of the three projects. His project, “Fundamental nuclear physics at the exascale and beyond,” will build a solid foundation of software resources for nuclear physicists to address key questions regarding the building blocks of the visible universe. The project seeks to help nuclear physicists tease out questions about the basic properties of particles, such as the ubiquitous proton.

“One of the key research questions that we hope to one day answer is what is the origin of a particle’s mass, what is the origin of its spin, and what are the emerging properties of a dense system of particles?” explained Edwards.

The $13 million project includes key scientists based at six DOE national labs and two universities, including Jefferson Lab, The DOE’s Argonne National Lab, The DOE’s Brookhaven National Lab, Oak Ridge National Lab, The DOE’sLawrence Berkeley National Lab, The DOE’s Los Alamos National Lab, The Massachusetts Institute of Technology and The College of William & Mary.

It aims to optimize the software tools needed for calculations of quantum chromodynamics (“QCD”). QCD is the theory that describes the structure of protons and neutrons – the particles that make up atomic nuclei – as well as provide insight to other particles that help build our universe. Protons are built of smaller particles called quarks held together by a force-fed glue manifesting as gluon particles. What’s not clear is how the proton’s properties arise from quarks and gluons.

“The evidence points to the mass of quarks as extremely tiny, only 1%. The rest is from the glue. So, what part does glue play in that internal structure?” he said.

Modeling the Subatomic Universe

The goal of the supercomputer calculations is to mimic how quarks and gluons experience the real world at their own teensy scale in a way that can be calculated by computers. To do that, the nuclear physicists use supercomputers to first generate a snapshot of the environment inside a proton where these particles live for the calculations. Then, they mathematically drop in some quarks and glue and use supercomputers to predict how they interact. Averaging over thousands of these snapshots gives physicists a way to emulate the particles’ lives in the real world.

Solutions from these calculations will provide input for experiments taking place today at Jefferson Lab’s Continuous Electron Beam Accelerator Facility (CEBAF)[below] and Brookhaven Lab’s Relativistic Heavy Ion Collider (RHIC).

CEBAF and RHIC are both DOE Office of Science user facilities.

“While we did not base this proposal on the requirements of the future Electron-Ion Collider, many of the problems that we are trying to address now, such as code infrastructures and methodology, will impact the EIC,” Edwards added.

The project will use a four-pronged approach to help streamline these calculations for better use on supercomputers, while also preparing for ever-more-powerful machines to come online.

The first two approaches relate to generation of the quarks’ and gluons’ little slice of the universe. The researchers aim to make this task easier for computers by streamlining the process with upgraded software and by using software to break down this process into smaller chunks of calculations that will be easier for a computer to calculate. The second part of this project will then bring in machine learning to see if the existing algorithms can be improved by additional computer modelling.

The third approach involves exploring and testing out new techniques for the portion of the calculations that model how quarks and gluons interact in their computer-generated universe.

The fourth and last approach will collect all of the information from the first three prongs and begin to scale them for use on next-generation supercomputers.

All three SciDAC projects awarded grants by DOE span efforts in nuclear physics research. Together, the projects address fundamental questions about the nature of nuclear matter, including the properties of nuclei, nuclear structure, nucleon imaging, and discovering exotic states of quarks and gluons.

“The SciDAC partnership projects deploy high-performance computing and enable world-leading science discoveries in our nuclear physics facilities,” said Timothy Hallman, DOE’s associate director of science for NP.

The total funding announced by DOE includes $35 million lasting five years, with $7.2 million in Fiscal Year 2022 and outyear funding contingent on congressional appropriations.

See the full article here .

Comments are invited and will be appreciated, especially if the reader finds any errors which I can correct. Use “Reply”.


Please help promote STEM in your local schools.

Stem Education Coalition

JLab campus
The DOE’s Thomas Jefferson National Accelerator Facility is supported by The Office of Science of the U.S. Department of Energy. DOE’s Office of Science is the single largest supporter of basic research in the physical sciences in the United States, and is working to address some of the most pressing challenges of our time. For more information, visit

Jefferson Science Associates, LLC, a joint venture of the Southeastern Universities Research Association, Inc. and PAE Applied Technologies, manages and operates the Thomas Jefferson National Accelerator Facility for the U.S. Department of Energy’s Office of Science.


The DOE’s Thomas Jefferson National Accelerator Facility was established in 1984 (first initial funding by the Department of Energy) as the Continuous Electron Beam Accelerator Facility (CEBAF); the name was changed to Thomas Jefferson National Accelerator Facility in 1996. The full funding for construction was appropriated by US Congress in 1986 and on February 13, 1987, the construction of the main component, the CEBAF accelerator begun. First beam was delivered to experimental area on 1 July 1994. The design energy of 4 GeV for the beam was achieved during the year 1995. The laboratory dedication took place 24 May 1996 (at this event the name was also changed). Full initial operations with all three initial experiment areas online at the design energy was achieved on June 19, 1998. On August 6, 2000 the CEBAF reached “enhanced design energy” of 6 GeV. In 2001, plans for an energy upgrade to 12 GeV electron beam and plans to construct a fourth experimental hall area started. The plans progressed through various DOE Critical Decision-stages in the 2000s decade, with the final DOE acceptance in 2008 and the construction on the 12 GeV upgrade beginning in 2009. May 18, 2012 the original 6 GeV CEBAF accelerator shut down for the replacement of the accelerator components for the 12 GeV upgrade. 178 experiments were completed with the original CEBAF.

In addition to the accelerator, the laboratory has housed and continues to house a free electron laser (FEL) instrument. The construction of the FEL started 11 June 1996. It achieved first light on June 17, 1998. Since then, the FEL has been upgraded numerous times, increasing its power and capabilities substantially.

Jefferson Lab was also involved in the construction of the Spallation Neutron Source (SNS) at DOE’s Oak Ridge National Laboratory . Jefferson built the SNS superconducting accelerator and helium refrigeration system. The accelerator components were designed and produced 2000–2005.


The laboratory’s main research facility is the CEBAF accelerator, which consists of a polarized electron source and injector and a pair of superconducting RF linear accelerators that are 7/8-mile (1400 m) in length and connected to each other by two arc sections that contain steering magnets.

As the electron beam makes up to five successive orbits, its energy is increased up to a maximum of 6 GeV (the original CEBAF machine worked first in 1995 at the design energy of 4 GeV before reaching “enhanced design energy” of 6 GeV in 2000; since then, the facility has been upgraded into 12 GeV energy). This leads to a design that appears similar to a racetrack when compared to the classical ring-shaped accelerators found at sites such as The European Organization for Nuclear Research [La Organización Europea para la Investigación Nuclear][Organization européenne pour la recherche nucléaire] [Europäische Organization für Kernforschung](CH)[CERN] or DOE’s Fermi National Accelerator Laboratory. Effectively, CEBAF is a linear accelerator, similar to The DOE’s SLAC National Accelerator Laboratory at Stanford University, that has been folded up to a tenth of its normal length.

The design of CEBAF allows the electron beam to be continuous rather than the pulsed beam typical of ring-shaped accelerators. (There is some beam structure, but the pulses are very much shorter and closer together.) The electron beam is directed onto three potential targets (see below). One of the distinguishing features of Jefferson Lab is the continuous nature of the electron beam, with a bunch length of less than 1 picosecond. Another is Jefferson Lab’s use of superconducting Radio Frequency (SRF) technology, which uses liquid helium to cool niobium to approximately 4 K (−452.5 °F), removing electrical resistance and allowing the most efficient transfer of energy to an electron. To achieve this, Jefferson Lab houses the world’s largest liquid helium refrigerator, and it was one of the first large-scale implementations of SRF technology. The accelerator is built 8 meters below the Earth’s surface, or approximately 25 feet, and the walls of the accelerator tunnels are 2 feet thick.

The beam ends in four experimental halls, labelled Hall A, Hall B, Hall C, and Hall D. Each hall contains specialized spectrometers to record the products of collisions between the electron beam or with real photons and a stationary target. This allows physicists to study the structure of the atomic nucleus, specifically the interaction of the quarks that make up protons and neutrons of the nucleus.

With each revolution around the accelerator, the beam passes through each of the two LINAC accelerators, but through a different set of bending magnets in semi-circular arcs at the ends of the linacs. The electrons make up to five passes through the linear accelerators.

When a nucleus in the target is hit by an electron from the beam, an “interaction”, or “event”, occurs, scattering particles into the hall. Each hall contains an array of particle detectors that track the physical properties of the particles produced by the event. The detectors generate electrical pulses that are converted into digital values by analog-to-digital converters (ADCs), time to digital converters (TDCs) and pulse counters (scalers).

This digital data is gathered and stored so that the physicist can later analyze the data and reconstruct the physics that occurred. The system of electronics and computers that perform this task is called a data acquisition system.

12 GeV upgrade

As of June 2010, construction began on a $338 million upgrade to add an end station, Hall D, on the opposite end of the accelerator from the other three halls, as well as to double beam energy to 12 GeV. Concurrently, an addition to the Test Lab, (where the SRF cavities used in CEBAF and other accelerators used worldwide are manufactured) was constructed.

As of May 2014, the upgrade achieved a new record for beam energy, at 10.5 GeV, delivering beam to Hall D.

As of December 2016, the CEBAF accelerator delivered full-energy electrons as part of commissioning activities for the ongoing 12 GeV Upgrade project. Operators of the Continuous Electron Beam Accelerator Facility delivered the first batch of 12 GeV electrons (12.065 Giga electron Volts) to its newest experimental hall complex, Hall D.

In September 2017, the official notification from the DOE of the formal approval of the 12 GeV upgrade project completion and start of operations was issued. By spring 2018, all fours research areas were successfully receiving beam and performing experiments. On 2 May 2018 the CEBAF 12 GeV Upgrade Dedication Ceremony took place.

As of December 2018, the CEBAF accelerator delivered electron beams to all four experimental halls simultaneously for physics-quality production running.