From UC Santa Barbara: “Tomorrow’s Data Centers”

UC Santa Barbara Name bloc
From UC Santa Barbara

November 12, 2019
Sonia Fernandez

The deluge of data we transmit across the globe via the internet-enabled devices and services that come online every day has required us to become much more efficient with the power, bandwidth and physical space needed to maintain the technology of our modern online lives and businesses.

L to r: Electrical and computer engineering professor Dan Blumenthal, and doctoral student researchers Grant Brodnik and Mark Harrington
Photo Credit: Sonia Fernandez

“Much of the world today is interconnected and relies on data centers for everything from business to financial to social interactions,” said Daniel Blumenthal, a professor of electrical and computer engineering at UC Santa Barbara. The amount of data now being processed is growing so fast that the power needed just to get it from one place to another along the so-called information superhighway constitutes a significant portion of the world’s total energy consumption, he said. This is particularly true of interconnects — the part of the internet infrastructure tasked with getting data from one location to another.

“Think of interconnects as the highways and the roads that move data,” Blumenthal said. There are several levels of interconnects, from the local types that move data from one device on a circuit to the next, to versions that are responsible for linkages between data centers. The energy required to power interconnects alone is 10% of the world’s total energy consumption and climbing, thanks to the growing amount of data that these components need to turn from electronic signals to light, and back to electronic signals. The energy needed to keep the data servers cool also adds to total power consumption.

“The amount of worldwide data traffic is driving up the capacity inside data centers to unprecedented levels and today’s engineering solutions break down,” Blumenthal explained. “Using conventional methods as this capacity explodes places a tax on the energy and cost requirements of physical equipment, so we need drastically new approaches.”

As the demand for additional infrastructure to maintain the performance of the superhighway increases, the physical space needed for all these components and data centers is becoming a limiting factor, creating bottlenecks of information flow even as data processing chipsets increase their capacity to a whopping 100 terabytes per second.

“The challenge we have is to ramp up for when that happens,” said Blumenthal, who also serves as director for UC Santa Barbara’s Terabit Optical Ethernet Center, and represents UC Santa Barbara in Microsoft’s Optics for the Cloud Research Alliance.

This challenge is a now job for Blumenthal’s FRESCO: FREquency Stabilized COherent Optical Low-Energy Wavelength Division Multiplexing DC Interconnects. Bringing the speed, high data capacity and low-energy use of light (optics) to advanced internet infrastructure architecture, the FRESCO team aims to solve the data center bottleneck while bringing energy usage and space needs to a more sustainable level.

The effort is funded by ARPA-e under the OPEN 2018 program and represents an important industry-university partnership with emphasis on technology transition. The FRESCO project involves important industry partners like Microsoft and Barefoot Networks (now Intel), who are looking to transition new technologies to solve the problems of exploding chip and data center capacities.

The keys, according to Blumenthal, are to shorten the distance between optics and electronics, while also drastically increasing the efficiency of maintaining the synchrony of the optical signal between the transmitting and receiving end of the interconnect.

FRESCO can accomplish this by bringing the performance of optical technology — currently relegated to long-haul transmission via fiberoptic cable — to the chip and co-locating both optic and electronic components on the same switch chip.

“The way FRESCO is able to do this is by bringing to bear techniques from large-scale physics experiments to the chip scale,” Blumenthal said. It’s a departure from the more conventional faceplate-and-plug technology, which requires signal to travel some distance to be converted before moving it along.

From Big Physics to Small Chips

Optical signals can be stacked in a technique known as coherent wave-division multiplexing (WDM), which allows signal to be sent over different frequencies — colors — over a single optical fiber. However, because of space constraints, Blumenthal said, the traditional measures used to process long-haul optical signals, including electronic digital signal processing (DSP) chips and very high bandwidth circuits, have to be removed from the interconnect links.

FRESCO does away with these components with an elegant and powerful technique that “anchors” the light at both transmitting and receiving ends, creating spectrally pure stable light that Blumenthal has coined “quiet light.”

“In order to do that we actually bring in light stabilization techniques and technologies that have been developed over the years for atomic clocks, precision metrology and gravitational wave detection, and use this stable, quiet light to solve the data center problem,” Blumenthal said. “Bringing key technologies from the big physics lab to the chip scale is the challenging and fun part of this work.”

Specifically, he and his team have been using a phenomenon called stimulated Brillouin scattering, which is characterized by the interaction of light — photons — with sound produced inside the material through which it is traveling. These sound waves — phonons — are the result of the collective light-stimulated vibration of the material’s atoms, which act to buffer and quiet otherwise “noisy” light frequencies, creating a spectrally pure source at the transmitting and receiving ends. The second part of the solution is to anchor or stabilize these pure light sources using optical cavities that store energy with such high quality that the lasers are anchored in a way that allows them to be aligned using low-energy electronic circuits used in the radio world.

The act of alignment requires that the light frequency and phase are kept equal so that data can be recovered. This normally requires high power analog electronics or high powered digital signal processors (DSPs), which are not viable solutions for bringing this capacity inside the data center (they have 100,000s of fiber connections in the data center, as compared to 10s of connections in the long-haul). Also, the more energy and space the technologies inside the data center take, an equal number or more get expended on the cooling of the data center.

“There is very little energy needed to just keep them aligned and finding each other,” Blumenthal said of FRESCO, “similar to that of electronic circuits used for radio. “That is the exciting part — we are enabling a transmission carrier at 400 THz to carry data using low-energy simple electronic circuits, as opposed to the use of DSPs and high bandwidth circuitry, which in essence throws a lot of processing power at the optical signal to hunt down and match the frequency and phase of the optical signal so that data can be recovered.” With the FRESCO method, the lasers from the the transmitting and receiving ends are “anchored within each other’s sights in the first place, and drift very slowly on the order of minutes, requiring very little effort to track one with the other,” according to Blumenthal.

On the Horizon, and Beyond

While still in early stages, the FRESCO team’s technology is very promising. Having developed discrete components, the team is poised to demonstrate the concept by linking those components, measuring energy use, then transmitting the highest data capacity over a single frequency with the lowest energy to date on a frequency stabilized link. Future steps include demonstrating multiple frequencies using a technology called optical frequency combs that are integral to atomic clocks, astrophysics and other precision sciences. The team is in the process of integrating these components onto a single chip, ultimately aiming to develop manufacturing processes that will allow for transition to FRESCO technology.

This technology is likely only the tip of the iceberg when it comes to possible innovations in the realm of optical telecommunications.

“We see our chipset replacing over a data center link what today would take between four to 10 racks of equipment,” Blumenthal said. “The fundamental knowledge gained by developing this technology could easily enable applications we have yet to invent, for example in quantum communications and computing, precision metrology and precision timing and navigation.”

“If you look at trends, over time you can see something that in the past took up a room full of equipment become something that was personally accessible through a technology innovation — for example supercomputers that became laptops through nanometer transistors,” he said of the disruption that became the wave in personal computing and everything that it enabled. “We know now how we want to apply the FRESCO technology to the data center scaling problem, but we think there also are going to be other unforeseen applications too. This is one of the primary reasons for research exploration and investment without knowing all the answers or applications beforehand.”

See the full article here .


Please help promote STEM in your local schools.

Stem Education CoalitionUC Santa Barbara Seal
The University of California, Santa Barbara (commonly referred to as UC Santa Barbara or UCSB) is a public research university and one of the 10 general campuses of the University of California system. Founded in 1891 as an independent teachers’ college, UCSB joined the University of California system in 1944 and is the third-oldest general-education campus in the system. The university is a comprehensive doctoral university and is organized into five colleges offering 87 undergraduate degrees and 55 graduate degrees. In 2012, UCSB was ranked 41st among “National Universities” and 10th among public universities by U.S. News & World Report. UCSB houses twelve national research centers, including the renowned Kavli Institute for Theoretical Physics.

#tomorrows-data-centers, #applied-research-technology, #basic-research, #bringing-the-speed-high-data-capacity-and-low-energy-use-of-light-optics-to-advanced-internet-infrastructure-architecture, #supercomputing, #the-amount-of-worldwide-data-traffic-is-driving-up-the-capacity-inside-data-centers-to-unprecedented-levels-and-todays-engineering-solutions-break-down, #the-deluge-of-data-we-transmit-across-the-globe-via-the-internet-enabled-devices-and-services-that-come-online-every-day-has-required-us-to-become-much-more-efficient, #the-keys-according-to-blumenthal-are-to-shorten-the-distance-between-optics-and-electronics, #this-challenge-is-a-now-job-for-blumenthals-fresco-frequency-stabilized-coherent-optical-low-energy-wavelength-division-multiplexing-dc-interconnects, #uc-santa-barbara, #while-still-in-early-stages-the-fresco-teams-technology-is-very-promising

From Symmetry: “How to share the data from LSST”

Symmetry Mag

Evelyn Lamb

The Large Synoptic Sky Survey will collect so much data that data scientists needed to figure out new ways for astronomers to access it.

M. Park/Inigo Films/LSST/AURA/NSF

The most detailed three-dimensional maps of the universe so far came from the Sloan Digital Sky Survey.

SDSS Telescope at Apache Point Observatory, near Sunspot NM, USA, Altitude 2,788 meters (9,147 ft)

Between 2000 and 2010, SDSS collected 20 terabytes of data, photographing one-third of the night sky.

When the Large Synoptic Survey Telescope high in the Chilean Andes becomes fully operational in 2022, its 3.2-gigapixel camera will collect the same amount of data—every night. And it will do so over and over again for ten years.

The LSST Vera Rubin Survey Telescope

LSST Camera, built at SLAC

LSST telescope, currently under construction on the El Peñón peak at Cerro Pachón Chile, a 2,682-meter-high mountain in Coquimbo Region, in northern Chile, alongside the existing Gemini South and Southern Astrophysical Research Telescopes.

LSST Data Journey, Illustration by Sandbox Studio, Chicago with Ana Kova

Back in the days of SDSS, scientists often downloaded data to their own institutions’ computers and ran analyses on their own equipment. That won’t be possible with LSST. “At half an exabyte, people are not going to be able to put this on their laptops,” Yusra AlSayyad, technical manager for the Princeton branch of the data management team, says of the LSST data.

Instead of bringing the data to scientists, LSST will need to bring scientists to the data.

The LSST data management team, consisting of approximately 80 people spread over six sites in the United States, is responsible for turning this deluge into something scientists can access and analyze.

LSST is under construction in Chile. W O’Mullane/LSST Project/NSF/AURA

These small motors, called actuators, will be installed to allow scientists to make small adjustments to the position of LSST’s combined primary/tertiary mirror. W O’Mullane/LSST Project/NSF/AURA.

This small version of the LSST camera called ComCam will test the observatory while the real camera is being constructed. LSST Project/NSF/AURA.

Once construction is complete, LSST will study the stars from the top of Cerro Pachón. M. Park/Inigo Films/LSST/AURA/NSF.

Keeping the instructions clear

LSST has two main objectives: immediate data processing and long-term data aggregation.

In the very short term—in the first 60 seconds after the LSST captures an image, to be precise—the National Center for Supercomputing Applications in Illinois will process the image.


It will send alerts to scientists who study supernovas, asteroids and other quickly-changing phenomena if there have been any changes to that portion of the sky when compared to a reference image. [See Data Journey above.]

In the long term, LSST will create comprehensive catalogs of the telescope’s observations—both the photographs themselves and tables of data extracted from them—to be published yearly.

LSST’s data management team has people working on both of these objectives.

The sheer magnitude of the data collected, the size of the team, and the number of different people and organizations who will want to access the data all pose challenges to the group. Making sure they have good documentation—human-readable information about what each piece of code is doing and how to use it—is one of them.

“The most popular projects out there have been popular not because their code is implemented the best way—they’re popular because their documentation is the best and easiest,” AlSayyad says.

Documentation is important for both scientists who will use LSST data and for the data management team working on code for use within the LSST project.

The team has a developer guide and regular code reviews to help keep coding practices consistent. Any team member can initiate requests for comments and modifications of policies.

With members of the team spread out geographically, the developer guide helps keep everyone on the same page from a distance. “I don’t get to go down the hall to help them with something,” says Jonathan Sick, a member of the Science Quality and Reliability Engineering team, which is based at NSF’s National Optical-Infrared Astronomy Research Laboratory in Tucson. “I have to spend a lot of time literally documenting how to document.”

A common challenge

Another challenge facing the LSST data management team is deciding what technologies to use. “Whatever you choose, it needs to be supported in the future, and it needs to be widely used in the future,” AlSayyad says.

That is not only to address the needs of scientists wanting to study LSST data when it is collected, but also to address the long-term needs of the profession, she says.

AlSayyad and the other project managers want to make sure early-career members find their time at LSST valuable whether they eventually wind up as astronomy faculty or in data science, programming, or other jobs and to make the platform useful for astronomy students who may be accessing LSST data years from now. “We understand that academia is a pyramid, and not everybody who majors in astronomy as an undergraduate is going to become faculty,” she says.

LSST is making use of the growing availability of cloud computing platforms. The team’s Science Platform, based on the JupyterLab software development environment, will allow anyone to run their code with LSST’s data right from their web browsers—no locally saved data required. The LSST Education and Public Outreach team is also working to make parts of the environment as user-friendly as possible so it can be used in classrooms and for citizen science projects.

It is nearly impossible to predict how technology will change over the course of LSST’s mission, so flexibility is key, says Fritz Mueller, a technical manager based at DOE’s SLAC National Accelerator Laboratory. “We have to be prepared to change and evolve,” he says.

One of the team’s priorities is to make sure that their design decisions do not commit them to a single way of dealing with the data. “You try to keep the individual pieces as flexible and general-purpose as you can, so that if you find you need to reorganize them later, that’s possible,” he says.

Financial support for LSST comes from the US National Science Foundation, the US Department of Energy and private funding raised by the LSST Corporation.

LSST funding requires that all LSST software be open source, meaning that anyone can freely use and modify the code. A major goal of the data management group is to deliver software that is as flexible as possible, allowing scientists to adapt the software easily for new types of analyses that were not built in from the beginning.

“The whole mission of a survey telescope is that you’re not necessarily making the discoveries yourself,” says Nate Lust, a member of the team in Princeton. “The LSST project is a community tool for all scientists to make discoveries. The same is true for all of its software.”

See the full article here .


Please help promote STEM in your local schools.

Stem Education Coalition

Symmetry is a joint Fermilab/SLAC publication.

#astronomy, #astrophysics, #basic-research, #cosmology, #instead-of-bringing-the-data-to-scientists-lsst-will-need-to-bring-scientists-to-the-data, #supercomputing, #symmetry-magazine, #the-large-synoptic-sky-survey-will-collect-so-much-data-that-data-scientists-needed-to-figure-out-new-ways-for-astronomers-to-access-it, #the-lsst-vera-rubin-survey-telescope

From UC Santa Cruz: “Powerful new supercomputer supports campus research in physical sciences”

UC Santa Cruz

From UC Santa Cruz

October 30, 2019
Tim Stephens

UCSC faculty are using the new system for research in astrophysics, climate science, materials science, chemistry, and other fields.

A new supercomputer at UC Santa Cruz is providing a state-of-the-art high-performance computing system for researchers in a wide range of fields. The new system, called lux, is substantially more powerful than previous campus supercomputers and was designed with the latest technologies to enable advanced computational studies in areas such as climate modeling, astrophysical simulations, and machine learning.

Astrophysicist Brant Robertson (left) and team members Nicholas Brummell (Applied Mathematics) and Brad Smith (Computer Science and Engineering) with the new lux supercomputer recently installed in the UCSC Data Center. (Photos by C. Lagattuta)

UCSC Lux supercomputer

“I’m excited about leveraging these new technologies to do computational studies we weren’t able to do before at UC Santa Cruz,” said Brant Robertson, associate professor of astronomy and astrophysics.

Robertson led a team of 20 UCSC faculty members from six departments to put together a proposal for the project, which won a $1.5 million grant from the National Science Foundation’s Major Research Instrumentation program. The team includes faculty in the Departments of Astronomy & Astrophysics, Chemistry & Biochemistry, Earth and Planetary Sciences, Physics, Applied Mathematics, and Computer Science & Engineering.

High-performance computing has become an increasingly important tool for researchers throughout the physical sciences. Sophisticated computer simulations can be used to model extremely complex phenomena, from the behavior of Earth’s climate system to the evolution of galaxies. In addition, scientists in a growing number of fields are applying the computationally intensive techniques of machine learning to problems involving large datasets.

“In astronomy, we are just starting to deploy deep learning at large scale. When we are able to automate the analysis of astronomical images from large surveys, it will revolutionize how we do astronomy,” Robertson said.

Although researchers may have access to much bigger supercomputers than lux at national computing facilities operated by the National Science Foundation and the Department of Energy, Robertson said it is crucial for UC Santa Cruz to have its own local system.

“If you want to develop code to run on the largest supercomputer in the world, you need to have a local system that has the same high-end components. We designed lux so that it can serve as a springboard for our researchers to get time on the national systems,” he said.

Lux is also important for training students in the latest computational techniques. It will be available to students in advanced computational courses and programs such as the Lamat program in computational astrophysics, as well as visiting scientists and participants in summer programs.

Robertson said there are already about 100 different accounts on the lux system. “People are up and running on it,” he said.

See the full article here .


Please help promote STEM in your local schools.

Stem Education Coalition

UCSC Lick Observatory, Mt Hamilton, in San Jose, California, Altitude 1,283 m (4,209 ft)


UC Observatories Lick Autmated Planet Finder, fully robotic 2.4-meter optical telescope at Lick Observatory, situated on the summit of Mount Hamilton, east of San Jose, California, USA

UCO Lick Shane Telescope
UCO Lick Shane Telescope interior
Shane Telescope at UCO Lick Observatory, UCSC

UC Santa Cruz campus
The University of California, Santa Cruz opened in 1965 and grew, one college at a time, to its current (2008-09) enrollment of more than 16,000 students. Undergraduates pursue more than 60 majors supervised by divisional deans of humanities, physical & biological sciences, social sciences, and arts. Graduate students work toward graduate certificates, master’s degrees, or doctoral degrees in more than 30 academic fields under the supervision of the divisional and graduate deans. The dean of the Jack Baskin School of Engineering oversees the campus’s undergraduate and graduate engineering programs.

UCSC is the home base for the Lick Observatory.

Lick Observatory's Great Lick 91-centimeter (36-inch) telescope housed in the South (large) Dome of main building
Lick Observatory’s Great Lick 91-centimeter (36-inch) telescope housed in the South (large) Dome of main building

#applied-research-technology, #basic-research, #lux-supercomputer, #supercomputing, #ucsc

From insideHPC: “ARCHER2 to be first Cray Shasta System in Europe”

From insideHPC

October 22, 2019

Today Cray, a Hewlett Packard Enterprise company, announced a £48 million contract award in the UK to expand its high-performance computing capabilities with Cray’s next-generation Shasta supercomputer. The new ARCHER2 supercomputer will be the first Shasta system announced in EMEA and the second system worldwide used for academic research. ARCHER2 will be the UK’s most powerful supercomputer and will be equipped with the revolutionary Slingshot interconnect, Cray ClusterStor high-performance storage, the Cray Shasta Software platform, and 2nd Gen AMD EPYC processors. The new supercomputer will be 11X higher performance than its predecessor, ARCHER.


UK Research and Innovation (UKRI) has once again contracted the team at CRAY to build their follow-up to the Archer supercomputer. Archer 2 is reported to offer up to 11x the throughput of the previous Archer supercomputer put into service back in late 2013. Archer 2 is going to be powered by 12,000 EPYC Rome 64 Core CPUs with 5,848 compute nodes, each having two of the 64 core behemoths. The total core count is 748,544 ( 1,497,088 threads) and 1.57PB for the entire system. The CPU speed is listed as 2.2GHz, which we must assume they are running off of the base clock, so that would be EPYC 7742 CPUs with a 225W TDP. These sorts of specs are insane but also will make some significant heat. Archer 2 will be cooled by 23 Shasta Mountain direct liquid cooling and associated liquid cooling cabinets. The back end for connectivity is Cray’s next-gen slingshot 100Gbps network compute groups. AMD GPUs are part of this array, but the information I have not found yet on which GPU units from AMD will be used. Estimated peak performance is 28 PFLOP/s and the transition for the Archer to the Archer 2 will begin in Q1 2020 and be completed late 1H 2020 as long as things go as planned.

“ARCHER2 will be an important resource for the UK’s research community, providing them with the capability to pursue investigations which are not possible using current resources, said Lynn Gladden, executive chair, professor at the Engineering and Physical Sciences Research Council (ESPRC). “The new system delivered by Cray will greatly increase the potential for researchers to make discoveries across fields such as physics, chemistry, healthcare and technology development.”

The new Cray Shasta-based ARCHER2 system will replace the existing ARCHER Cray XC30 in 2020 and be an even greater capability resource for academic researchers and industrial users from the UK, Europe and the rest of the world. At rates previously unattainable, the new supercomputer will achieve 11X higher performance with only a 27% increase in grid power. The ARCHER2 project provides resources for exploration in research disciplines including oil and gas, sustainability and natural resources, mental and physical health, oceanography, atomistic structures, and technology advancement.

“We’re pleased to continue supporting UKRI’s mission and provide the most advanced high-end computing resources for the UK’s science and research endeavors,” said Peter Ungaro, president and CEO at Cray, a Hewlett Packard Enterprise company. “As traditional modeling and simulation applications and workflows converge with AI and analytics, a new Exascale Era architecture is required. Shasta will uniquely provide this new capability and ARCHER2 will be the first of its kind in Europe, as its next-gen architecture will provide UK and neighboring scientists and researchers the ability to meet their research requirements across a broad range of disciplines, faster.”

The new Shasta system will be the third Cray supercomputer delivered to UKRI, with the previous systems being HECToR and ARCHER. ARCHER2 will be supported by 2nd Gen AMD EPYC processors.


“AMD is incredibly proud to continue our collaboration with Cray to deliver what will be the most powerful supercomputer in the UK, helping to process data faster and reduce the time it takes to reach critical scientific conclusions,” said Forrest Norrod, senior vice president and general manager, AMD Datacenter and Embedded Systems Group. “Investments in high-performance computing technology are imperative to keep up with today’s increasingly complex problems and explosive data growth. The 2nd Gen AMD EPYC processors paired with Cray Shasta will provide a powerful resource for the next generation of research in the UK when ARCHER2 is delivered next year.”

See the full article here .


Please help promote STEM in your local schools.

Stem Education Coalition

Founded on December 28, 2006, insideHPC is a blog that distills news and events in the world of HPC and presents them in bite-sized nuggets of helpfulness as a resource for supercomputing professionals. As one reader said, we’re sifting through all the news so you don’t have to!

If you would like to contact me with suggestions, comments, corrections, errors or new company announcements, please send me an email at Or you can send me mail at:

2825 NW Upshur
Suite G
Portland, OR 97239

Phone: (503) 877-5048

#applied-research-technology, #cray-archer2, #cray-shasta-architecture, #insidehpc, #supercomputing

From insideHPC: “Supercomputing the Building Blocks of the Universe”

From insideHPC

October 13, 2019

In this special guest feature, ORNL profiles researcher Gaute Hagen, who uses the Summit supercomputer to model scientifically interesting atomic nuclei.

Gaute Hagen uses ORNL’s Summit supercomputer to model scientifically interesting atomic nuclei. To validate models, he and other physicists compare computations with experimental observations. Credit: Carlos Jones/ORNL

At the nexus of theory and computation, physicist Gaute Hagen of the Department of Energy’s Oak Ridge National Laboratory runs advanced models on powerful supercomputers to explore how protons and neutrons interact to “build” an atomic nucleus from scratch. His fundamental research improves predictions about nuclear energy, nuclear security and astrophysics.

“How did matter that forms our universe come to be?” asked Hagen. “How does matter organize itself based on what we know about elementary particles and their interactions? Do we fully understand how these particles interact?”

The lightest nuclei, hydrogen and helium, formed during the Big Bang. Heavier elements, up to iron, are made in stars by progressively fusing those lighter nuclei. The heaviest nuclei form in extreme environments when lighter nuclei rapidly capture neutrons and undergo beta decays.

For example, building nickel-78, a neutron-rich nucleus that is especially strongly bound, or “doubly magic,” requires 28 protons and 50 neutrons interacting through the strong force. “To solve the Schrödinger equation for such a huge system is a tremendous challenge,” Hagen said. “It is only possible using advanced quantum mechanical models and serious computing power.”

Through DOE’s Scientific Discovery Through Advanced Computing program, Hagen participates in the NUCLEI project to calculate nuclear structure and reactions from first principles; its collaborators represent 7 universities and 5 national labs. Moreover, he is the lead principal investigator of a DOE Innovative and Novel Computational Impact on Theory and Experiment award of time on supercomputers at Argonne and Oak Ridge National Laboratories for computations that complement part of the physics addressed under NUCLEI.

Theoretical physicists build models and run them on supercomputers to simulate the formation of atomic nuclei and study their structures and interactions. Theoretical predictions can then be compared with data from experiments at new facilities producing increasingly neutron-rich nuclei. If the observations are close to the predictions, the models are validated.

‘Random walk’

“I never planned to become a physicist or end up at Oak Ridge,” said Hagen, who hails from Norway. “That was a random walk.”

Graduating from high school in 1994, he planned to follow in the footsteps of his father, an economics professor, but his grades were not good enough to get into the top-ranked Norwegian School of Economics in Bergen. A year of mandatory military service in the King’s Guard gave Hagen fresh perspective on his life. At 20, he entered the University of Bergen and earned a bachelor’s degree in the philosophy of science. Wanting to continue for a doctorate, but realizing he lacked math and science backgrounds that would aid his dissertation, he signed up for classes in those fields—and a scientist was born. He went on to earn a master’s degree in nuclear physics.

Entering a PhD program, he used pen and paper or simple computer codes for calculations of the Schrödinger equation pertaining to two or three particles. One day his advisor introduced him to University of Oslo professor Morten Hjorth-Jensen, who used advanced computing to solve physics problems.

“The fact that you could use large clusters of computers in parallel to solve for several tens of particles was intriguing to me,” Hagen said. “That changed my whole perspective on what you can do if you have the right resources and employ the right methods.”

Hagen finished his graduate studies in Oslo, working with Hjorth-Jensen and taking his computing class. In 2005, collaborators of his new mentor—ORNL’s David Dean and the University of Tennessee’s Thomas Papenbrock—sought a postdoctoral fellow. A week after receiving his doctorate, Hagen found himself on a plane to Tennessee.

For his work at ORNL, Hagen used a numerical technique to describe systems of many interacting particles, such as atomic nuclei containing protons and neutrons. He collaborated with experts worldwide who were specializing in different aspects of the challenge and ran his calculations on some of the world’s most powerful supercomputers.

“Computing had taken such an important role in the work I did that having that available made a big difference,” he said. In 2008, he accepted a staff job at ORNL.”

That year Hagen found another reason to stay in Tennessee—he met the woman who became his wife. She works in TV production and manages a vintage boutique in downtown Knoxville.

Hagen, his wife and stepson spend some vacations at his father’s farm by the sea in northern Norway. There the physicist enjoys snowboarding, fishing and backpacking, “getting lost in remote areas, away from people, where it’s quiet and peaceful. Back to the basics.”


Hagen won a DOE early career award in 2013. Today, his research employs applied mathematics, computer science and physics, and the resulting descriptions of atomic nuclei enable predictions that guide earthly experiments and improve understanding of astronomical phenomena.

A central question he is trying to answer is: what is the size of a nucleus? The difference between the radii of neutron and proton distributions—called the “neutron skin”— has implications for the equation-of-state of neutron matter and neutron stars.

In 2015, a team led by Hagen predicted properties of the neutron skin of the calcium-48 nucleus; the results were published in Nature Physics. In progress or planned are experiments by others to measure various neutron skins. The COHERENT experiment at ORNL’s Spallation Neutron Source did so for argon-40 by measuring how neutrinos—particles that interact only weakly with nuclei—scatter off of this nucleus. Studies of parity-violating electron scattering on lead-208 and calcium-48—topics of the PREX2 and CREX experiments, respectively—are planned at Thomas Jefferson National Accelerator Facility.

One recent calculation in a study Hagen led solved a 50-year-old puzzle about why beta decays of atomic nuclei are slower than expected based on the beta decays of free neutrons. Other calculations explore isotopes to be made and measured at DOE’s Facility for Rare Isotope Beams, under construction at Michigan State University, when it opens in 2022.

Hagen’s team has made several predictions about neutron-rich nuclei observed at experimental facilities worldwide. For example, 2016 predictions for the magicity of nickel-78 were confirmed at RIKEN in Japan and published in Nature this year. Now the team is developing methods to predict behavior of neutron-rich isotopes beyond nickel-78 to find out how many neutrons can be added before a nucleus falls apart.

“Progress has exploded in recent years because we have methods that scale more favorably with the complexity of the system, and we have ever-increasing computing power,” Hagen said. At the Oak Ridge Leadership Computing Facility, he has worked on Jaguar (1.75 peak petaflops), Titan (27 peak petaflops) and Summit [above] (200 peak petaflops) supercomputers. “That’s changed the way that we solve problems.”

ORNL OCLF Jaguar Cray Linux supercomputer

ORNL Cray XK7 Titan Supercomputer, once the fastest in the world, to be decommissioned

His team currently calculates the probability of a process called neutrino-less double-beta decay in calcium-48 and germanium-76. This process has yet to be observed but if seen would imply the neutrino is its own anti-particle and open a path to physics beyond the Standard Model of Particle Physics.

Looking to the future, Hagen eyes “superheavy” elements—lead-208 and beyond. Superheavies have never been simulated from first principles.

“Lead-208 pushes everything to the limits—computing power and methods,” he said. “With this next generation computer, I think simulating it will be possible.”

See the full article here .


Please help promote STEM in your local schools.

Stem Education Coalition

Founded on December 28, 2006, insideHPC is a blog that distills news and events in the world of HPC and presents them in bite-sized nuggets of helpfulness as a resource for supercomputing professionals. As one reader said, we’re sifting through all the news so you don’t have to!

If you would like to contact me with suggestions, comments, corrections, errors or new company announcements, please send me an email at Or you can send me mail at:

2825 NW Upshur
Suite G
Portland, OR 97239

Phone: (503) 877-5048

#supercomputing-the-building-blocks-of-the-universe, #applied-research-technology, #insidehpc, #ornl, #ornl-ibm-summit-supercomputer, #supercomputing

From insideHPC: “Harvard Names New Lenovo HPC Cluster after Astronomer Annie Jump Cannon”

From insideHPC

October 9, 2019

Harvard has deployed a liquid-cooled supercomputer from Lenovo at it’s FASRC computing center. The system, named “Cannon” in honor of astronomer Annie Jump Cannon, is a large-scale HPC cluster supporting scientific modeling and simulation for thousands of Harvard researchers.

Assembled with the support of the Faculty of Arts and Sciences, but since branching out to serve many Harvard units, Cannon occupies more than 10,000 square feet with hundreds of racks spanning three data centers separated by 100 miles. The primary compute is housed in MGHPCC, our green (LEED Platinum) data center in Holyoke, MA. Other systems, including storage, login, virtual machines, and specialty compute, are housed in our Boston and Cambridge facilities.


“This new cluster will have 30,000 cores of Intel 8268 “Cascade Lake” processors. Each node will have 48 cores and 192 GB of RAM. The interconnect is HDR 100 Gbps Infiniband (IB) connected in a single Fat Tree with 200 Gbps IB core. The entire system is water cooled which will allow us to run these processors at a much higher clock rate of ~3.4GHz. In addition to the general purpose compute resources we are also installing 16 SR670 servers each with four Nvidia V100 GPUs and 384 GB of RAM all connected by HDR IB.”



Compute: The Cannon cluster is primarily comprised of 670 Lenovo SD650 NeXtScale servers, part of their new liquid-cooled Neptune line. Each chassis unit contains two nodes, each containing two Intel 8268 “Cascade Lake” processors and 192GB RAM per node. The nodes are interconnected by HDR 100 Gbps Infiniband (IB) in a single Fat Tree with a 200 Gbps IB core. The liquid cooling allows for efficient heat extraction while running higher clock speeds.
Storage: FASRC now maintains over 40 PB of storage, and this keeps growing. Robust home directories are housed on enterprise-grade Isilon storage, while faster Lustre filesystems serve more performance-driven needs such as scratch and research shares. Our middle tier laboratory storage uses a mix of Lustre, Gluster and NFS filesystems. See our storage page for more details.
Interconnect: Odyssey has two underlying networks: A traditional TCP/IP network and low-latency InfiniBand networks that enable high-throughput messaging for inter-node parallel-computing and fast access to Lustre mounted storage. The IP network topology connects the three data centers together and presents them as a single contiguous environment to FASRC users.
Software: The core operating system is CentOS. FASRC maintains the configuration of the cluster and all related machines and services via Puppet. Cluster job scheduling is provided by SLURM (Simple Linux Utility for Resource Management) across several shared partitions, processing approximately 29,000,000 jobs per year.


See the full article here .


Please help promote STEM in your local schools.

Stem Education Coalition

Founded on December 28, 2006, insideHPC is a blog that distills news and events in the world of HPC and presents them in bite-sized nuggets of helpfulness as a resource for supercomputing professionals. As one reader said, we’re sifting through all the news so you don’t have to!

If you would like to contact me with suggestions, comments, corrections, errors or new company announcements, please send me an email at Or you can send me mail at:

2825 NW Upshur
Suite G
Portland, OR 97239

Phone: (503) 877-5048

#harvard-names-new-lenovo-hpc-cluster-after-astronomer-annie-jump-cannon, #applied-research-technology, #harvard-university, #insidehpc, #supercomputing

From MIT News: “Lincoln Laboratory’s new artificial intelligence supercomputer is the most powerful at a university”

MIT News

From MIT News

September 27, 2019
Kylie Foy | Lincoln Laboratory

The new TX-GAIA (Green AI Accelerator) computing system at the Lincoln Laboratory Supercomputing Center (LLSC) has been ranked as the most powerful artificial intelligence supercomputer at any university in the world.

HPE INTEL NVIDIA TX-GAIA Supercomputer at MIT Lincoln Laboratory Supercomputing Center (LLSC)

Lincoln Laboratory Supercomputing Center | MIT Lincoln Laboratory

The ranking comes from TOP500, which publishes a list of the top supercomputers in various categories biannually. The system, which was built by Hewlett Packard Enterprise, combines traditional high-performance computing hardware — nearly 900 Intel processors — with hardware optimized for AI applications — 900 Nvidia graphics processing unit (GPU) accelerators.

“We are thrilled by the opportunity to enable researchers across Lincoln and MIT to achieve incredible scientific and engineering breakthroughs,” says Jeremy Kepner, a Lincoln Laboratory fellow who heads the LLSC. “TX-GAIA will play a large role in supporting AI, physical simulation, and data analysis across all laboratory missions.”

TOP500 rankings are based on a LINPACK Benchmark, which is a measure of a system’s floating-point computing power, or how fast a computer solves a dense system of linear equations. TX-GAIA’s TOP500 benchmark performance is 3.9 quadrillion floating-point operations per second, or petaflops (though since the ranking was announced in June 2019, Hewlett Packard Enterprise has updated the system’s benchmark to 4.725 petaflops). The June TOP500 benchmark performance places the system No. 1 in the Northeast, No. 20 in the United States, and No. 51 in the world for supercomputing power. The system’s peak performance is more than 6 petaflops.

But more notably, TX-GAIA has a peak performance of 100 AI petaflops, which makes it No. 1 for AI flops at any university in the world. An AI flop is a measure of how fast a computer can perform deep neural network (DNN) operations. DNNs are a class of AI algorithms that learn to recognize patterns in huge amounts of data. This ability has given rise to “AI miracles,” as Kepner puts it, in speech recognition and computer vision; the technology is what allows Amazon’s Alexa to understand questions and self-driving cars to recognize objects in their surroundings. The more complex these DNNs grow, the longer it takes for them to process the massive datasets they learn from. TX-GAIA’s Nvidia GPU accelerators are specially designed for performing these DNN operations quickly.

TX-GAIA is housed in a new modular data center, called an EcoPOD, at the LLSC’s green, hydroelectrically powered site in Holyoke, Massachusetts. It joins the ranks of other powerful systems at the LLSC, such as the TX-E1, which supports collaborations with the MIT campus and other institutions, and TX-Green, which is currently ranked 490th on the TOP500 list.

Kepner says that the system’s integration into the LLSC will be completely transparent to users when it comes online this fall. “The only thing users should see is that many of their computations will be dramatically faster,” he says.

Among its AI applications, TX-GAIA will be tapped for training machine learning algorithms, including those that use DNNs. It will more quickly crunch through terabytes of data — for example, hundreds of thousands of images or years’ worth of speech samples — to teach these algorithms to figure out solutions on their own. The system’s compute power will also expedite simulations and data analysis. These capabilities will support projects across the laboratory’s R&D areas, such as improving weather forecasting, accelerating medical data analysis, building autonomous systems, designing synthetic DNA, and developing new materials and devices.

TX-GAIA, which is also ranked the No. 1 system in the U.S. Department of Defense, will also support the recently announced MIT-Air Force AI Accelerator. The partnership will combine the expertise and resources of MIT, including those at the LLSC, and the U.S. Air Force to conduct fundamental research directed at enabling rapid prototyping, scaling, and application of AI algorithms and systems.

See the full article here .

Please help promote STEM in your local schools.

Stem Education Coalition

MIT Seal

The mission of MIT is to advance knowledge and educate students in science, technology, and other areas of scholarship that will best serve the nation and the world in the twenty-first century. We seek to develop in each member of the MIT community the ability and passion to work wisely, creatively, and effectively for the betterment of humankind.

MIT Campus

#applied-research-technology, #hpe-intel-nvidia-tx-gaia-supercomputer-at-mit-lincoln-laboratory-supercomputing-center-llsc, #mit-lincoln-laboratory, #supercomputing