From Science Node: “LIGO and OSG peer into the Dark Energy Camera”

Science Node bloc
Science Node

06 Apr, 2016
Greg Moore

The Laser Interferometer Gravitational-wave Observatory (LIGO) gravitational wave announcement awaits corroborating observation. Using Open Science Grid (OSG) computing resources, they looked to the Dark Energy Camera images for visual evidence of the cosmic collision they detected.

MIT/Caltech Advanced aLIGO Hanford Washington USA installation
MIT/Caltech Advanced aLIGO Hanford Washington USA installation

On September 14, 2015, gravitational waves were directly observed for the first time by both detectors of the Laser Interferometer Gravitational-wave Observatory (LIGO). The detection confirmed Einstein’s proposal in his general theory of relativity. Now, scientists are seeking the wave source.

Gravitational waves. Credit: MPI for Gravitational Physics/W.Benger-Zib
Gravitational waves. Credit: MPI for Gravitational Physics/W.Benger-Zib

Seeking the source

While LIGO, funded by the US National Science Foundation (NSF), can pick out the general direction of the source of gravitational waves, it can’t identify the exact location. So, LIGO scientists are coordinating their measurements with observations made by the Dark Energy Camera (DECam) on the Blanco Telescope at the Cerro Tololo Inter-American Observatory in Chile.

DECam, built at FNAL
NOAO/CTIO Victor M Blanco 4m Telescope which houses the DECam
NOAO/CTIO Victor M Blanco 4m Telescope which houses the DECam at Cerro Tololo

Scientists at Fermilab and other institutions in the Dark Energy Survey use the camera to understand dark energy — a force scientists believe is helping the universe expand. A subset of members known as the Dark Energy Survey-Gravitational Wave (DES-GW) group are using the camera and the Open Science Grid (OSG) to build on LIGO’s groundbreaking findings.

“Our focus primarily is the search for dark energy,” says Marcelle Soares-Santos, associate scientist at the US Department of Energy’s Fermilab. “Since we have experience detecting things through magnetic emissions, we coordinated with LIGO to find a source that we would find useful in our own research. Unfortunately this time we did not see anything, but we are now much better prepared when LIGO becomes active again later this year.”

How to find a needle in a universe-sized haystack

The area of sky DES-GW members observe is very large — 700 square degrees of sky, or about 2,800 times the size of the full moon — and requires rapid image processing. That’s where the OSG comes in. Without OSG, Soares-Santos says they couldn’t keep up.

“For this event, we had something like 4-5,000 jobs. We must break every image down into smaller parts and process them in parallel on the OSG. It is critical to get our results fast — within 24 hours.”

Scientists then analyze their observations with a spectrograph — which is expensive — so it’s important to narrow the choices down to only a few candidates. “At first, our turnaround time was not very fast, but thanks to our close partnership with the computing side here at Fermilab, now it is. We have great confidence that when LIGO observations start again in early August, we will be ready and hopefully see something.”

Kenneth Herner, an application developer and systems analyst at Fermilab, is one of those key partners on the computing side. He makes sure the DES group has as many resources as they need and devotes part of his time to OSG.

“Opportunistic OSG resources really help with the computing needs and the time crunch,” says Herner. “When we submit jobs, we get the first resources that meet the requirements no matter where they may be. We use the CernVM File System to pull in a code repository over HTTP to a local cache on a worker node. It only pulls down what it needs as it needs it. We don’t have to configure each OSG site — it just works. All OSG sites then look the same and all the site has to do is mount a repository.”

In preparation for the LIGO partnership, Herner’s group prepared a code pipeline and made sure everything would work. The LIGO alert came on the 14th. “We had to wait on the telescope—and on top of that an earthquake in Chile,” says Herner. “We worked our plan, checked our code, transferred images from Chile up to the US, and submitted our jobs.”

Almost all the jobs ran at Fermilab, but Herner says they could have gone anywhere on the OSG. “This was our shakedown cruise,” said Herner. “The first event used about 15,000 CPU hours for a full pass over all nights, but with multiple passes and preprocessing it was over 25,000 hours.” Without OSG resources, the group would have taken Fermilab computing resources away from other experiments, he says.

Observing the sources of these gravitational waves will tell Soares-Santos how systems work and give her and her colleagues deeper insight into the physics. “It is quite challenging to observe these events,” says Soares-Santos.

“We have to be quick to respond to see them. We have to be on the spot sooner and it is the computing that makes that possible. We couldn’t do it without the OSG because of the volume of data. We must have massive parallel computing and quick turnaround and hopefully next time we will see something exciting.”

See the full article here .

Please help promote STEM in your local schools.

Stem Education Coalition

Science Node is an international weekly online publication that covers distributed computing and the research it enables.

“We report on all aspects of distributed computing technology, such as grids and clouds. We also regularly feature articles on distributed computing-enabled research in a large variety of disciplines, including physics, biology, sociology, earth sciences, archaeology, medicine, disaster management, crime, and art. (Note that we do not cover stories that are purely about commercial technology.)

In its current incarnation, Science Node is also an online destination where you can host a profile and blog, and find and disseminate announcements and information about events, deadlines, and jobs. In the near future it will also be a place where you can network with colleagues.

You can read Science Node via our homepage, RSS, or email. For the complete iSGTW experience, sign up for an account or log in with OpenID and manage your email subscription from your account preferences. If you do not wish to access the website’s features, you can just subscribe to the weekly email.”