From The University of Michigan: “Visualizing nanoscale structures in real time”

U Michigan bloc

From The University of Michigan

8.18.22 [Received via Brookhaven Laboratory 9.16.22.]
Written by Jim Lynch | College of Engineering

Media contact
Kate McAlpine
Research News Editor
(734) 647-7087

A real-time reconstruction of platinum nanoparticles on a carbon nanowire produced with the weighted back projection algorithm in tomviz.

Computer chip designers, materials scientists, biologists and other scientists now have an unprecedented level of access to the world of nanoscale materials thanks to 3D visualization software that connects directly to an electron microscope. It enables researchers to see and manipulate 3D visualizations of nanomaterials in real time.

Developed by a University of Michigan-led team of engineers and software developers, the capabilities are included in a new beta version of tomviz, an open-source 3D data visualization tool that’s already used by tens of thousands of researchers. The new version reinvents the visualization process, making it possible to go from microscope samples to 3D visualizations in minutes instead of days.

In addition to generating results more quickly, the new capabilities enable researchers to see and manipulate 3D visualizations during an ongoing experiment. That could dramatically speed research in fields like microprocessors, electric vehicle batteries, lightweight materials and many others.

“It has been a longstanding dream of the semiconductor industry, for example, to be able to do tomography in a day, and here we’ve cut it to less than an hour,” said Robert Hovden, assistant professor of materials science and engineering at U-M and corresponding author on the study published in Nature Communications [below]. “You can start interpreting and doing science before you’re even done with an experiment.”

A real-time reconstruction of cobalt phosphate hyberbranched nanoparticles produced with the simultaneous iterative reconstruction technique algorithm in tomviz.

This rendering of platinum nanoparticles on a carbon support shows how tomviz interprets microscopy data as it’s created, resolving from a shadowy image to a detailed rendering.

Hovden explains that the new software pulls data directly from an electron microscope as it’s created and displays results immediately, a fundamental change from previous versions of tomviz. In the past, researchers gathered data from the electron microscope, which takes hundreds of two-dimensional projection images of a nanomaterial from several different angles.

Next, Hovden and colleagues took the projections back to the lab to interpret and prepare them before feeding them to tomviz, which would take several hours to generate a 3D visualization of an object. The entire process took days to a week, and a problem with one step of the process often meant starting over.

The new version of tomviz does all the interpretation and processing on the spot. Researchers get a shadowy but useful 3D render within a few minutes, which gradually improves into a detailed visualization.

“When you’re working in an invisible world like nanomaterials, you never really know what you’re going to find until you start seeing it,” Hovden said. “So the ability to begin interpreting and making adjustments while you’re still on the microscope makes a huge difference in the research process.”

The sheer speed of the new process could also be useful in industry—semiconductor chip makers, for example, could use tomography to run tests on new chip designs, looking for failures in 3D nanoscale circuitry far too small to see. In the past, the tomography process was too slow to run the hundreds of tests required in a commercial facility, but Hovden believes tomviz could change that.

Hovden emphasizes that tomviz can be run on a standard consumer-grade laptop. It can connect to newer or older models of electron microscopes. And because it’s open-source, the software itself is accessible to everyone.

“Open-source software is a great tool for empowering science globally. We made the connection between tomviz and the microscope agnostic to the microscope manufacturer,” he said. “And because the software only looks at the data from the microscope, it doesn’t care whether that microscope is the latest model at U-M or a 20-year-old machine.”

This diagram illustrates the process of pulling two-dimensional projection images from an electron microscope and rendering them into a three-dimensional visualization.

To develop the new capabilities, the U-M team drew on its longstanding partnership with software developer Kitware and also brought on a team of scientists who work at the intersection of data science, materials science and microscopy. At the start of the process, Hovden worked with Marcus Hanwell of Kitware and The DOE’s Brookhaven National Laboratory to hone the idea of a version of tomviz that would enable real-time visualization and experimentation.

Then, Hovden and Kitware’s developers collaborated with U-M materials science and engineering graduate researcher Jonathan Schwartz, microscopy researcher Yi Jiang and machine learning and materials science expert Huihuo Zheng, both of The DOE’s Argonne National Laboratory, to build algorithms that could quickly and accurately turn electron microscopy images into 3D visualizations.

Once the algorithms were complete, Cornell University professor of applied and engineering physics David Muller and Peter Ericus, a staff scientist at the The DOE’s Berkeley Lab’s Molecular Foundry, worked with Hovden to design a user interface that would support the new capabilities.

Finally, Hovden teamed up with materials science and engineering professor Nicholas Kotov, undergraduate data scientist Jacob Pietryga, biointerfaces research fellow Anastasiia Visheratina and chemical engineering research fellow Prashant Kumar, all at U-M, to synthesize a nanoparticle that could be used for real-world testing of the new capabilities, to both ensure their accuracy and show off their capabilities.

They settled on a nanoparticle shaped like a helix, about 100 nanometers wide and 500 nanometers long. The new version of tomviz worked as planned; within minutes, it generated an image that was shadowy but detailed enough for the researchers to make out key details like the way the nanoparticle twists, known as chirality. About 30 minutes later, the shadows resolved into a detailed, three-dimensional visualization.

A screenshot from tomviz 2.0.

The source code for the new beta version of tomviz is freely available for download at GitHub. Hovden believes it will open new possibilities to fields beyond materials-related research; fields like biology are also poised to benefit from access to real-time electron tomography. He also hopes the project’s “software as science” approach will spur new innovation across the fields of science and software development.

“We really have an interdisciplinary approach to research at the intersections of computer science, material science, physics, chemistry,” Hovden said. “It’s one thing to create really cool algorithms that only you and your graduate students know how to use. It’s another thing if you can enable labs across the world to do these state-of-the-art things.”

Kitware collaborators on the project were Chris Harris, Brianna Major, Patrick Avery, Utkarsh Ayachit, Berk Geveci, Alessandro Genova and Hanwell. Kotov is also the Irving Langmuir Distinguished University Professor of Chemical Sciences and Engineering, Joseph B. and Florence V. Cejka Professor of Engineering, and a professor of chemical engineering and macromolecular science and engineering.

“I’m excited for all the new science discoveries and 3D visualizations that will come out of the material science and microscopy community with our new real-time tomography framework,” Schwartz said.

Science paper:
Nature Communications

See the full article here .


Please support STEM education in your local school system

Stem Education Coalition

U MIchigan Campus

The University of Michigan is a public research university located in Ann Arbor, Michigan, United States. Originally, founded in 1817 in Detroit as the Catholepistemiad, or University of Michigania, 20 years before the Michigan Territory officially became a state, the University of Michigan is the state’s oldest university. The university moved to Ann Arbor in 1837 onto 40 acres (16 ha) of what is now known as Central Campus. Since its establishment in Ann Arbor, the university campus has expanded to include more than 584 major buildings with a combined area of more than 34 million gross square feet (781 acres or 3.16 km²), and has two satellite campuses located in Flint and Dearborn. The University was one of the founding members of the Association of American Universities.

Considered one of the foremost research universities in the United States, the university has very high research activity and its comprehensive graduate program offers doctoral degrees in the humanities, social sciences, and STEM fields (Science, Technology, Engineering and Mathematics) as well as professional degrees in business, medicine, law, pharmacy, nursing, social work and dentistry. Michigan’s body of living alumni (as of 2012) comprises more than 500,000. Besides academic life, Michigan’s athletic teams compete in Division I of the NCAA and are collectively known as the Wolverines. They are members of the Big Ten Conference.

At over $12.4 billion in 2019, Michigan’s endowment is among the largest of any university. As of October 2019, 53 MacArthur “genius award” winners (29 alumni winners and 24 faculty winners), 26 Nobel Prize winners, six Turing Award winners, one Fields Medalist and one Mitchell Scholar have been affiliated with the university. Its alumni include eight heads of state or government, including President of the United States Gerald Ford; 38 cabinet-level officials; and 26 living billionaires. It also has many alumni who are Fulbright Scholars and MacArthur Fellows.


Michigan is one of the founding members (in the year 1900) of the Association of American Universities. With over 6,200 faculty members, 73 of whom are members of the National Academy and 471 of whom hold an endowed chair in their discipline, the university manages one of the largest annual collegiate research budgets of any university in the United States. According to the National Science Foundation, Michigan spent $1.6 billion on research and development in 2018, ranking it 2nd in the nation. This figure totaled over $1 billion in 2009. The Medical School spent the most at over $445 million, while the College of Engineering was second at more than $160 million. U-M also has a technology transfer office, which is the university conduit between laboratory research and corporate commercialization interests.

In 2009, the university signed an agreement to purchase a facility formerly owned by Pfizer. The acquisition includes over 170 acres (0.69 km^2) of property, and 30 major buildings comprising roughly 1,600,000 square feet (150,000 m^2) of wet laboratory space, and 400,000 square feet (37,000 m^2) of administrative space. At the time of the agreement, the university’s intentions for the space were not set, but the expectation was that the new space would allow the university to ramp up its research and ultimately employ in excess of 2,000 people.

The university is also a major contributor to the medical field with the EKG and the gastroscope. The university’s 13,000-acre (53 km^2) biological station in the Northern Lower Peninsula of Michigan is one of only 47 Biosphere Reserves in the United States.

In the mid-1960s U-M researchers worked with IBM to develop a new virtual memory architectural model that became part of IBM’s Model 360/67 mainframe computer (the 360/67 was initially dubbed the 360/65M where the “M” stood for Michigan). The Michigan Terminal System (MTS), an early time-sharing computer operating system developed at U-M, was the first system outside of IBM to use the 360/67’s virtual memory features.

U-M is home to the National Election Studies and the University of Michigan Consumer Sentiment Index. The Correlates of War project, also located at U-M, is an accumulation of scientific knowledge about war. The university is also home to major research centers in optics, reconfigurable manufacturing systems, wireless integrated microsystems, and social sciences. The University of Michigan Transportation Research Institute and the Life Sciences Institute are located at the university. The Institute for Social Research (ISR), the nation’s longest-standing laboratory for interdisciplinary research in the social sciences, is home to the Survey Research Center, Research Center for Group Dynamics, Center for Political Studies, Population Studies Center, and Inter-Consortium for Political and Social Research. Undergraduate students are able to participate in various research projects through the Undergraduate Research Opportunity Program (UROP) as well as the UROP/Creative-Programs.

The U-M library system comprises nineteen individual libraries with twenty-four separate collections—roughly 13.3 million volumes. U-M was the original home of the JSTOR database, which contains about 750,000 digitized pages from the entire pre-1990 backfile of ten journals of history and economics, and has initiated a book digitization program in collaboration with Google. The University of Michigan Press is also a part of the U-M library system.

In the late 1960s U-M, together with Michigan State University and Wayne State University, founded the Merit Network, one of the first university computer networks. The Merit Network was then and remains today administratively hosted by U-M. Another major contribution took place in 1987 when a proposal submitted by the Merit Network together with its partners IBM, MCI, and the State of Michigan won a national competition to upgrade and expand the National Science Foundation Network (NSFNET) backbone from 56,000 to 1.5 million, and later to 45 million bits per second. In 2006, U-M joined with Michigan State University and Wayne State University to create the the University Research Corridor. This effort was undertaken to highlight the capabilities of the state’s three leading research institutions and drive the transformation of Michigan’s economy. The three universities are electronically interconnected via the Michigan LambdaRail (MiLR, pronounced ‘MY-lar’), a high-speed data network providing 10 Gbit/s connections between the three university campuses and other national and international network connection points in Chicago.