An ambitious project is underway to develop a robot with sufficient intelligence, strength and sensitivity to repair fragmented archaeological remains

Imagine you have a puzzle with 10,000 puzzle pieces but no pictures on the box. In fact, you don’t even have a box—it was destroyed nearly 2,000 years ago. These puzzles are fragments of ancient Roman Pompeii frescoes that were razed to the ground or buried by the eruption of Mount Vesuvius in AD 79. Some fragments are missing, some are broken. They are not precisely cut shapes intended to interlock neatly, but irregular fragments of damage. How would you solve this problem?

Scientists at the Italian Institute of Technology (IIT) have a plan: send in robots. Their project-called RePAIR (Reconstruction of the Past: Artificial Intelligence and Robots and Cultural Heritage)-combines robotics, artificial intelligence and archeology to try to reconstruct the architectural features of Pompeii, otherwise these features will be incomplete, because they either Too complicated, or it requires impossible manpower and time.

Over the next few months, IIT researchers and their collaborators at several international universities will build, train, test, and deploy a robot that will piece together the murals destroyed in the two buildings. In the first building, they knew what the murals should look like, because they were intact until recently. Investigators did not know what the murals in the second building depicted. For decades, its fragments have been in the storage room waiting for someone or something to put them back together.

RePAIR is experimental and may fail, admits Arianna Traviglia, director of the IIT Cultural Heritage and Technology Center in Venice and lead researcher of the project. Its funding comes from a 3.5 million euro grant from the European Commission Fund, which supports venture projects aimed at "new future technologies". Marcello Pelillo, a professor of computer science and artificial intelligence at the University of Venice and project coordinator, said that if the project is successful, the technology can be used to reconstruct various fragmented cultural relics. The restoration of these cultural relics is "unattainable by humans." ".

"Let us hope we can succeed because it has never been tried before," Traviglia said with a smile. "It's really crazy at the highest level."

Robots that have not yet been named by RePAIR must have brains, power, and sensitive touch. The brain is a mixture of high-tech computer vision for scanning mural puzzles, machine learning algorithms for solving puzzles, and human expertise to guide artificial intelligence.

"We believe that using artificial intelligence-based systems is not enough," said Pellillo, whose team is working with a team at Ben Gurion University of the Negev to develop artificial intelligence for puzzle solving. This is why they will receive guidance from archaeologists and art historians on the different styles of frescoes found in Pompeii.

In Pompeii, researchers are manually digitizing each mural fragment in two test sites to create a digital database of RePAIR robots. However, once the entire system is fully operational, "we will let the robot complete the scanning process by itself," Pellillo said.

When robots can handle conventional digitization, humans are free to handle more complex tasks. Graduate students cannot scan 24/7 without food, water or sleep-but robots can. Traviglia said: "Our idea is to automate this time-consuming and boring digitalization of cultural heritage as much as possible."

The humanoid robot consists of a torso and arms and was developed by Nikolaos Tsagarakis and his colleagues at the Humanoid and Human-Centered Mechatronics Laboratory of IIT in Genoa. The robot’s arm is 80 to 100 cm long and weighs 25 to 30 kg, which is about the same size as the upper body of an ordinary person. Its design is based in part on the early "compliant humanoid" robots developed by the Tsargarakis laboratory for use in disaster sites, including WALK-MAN, which explored a damaged building in the Italian village of Amatrice in 2016. The earthquake has killed nearly 300 people.

The hand will be connected to the arm through the socket on the wrist. They were designed in the laboratory of Antonio Bicchi, a senior scientist at Genoa IIT and chairman of robotics at the University of Pisa, who developed manipulators for industrial and prosthetic limbs. In Pompeii, a soft manipulator needs to be extremely careful to grasp, move and position fragments of different sizes and weights, and collect information about them in the process. These hands are like soft but smart gloves, embedded with tactile, kinesthetic and position sensors. Bicchi said that ideally, people would one day be able to wear them on their hands to collect additional data that cannot be obtained by human hands.

Researchers are still working on the final design of the robot (another design abandons the torso and just hangs the arm on a metal frame), but it may be mounted on a slider on the workbench so that it can be easily moved from one position Move the other end of the workspace. After installation, the soft hand will grab the fragments, scan them in 3D, and send the data to the puzzle-solving AI, which will try to find a virtual match. Once the solution is in place, the artificial intelligence will send the data to the hands of the assembled parts.

The first mural puzzle that RePAIR scientists will try to solve with a new robot is part of Schola Armaturarum, the headquarters of a military style association located in Via dell'Abbondanza, Pompeii's main street, where members plan military activities and gladiator fights. 1915 Unearthed, its hall is decorated with trophies and weapons, lined with wooden cabinets, and armor and trophies may have been displayed. It survived the volcanic eruption and the direct impact of an Allied bomb in 1943, but after a heavy rain in Pompeii, part of it collapsed in 2010, including the mural that researchers will try to rebuild. Because they know the original appearance of the mural, this is a good place to test the capabilities of the robot.

If the robot successfully completes the Schola Armaturarum mural, the team will next try to solve the mural puzzle found in the two collapsed rooms of the working painter's house. The building is so named because artisans were painting murals when Mount Vesuvius erupted; archaeologists unearthed their buckets and paintbrushes, as well as the fuzzy outlines of murals waiting for colors, shadows and details that never appeared. Part of the building was destroyed at the time, and it was also severely damaged by the bombing of the Second World War.

Because each collapsed wall represents a mural puzzle, and there are no pictures to refer to, this website presents an even greater challenge. "Imagine you have five different puzzles-one is related to the Eiffel Tower, the other is related to the stadium, etc.," Pellillo said. "You open the box and mix all the pieces together. Then you are asked to solve five puzzles. This is actually what we are going to do."

The goal of their RePAIR scientists is to install the robot in Pompeii in the spring or summer of 2022 (the exact location is still being determined). They have some time to experiment; their grant supports the project until 2025.

Researchers see the future world of robots. Museums and universities around the world hold scattered remains, many of which have been neglected for decades. "We hope that if we successfully complete this project, we will be able to provide a technology that will allow many museums around the world to rebuild large, damaged murals or similar artifacts," said Pellilo. "We mainly focus on mural painting, but this is only a case study. If we succeed, we think we can export this technique to other artifacts, even papyrus, for example."

Jen Pinkowski is a science journalist based in Berlin.

Alyssa Haas, Kelsey Ellis and the United States in conversation

Thomas frank and electronic electrical news

Tess Joss, Beth Zayken, and Jane Christiansen

Discover the science that changes the world. Explore our digital archive dating back to 1845, including articles by more than 150 Nobel Prize winners.

© 2021 Scientific American, a division of Springer Nature America, Inc.

Thanks for reading Scientific American. Knowledge waits.

Already a subscriber? login.

Thanks for reading Scientific American. Create your free account or log in to continue.

Continue reading to subscribe to Scientific American.

You can cancel at any time.