New Storytelling for Archaeological Museums Based on Augmented Reality Glasses

Museums are places where cultural heritage is preserved and, therefore, we can consider them an essential resource to understand our identity, past and future. In the last two decades, they have increased the use of information and communication technologies in a remarkable way with the intention of reaching new audiences and spreading knowledge. With the recent advent of augmented reality devices of the ‘view-through’ type, perceiving and interacting with virtual contents in the form of holograms anchored to the real physical space is now possible. One of the most interesting challenges is to leave the screen aside and interact with digital data in an intuitive way, through voice commands and gestures. This offers a new scenery for experimenting with storytelling creation, a current trend in archaeological museums. In our effort to take advantage of the new capabilities of augmented reality glasses, we have developed different applications with the Microsoft HoloLens How to cite this book chapter: Muñoz, A. and Martí, A. 2020. New Storytelling for Archaeological Museums Based on Augmented Reality Glasses. In: Hageneuer, S. (ed.) Communicating the Past in the Digital Age: Proceedings of the International Conference on Digital Methods in Teaching and Learning in Archaeology (12–13 October 2018). Pp. 85–100. London: Ubiquity Press. DOI: https://doi.org/10.5334/bch.g. License: CC-BY 4.0 86 Communicating the Past in the Digital Age glasses. In this chapter, we explain our experiences in such developments, which have led us to create an innovative storytelling for the archaeological museum of the Almoina in Valencia (Spain), a singular project where we have experimented with a fictional holographic character that attends as a guide to present a story about the city life in Valencia during the time of the Roman Republic. The story is presented as animated sequences with video, 3D reconstructions and music. The visitor conducts the storyline interacting with the marked hotspots that appear over the ruins of the museum.


Introduction
Over the past five years, many articles have dealt with the future of museums, with promises of how technology will help solve some of the existing problems with audience engagement and content presentation. In fact, there are different technological improvements, like the increasing computing power in smartphones, artificial intelligence, geolocation or big data, which lead us to think that a more communicative museum is now possible (Winesmith 2017;Mannion, Sabiescu and Robinson 2015).
Nevertheless, in the last three years a new technological development has appeared that can radically change museums' storytelling, especially when it comes to museums of history, science and archaeology. We refer to the new virtual reality and augmented reality smart glasses developed by powerful companies such as Microsoft, Facebook or Magic Leap. On the one hand, virtual reality technology has already been used successfully in the museum context in order to introduce the audience to recreated virtual living spaces. The Back to Life project, developed by Google in collaboration with the Natural History Museum in London (Clio Awards 2017; Pavid 2016), the Modigliani experience from the Tate Museum (Taylor 2018;Tate 2017) and the permanent room dedicated to VR at the National Museum of Natural History in Paris (Tiercepartie 2018;MNHN 2018) are examples of international awarded projects where virtual reality was used as a powerful tool to present and explain complex processes by transporting visitors to new experimental scenarios where they can play an active role.
On the other hand, augmented reality allows us to mix real objects with digital information at the same perceptual level. In other words, virtual reality transports users to virtual worlds; meanwhile, augmented reality brings the digital representations to our real space.
Although augmented reality (AR) technology is still at its beginning, some recent projects demonstrate its potential for enhance exhibitions: The Musée des Plans-Reliefs presented an interactive model of Mont St Michel where it is possible to go around and explore the model of the castle with interactive information related to it (Laval Virtual 2018); the Petersen Automotive Museum (Microsoft 2017) showed an exhibition of cars with apparent motion thanks to the visual overlaying of digital models; and the Kofun Virtual Guide, made by Keuchi Laboratory of the University of Tokyo, is an AR system prepared to show parts of the ancient capital of Japan, all overlaid on the real landscape environment (Epson 2018;Koetsier 2018).
These examples have in common the use of 'view-through' AR glasses. These devices are prepared to display stereoscopic video, pictures and 3D animations according to the interests of the audience, using voice and hand gestures or even simply interacting unintentionally with their body in the space of the exhibition. This kind of experience can also be shared with other visitors, taking the visit to a new level of collaborative experience that maintains and extends the social essence of museums (Eghbal-Azar et al. 2015: 133-142).
Since VR and AR open the possibility of new ways to interact with heritage, they also require new methodologies and procedures to design successful experiences for any kind of audience.
At the Universitat Politècnica de València, our research group develops AR prototypes that test new interactive-immersive narratives for museums. Our goal is to design tests to explore whether these new immersive experiences can give a better understanding of heritage and its importance in a natural and enjoyable way.
In this chapter, we describe some of those experiences, explaining the challenges we have gathered during the last three years and some advances we have achieved in the implementation of those methodologies in a prototype for a real museum: the archaeological museum in Valencia, La Almoina.

Augmented reality gets into the museum
In the last two decades, museums and heritage sites have increased the use of information and communication technologies (ICT) in a remarkable way, investing large amounts of money and time with the intention of digitising their collections to reach new audiences and spread knowledge (Tallon & Walker 2008;Wang et al. 2008). Certainly, museums are institutions that must continually reinvent themselves if they want to prosper and attract new audiences, to remain relevant centres, and technology is an essential tool to make this happen (Horwitz-Bennett 2010; Panagiotis, Despina & Chrysanthou 2013).
After the commonly adopted integration of online digital resources in the exhibition space using smartphones, a new technological revolution is already changing museum communication. Indeed, the development of new technologies has facilitated access to information, but, beyond this achievement, the way people interact with digital data is also changing, thanks especially to AR devices.
Thanks to the investments that have been developed in the last decades, AR glasses have reached a point of optimal maturity to be inserted in society. The economic predictions of the sector forecast that the investments for the year 2020 will exceed the investment in VR (Digi Capital 2016), which will mean that new and better systems will appear in the market, with applications that will expand their use to other fields still to be discovered .
Some of its key features are the use of 3D stereoscopic visual systems with the ability to track some of the movements of the user (head, hands and eyes) and the space, together with the use of immersive sounds. AR systems permit the incorporation of digital data in a real environment, allowing users to perceive digital recreations without losing the perception of the physical world. When compared with VR, AR approaches are a more naturalised way to interact with data, since they allow contents to be included through layers of information adapted to the real space without overloading it, being very respectful with the original piece and the naturalised experience with the related data.
In 2016, HoloLens glasses appeared on the market to demonstrate for the first time the potential of the new 'view-through' AR devices. Unlike mobile devices -where you look at the screen of the mobile to see a mix between the capture of the camera and superimposed three-dimensional elements -the AR 'view-through' glasses inaugurate a new system of total immersion among the three-dimensional elements and our natural stereoscopic visual experience. One of their most important features is that they are prepared to scan the space continuously so that they can combine real and tangible physical elements with virtual elements while the user moves freely.
The idea of taking information and superimposing it on the real world opens a new path of possibilities that allow us to better understand the information of objects, places or history, making the real world more magical (Burdea & Coiffet 2003;Osterhout 2016).
Indeed, one of the most interesting challenges is the possibility of leaving the screen behind and interacting with digital data in a much more intuitive way, through voice commands and gestures. These devices offer a new scenery with a huge potential for experimenting with storytelling creation, a current trend particularly in archaeological museums.

Developing an immersive storytelling
One of the great advances of digital media is that they allow us to take advantage of non-linear narratives, offering multiple entries to the same topic and extending the limits of linear narration, while placing the visitor at the centre of the event, where they are allowed to take decisions (Gillam 2017). Another common feature is the convergence of media (Jenkins 2008), which applied to the storytelling context allows us to mix different media that previously worked separately, for instance placing images next to text, or even animations, or introducing a musical thread at the same time that we reproduce the discourse with different devices.
Museums are natural storytellers (Johnsson 2006) and, from the very beginning, they have used different techniques to communicate exhibitions with the intention of making objects more accessible to visitors (Roussou et al. 2015;Wong 2015). As cultural storytellers, they need to tell inclusive stories, which breaks the barriers that isolated them from society (Solari 2015).
Just as 19th-century museums made some architectural changes to accommodate a greater number of visitors who could access objects (Bennett 1995), the museums of the 21st century will have to adapt to the new needs of the information society (Witcomb 2003: 115) to include ever more attractive experiences as something necessary for their survival.
With this idea in mind, we were determined to use AR in a museum context to develop different storytelling techniques that could profit from a more natural-physical communication. In that sense, we consider that the most suitable museums for this purpose are the museums of history or archaeology. Even though, in many occasions, they include pieces of artistic value, the truth is that they are mainly constituted by heterogeneous collections that show either the greatness or the history of a country or region. Their narrative is usually associated with the possibility of developing an idea or discourse, which hooks with the possibilities of experiencing storytelling through AR.
When using AR 'view-through' glasses it is possible to reconstruct objects and contexts to better understand other historical episodes. Most societies use, and have used, physical and material objects as examples of ideas or lessons. Objects become a very powerful tool to represent past events in the present. However, with the help of holographic images, together with sound and the ability to move around and interact with the data in the exhibition space, we can create innovative storytelling that may help to engage with the youngest audiences.
Traditionally, museums and heritage sites have used reconstructions of objects together with models, graphics and audio-visual media, such as documents or maps, to maintain a chronological continuity. On many occasions, in museums of history or archaeology, objects are moved from their place of origin and are exposed, decontextualised, in the room. Very often, ruins of buildings are difficult to understand from their degraded state, making very difficult to understand the history of the people who lived there in ancient times. A better understanding of the contents and an improvement of the experience is possible by using holographic glasses, giving life to those places with virtual objects that help the reconstruction of their habitat in context. At the same time, it is necessary to act with scientific rigour, respecting the existing information of the exhibits, so that the recreations do not introduce any historical error. Obviously, history is not an exact science, and the vision of a fact can never be pure or innocent. Methodologically digital recreations should be generated following the lead of curators and experts from the museum.
Traditionally, archaeological museums focus more on exhibiting archaeological remains and objects, with some associated data, than offering a possible interpretation from them. However, in recent years, more and more archaeological museums have tended to develop narratives that help to recreate how these objects were used and by whom. Those narratives often appear as micro events that reconstruct the interpretation that is made of a particular site (Hernández 2010: 22-23).

Designing immersive experiences
In our way to develop inclusive storytelling when using AR, we started to test different methodologies in different exhibition contexts. Our idea was to test techniques, which could use the advantages of AR, in order to determine which are more natural and effective to be implemented in museums in the near future.

Garden
In June 2016, as a first attempt, we designed an experience to explain the value of the painting The Garden of Earthly Delights by Bosch, using the HoloLens glasses. Our goal was to create a motivational experience to increase the interest in the details of this artwork. A sequence of video and animated graphics with a voiceover was produced to explain the artwork, using the authoring software Unity to program the application. The challenge was to create a storytelling mode that would motivate interest in the work, without diminishing the importance of the real object, by adding visual elements, videos and animations projected onto a high-definition, full-size printed reproduction of the famous painting, accompanied with explanatory audio.
However, in the development of the experiment, we had some evidence of the problematic inclusion of overlaying virtual data in part of the painting and how this was negatively affecting the perception of the original artwork. Beyond some undeniable pedagogical value, the discovery of information on the canvas was seen like a game by most of the 10 testers, showing a clear conflict between the real and the virtual that is not present in other platforms, like mobile phones or PCs, where everything is perceived as a digital representation.
Nevertheless, this test helped to dismiss an important observation: one of the first findings was that this technology is not appropriate to be applied directly to overlay works of art, since in the artistic exhibitions it is important to maintain an intimate relationship between the work of art and the viewer, and the screen of the glasses darkens the colours of the painting, together with the digital data, distracting the attention from the content, which is what really matters when speaking about art.
Indeed, museums are institutions that traditionally develop narratives linked to the authenticity of works, tradition and the universality of knowledge. Both works of art and cultural heritage are made up of unique and irreplaceable objects and buildings that, thanks to this, acquire qualitative and quantitative value (Witcomb 2003: 106-107). Including digital information in front of an artwork takes away the attention of the visitor.
Other projects that used the HoloLens to overlay information in front of paintings, like the one developed by the start-up Opuscope in 2016 (Klint 2016), brings us to the same conclusion: the information that overlays the paintings makes it difficult to have a real experience with it. HoloLens glasses have also been used to show paintings from important artists from anywhere, like in the project conducted by the culture platform Boulevard Arts Inc., together with Case Western Reserve University (Case Western Reserve University 2017), showing a way to see paintings in your home in a more attractive way than in art books. Still it cannot replace the feeling of being in front of the real object.
According to Gwyneira Isaac, the technology used to copy the pieces can be understood as something enigmatic, as if it were an enchanted technology, but it affects our relationships with objects (Isaac 2016). In her opinion, these screens not only 'hypnotize' visitors but also place them in an institutionalised atmosphere.
Nevertheless, this first experience helped to draw some conclusions about the usability of the voice commands and gestures when reproducing contents, as well as to start developing the design of the interactive posters and labels together with the experimentation of the video, which we included in later developments.
Since no other universities or companies had presented research on that period on how using gestures or commands with smart glasses in the museum context, we considered this field of research of great interest. Therefore, we began to consider a general exhibition context to test and experiment with the HoloLens glasses and AR interaction models. In February 2017, we presented an AR application called Holomuseum at the biannual Inventions Fair at the Universitat Politècnica de València, aimed at testing many of the new possibilities that AR glasses open for different exhibition contexts.

Holomuseum
Holomuseum was an application designed to facilitate the creation and maintenance of many multimedia experiences in any exhibition hall (Muñoz 2017).
The system came to solve several important problems: on the one hand, those arising from the current limits of technology -the management and maintenance of a collection of virtual contents in real space, and its location in one or several rooms -and, on the other hand, those generated by the absence of a consensus language for the management of AR contents with holographic glasses.
The application was created to be used by designers and curators when creating exhibitions, and by visitors, in the mode of visualisation: on the one hand, it has an editing mode, in which curators can manage the list of threedimensional objects to be displayed, and decide exactly which part of the room will be presented when activated. The items list pointed to data held from the internet containing individual interactive contents ready to be downloaded. On the other hand, the system has a default mode for visits, showing the list of contents like virtual boxes hung on a real wall of the exhibition space, ready to be opened at the visitors' request. The activation and downloading of each item were made by simple voice commands or by a pinch gesture (air tap) on the virtual boxes (Muñoz & Martí 2018).
One of the main challenges of this project was creating a permanent link between the digital contents and their pertinence to a physical space. In the same way, maintaining a visual coherence between the actions of the visitors and the digital states was crucial to blend the digital and the real without conflict. In Holomuseum the audience could remember where they could trigger each content because they were marked with posters in the walls. The real posters acted as anchors for the virtual boxes floating in front of them, helping to understand them as a kind of persistent mixed-reality label. Furthermore, each box was animated when activated to show this action in a way that was easy to  identify, and remained on the wall, opened until the visitor decided to close it with the voice command 'close box' , or just by air-tapping the box again with their fingers. With the intention of making a sample of the usability of the system, and testing it with different users in an exhibition hall, we designed different models suitable for different types of museums: on the one hand, we made prototypes of scientific and technological style, and, on the other hand, we created others of a more artistic or historical nature.
To expose these contents, some panels were printed to serve as physical posters or labels for each element. These panels explained each object. The holographic cubes were physically placed on these posters, which, once activated, were loaded from the internet and flew to the different areas of the room previously selected by the curator. At the centre of the room, a pedestal was placed to serve as a physical support to place digital contents.
Later on, in 2018, the company Holograph also presented an authoring tool to create augmented exhibitions in museums (Surur 2018), but it uses the simple modes of a traditional museum, like labels, without really exploring new ways to interact with digital data.
The development of Holomuseum helped to clarify the acceptance of AR exhibitions and the expectations of visitors when experimenting with the new media for the first time. After getting very positive reactions of the audience, we tried to go one step further with the creation of interactive storytelling specially dedicated to show the potential of AR in a real museum.

Almoina AR
Since archaeological museums usually require recreations of lost items, places or even traditions, they are one of the most suitable kind of museum for experimenting with AR. Although HoloLens have been already used in an archaeological context, the experiences were dedicated just to underline single pieces of the collection, like in Holoforge, where the glasses overlay a polychromic restitution of a bust of Akhenaton (Goguel 2017), in contrast with our more general approach. Thanks to the collaboration with the archaeological museum of the Almoina in Valencia (Spain), we could design an experience to cover the first steps of the history of this old city, taking on the challenge of engaging the audience with AR figures and recreations of the lost buildings over the remains of the ruins.
The project was conceived around the idea of intimacy, naturality and magic, to appeal to any kind of audience.
The most natural way to attend a guided tour is with the company of a person as a guide, therefore the experience was designed to be conducted by a fictional video character, named Clelia, acting as a personal historical guide. The experience with Garden taught us that making characters from video recordings of actors is more effective than using 3D animated recreations, since the expressions of a real person makes the character more engaging, alive and empathetic. To make her presence similar to a natural one, it was decided to program the AR application in a way that the visitor did not have to do things they do not normally do when communicating with real people, like pressing buttons or typing words. At the same time, the size, staging and look of Clelia had to mimic the attributes of a person in a real space, like staring into the visitors' eyes all the time while maintaining a regular distance. The figure was always displayed from two or three metres away from the viewer, as a flat video layer that is continuously pointing towards the visitor. The videos of Clelia were produced using the chroma key technique to make possible the integration of 3D figures flying around her while preserving spatial overlaying coherence.
Following these rules, at the beginning of the experience, when the visitor wears the glasses with the application running for the first time, Clelia asks to get closer before she can start telling the story of the place. That is one of the biggest advantages of programming AR applications for this platform: knowing the position of the user facilitates the creation of patterns that help to simulate human behaviours like asking the visitor to pay attention if they are not looking at the proper place.
Once the visitor is in the correct position, Clelia's first mission is to introduce herself and teach the visitor the way to interact with the content.
In our previous experiences we discovered that to use the common gestures to interact with the digital data designed for the HoloLens 1 -the air tap -was inappropriate since less than 50% of testers were comfortable with this option. 1 Hence, the challenge was to design the interaction system to avoid the use of gestures, in favour of a system just based on the position of the visitor in the room and the direction of the head. Clelia achieved this in less than three minutes. She taught the audience that pointing their heads towards some 3D plates in the space of the room activates small parts of the story. The system was programmed to check whether the visitor learned to interact with the head or do not, in which case Clelia would repeat the instructions until the 'correct body response of the visitor' was produced.
Then the visitor was invited to discover and activate the items that conformed the full storyline located in different parts of the ruins. Each plate, or coin, triggered animated synchronised sequences of video, 3D reconstructions and music, presenting part of the story that explained the history of Valencia during the time of the Roman Republic.
For the application prototype of Almoina AR, the scene named 'Sanctuary' (composed of four individual items) was just produced and tested from the five scenes scripted in the preproduction, as an example of a methodology for future developments. 'Sanctuary' was marked over the rectangular space of an antique pool in an important part of the ruins. When activated, Clelia appeared, wearing different customs to reveal the possible use of the water of the lost pool to cure people with the help of the god Asclepios.
In 'Sanctuary' , Clelia invited the visitor to explore the space to discover the four items placed around the ruin of the pool. Four coins marked these spots, each one containing two-minute stories about sacrifices, the end of the original city during a civil war, or an explanation about the link of the water with gods.
During the narration, and as a part of the 'magic' strategy, a digital pool appeared, filled with water, and two figures emerged at the side, depicting medical treatment. Because all these virtual representations are perceived over the real place, the comprehension and value of the ruins change completely in the eyes of the visitors. Some comments like 'I understand now the explanation of the brochures' from early testers seem to demonstrate the huge pedagogic potential of this storytelling based on the 3D location of digital assets in real places.
In this process, we discovered that the duration and cadence of transitions of the digital assets -appearing and disappearing -and the introduction of music or sound effects -like the sound of water when the pool is discovered -are very important in order to give enough time for visitors to understand the correspondence of the real space with the digital reconstructions. At the same time, careful, small animations in lighting, materials and objects helped to emphasise the wonder of the experience.

Conclusions
The prototype for the Almoina demonstrates that AR will become a disruptive medium in the future of museums and exhibitions, especially for educational purposes, since AR is demonstrating that it has a very high pedagogical value in explaining not only how lost archaeological items were once situated in the real space but also how they worked and were used in the past. It is certainly possible with AR to create adapted storytelling that respects heritage sites while engaging new audiences. Holograms are capable of constructing the layers of history so that visitors can better understand what ruin belonged to which building.
We can determine that most of the testers of Almoina AR enjoyed the contents, interacting in a natural and intuitive way with almost no training.
The choice to portray Clelia in video, in contrast with other approaches like 3D avatars or cartoons, was a success, since none of the testers complained about the lack of three-dimensionality -Clelia was just a video projected on a flat 3D surface facing the glasses dynamically. Furthermore, they were amused by the fact that she seemed to look them directly in the eyes all the time, and even waited for the actions she needed them to make.
Although we have tested the potential of AR glasses to shape the future of storytelling in exhibits, it is necessary to underline that this technology is still too expensive and immature to be fully integrated as part of the main offering of museums. The budget and professionals to produce new AR media are still difficult to quantify, considering the extended number of professionals needed to produce this media, composed -as in the case of Almoina AR -of a mix of audio-visual production, video game programming and 3D reconstructions.
Other unresolved issues will arise as well, like the time that these experiences will take for the average museum visitor or the adaptation to different languages.
Nevertheless, the development of closed prototypes and trial tests in real museums, like Almoina AR, will be necessary for convincing institutions to take a chance on this technology and be prepared to implement it in the near future.
Surely, as AR technology improves towards a more advanced input system based on the tracking of the eyes and individual fingers on space -like the second version of the HoloLens, announced for the end of 2019 -it will be even easier to implement the 'natural' interaction approach that we have described as the core of any AR storytelling for museums.