Multisensory Museum Experience, Artium
Project Overview
We designed Artium, an immersive and interactive AR museum experience that fosters a deeper understanding and connection for those visiting the Carnegie Museum of Art (CMOA).
Challenge & Goal
Today, many museums are facing the challenge of transitioning from rigid institutions to experiential and flexible spaces; it has remained relatively static for the past decade and is facing challenges to attract more visitors with greater engagement. Therefore, the Carnegie Museum of Art (CMOA) museum needed to increase the accessibility of contextual information for museumgoers to provide an approachable, informative, and memorable experience.
Timeline
Oct - Dec 2022
Interaction Design Studio I
Team
Iris Cai
Kristen Cai
Angela Nam
Shafiqah Zulfikar
Role
User Research
AR Gesture Study
User Flow
AR Technology Research
Concept Development
Prototyping
Tools
Figma
Adobe CC (Ae, Pr)
Unity
Cinema 4D
Spatial.io
The Outcome
Artium is an augmented reality museum experience designed to make the Carnegie Museum of Art (CMOA) more engaging and memorable for its visitors. Our goal is to use accessible storytelling and interactive experiences that engage multiple senses to create an immersive and informative experience that not only turns visitors into learners but also inspires them to become collectors.
Onboarding
Visitors purchase admission tickets and receive Hololens2 at the admission counter to start the Artium experience.
Instruction
The gestural tutorial is prompted to the visitors before starting the museum experience.
Personalized Experience
Visitors can personalize their Artium experience by setting desired time and themes
Interactive Arts
Interactions in the Artium experience helps users to understand the artwork in depth
Understanding the hidden context
Artium helps users learn about the artwork in depth by engaging with interactions that are designed specifically to understand the story behind the artwork
Reveal hidden layers
Users can uncover hidden aspects of artworks through 3D components to manipulate elements.
Memory keeping artifacts
Users can collect digital souvenirs from the Artium experience after unlocking all of its interactions
Exhibit collected Artifacts
Following up with the museum visit, users can curate a collection of artifacts in a virtual museum
How does Artium Works?
Artium aims to enhance visitors' comprehension and emotional attachment to art at Carnegie Museum of Art by re-integrating it into its original setting. To maintain a manageable scope for the project, we focused on three exhibition areas within the CMOA: the Impressionism collection in the Scaife Gallery, the Religious Paintings section, and the Hall of Architecture.
Process
1. Preliminary Research
We first researched multimodal immersive technology trends and competitive analysis of museums to better understand the current landscape of multimodal interactions in museum settings.
Multimodal Immersive Technology Trend Research
Museums Competitive Analysis
2. User Interview
We conducted user research with two groups to find insights from various perspectives. We probed for their general museum experiences and preferences and narrowed them down to their most recent museum experience.
1 Curator
Shared data and insights about the museum, demographics of visitors

6 Museum Visitors
Conducted user interviews, contextual inquiry

Interview with Curator from CMOA - Rachel
The curator shared that the visitors want them to rethink their own relationship with the material world and place themselves into the designer’s thinking and process of creating.
User Interview
We met 6 museum visitors and heard their voices. The most common reaction about their journey was they want to engage in art collection with digestible information. They also wanted to take back a piece of memory after visiting the museum.
Insights
Enhance engagement and learning by developing a dynamic and playful museum experience
Create a way to make the museum experience more approachable and welcoming
Go beyond two-dimensional engagement and participation with art by implementing a multi-sensorial experience
Allow affordable & creative memory-keeping by enabling them to compile art pieces and build their personalized collection
Design Principles
Approachable
Allow visitors to explore the space at their own pace and create experiences that reduce cognitive barriers to connecting with art
Informative
Help visitors gain knowledge about the art pieces while interacting
Memorable
Allow the audience to have a physical and mental attachment to the museum
Problem Statement
How might we increase the accessibility of contextual information for museum-goers so that they can have an approachable, informative, and memorable experience?
Ideation
Our team aimed to address the main problem statement by prioritizing Artium's emphasis on approachable storytelling, multi-sensory and two-directional experiences to transform the museum into a valuable learning resource and to encourage visitors to become collectors. We considered various emerging technologies and visualized our initial findings to create a concept that aligns with Artium's goals.
1. AR Technology Research
To better understand how our initial discoveries and concepts would be put into action, our team experimented with various types of emerging technologies. Our exploration centered on testing the capabilities of Oculus and Hololens2 to determine their potential for creating a more engaging and immersive museum experience in a dynamic and diverse format.
2. Gesture Study
We put a lot of effort into making gestural commands simple because the more gestures we have, the harder it will be for people to learn them. Since museum visits are usually short and infrequent, visitors should be able to learn all the commands quickly, ideally within a minute. To make it easier, we borrowed the concept of a watch for two gestures: "Raise the wrist" and "Pinch and collect."
Cognitive Load
Create a digital system that integrates dispersed ordering and receiving experiences
Intuition
Gestural interaction should be intuitive and easy to learn so the user can quickly and easily use the device without extensive training.
Immediate Feedback
The AR glass should provide immediate feedback to the user when a gesture is recognized to confirm that the action has been registered.
Calibrated Projection
The AR projection should be calibrated correctly so that the virtual content aligns properly with the physical environment. This calibration should be done at regular intervals to maintain accuracy.
Field of View
The AR content should be projected at an appropriate field of view to ensure the user can see it clearly without tinting their head or device too much. This viewing angle should be comfortable and intuitive for the user.
3. System Diagram
Through our research on design guidelines for immersive technology, we created a system diagram that shows an overview of the system's flow in how it shows interactive contents through HoloLens2.
Using this diagram, we identified
11 gestural interactions for our Artium experience
The range between 1.25m-2.5m is the most appropriate focal distance to display hologram objects
Thin font weight should be avoided to reduce eye strain and maintain user comfort because the vertical strokes can jitter in small font sizes
More than two different font families will break the harmony and consistency of the user experience and make it harder to consume information
AR Interaction Research
Gestural Interactions
Following the experimentation phase, we concluded that Microsoft Hololens2 would be ideal for in-person museum experiences, while Meta Oculus would be better suited for post-museum experiences. This decision was based on several factors.
Connection with artworks
The use of Augmented Reality enables museums to merge the digital and physical worlds, offering visitors an innovative and interactive way to experience artworks
Virtual Objects
Augmented Reality allows visitors to interact with virtual objects rather than simply having additional information overlaid onto the artwork's surface
Subtle Interaction
By incorporating Augmented Reality interactions onto virtual surfaces, visitors can engage with artworks in a new and exciting way, using hand gestures to interact with the augmented reality elements
Development
1. User Scenario
With an understanding of the scenarios where Artium could really help museum visitors, we created journey map and storyboard of which situations that helped us understand the needs and nuances for which we needed to design.
Storyboard
After researching and investigating, we developed a concept and created a storyboard to explore initial design possibilities. The storyboarding process allowed us to create a framework for the museum experience and receive valuable feedback to refine certain aspects of it.
Journey Map
We created a journey map to illustrate a better picture of the whole experience that visitors will face with our product. We listed a linear path of an exhibit first and then mapped place, interaction, sensory, and modalities that match. This process also helped us to define what technology we need for each step.
2. Design Systems
Once we had developed the concept for Artium, our next objective was to create a cohesive and all-encompassing experience. We delved into the visual system of Artium, utilizing a saturated color palette to draw attention to the most important aspects of the Artium experience. The typography style was carefully selected to balance the intensity of the color palette, resulting in an overall visual identity that accurately reflects the Artium brand.
Visual Design
Interface Design
2. Refined Gesture
To provide a detailed user journey for an art exhibition or installation, we have incorporated and arranged specific hand gestures that visitors will use during the experience in each stage. This approach can also be applied to other types of user journeys, as the hand gestures utilized by visitors will vary based on the particular pieces they interact with.
Trigger Wrist Menu
Resize and Rotate
Grasp
Collect
Scroll
Recolor
3. Prototypes
Prototyping is a crucial part of the AR experience. By iterating and testing 3D virtual objects using Unity and Hololens, we were able to create elements that are integral to the Artium experience and promote user engagement in this immersive art museum setting. In order to understand the interaction of Artium's visual elements within both physical and digital spaces, we carried out physical computing using multiple prototyping software tools. This process helped us pinpoint prospective interactive patterns that could be incorporated into the overall Artium experience.
4. Final Prototypes / Demo
We opened a live demo session in Design Week at the end of the semester. The live demo includes the concept video, Pantheon detection(using a mobile phone), and personal collection exhibition (Spatial.io).
Reflection & Next Step
1. Limitation of Gestural Commands
Our secondary research on gesture-tracking technology for developing gestural commands for Artium has revealed several limitations. For instance, gestures performed outside the camera's view may not be accurately tracked, and excessive hand movements, intentional or unintentional, necessitate additional measures to avoid false activations. Moreover, hands-free scenarios should be considered since hands are essential for various tasks.
2. Getting out of Comfort Zone
Exploring uncharted territory was an exhilarating adventure as we had limited knowledge of multimodal experiences and the relations between AR technology and museums at the outset of the project. Nonetheless, through investigation and learning new knowledge, we broadened our design horizons and developed problem-solving skills using design techniques.
3. Next Steps
Exploring uncharted territory was an exhilarating adventure as we had limited knowledge of multimodal experiences and the relations between AR technology and museums at the outset of the project. Nonetheless, through investigation and learning new knowledge, we broadened our design horizons and developed problem-solving skills using design techniques.