As museum lovers, Robert and I decided to approach the massive problem of multi-modal interaction design with a smaller context of virtual exhibition.
‍
Though virtual exhibitions are more common nowadays, they are still generally inconvenient to operate and offer a compromised experience. For example, the common design of fixed-point movement limits the exploratory nature of the space. Another example is that joystick control is often difficult to operate, especially for users who are unfamiliar with game controllers.

While touch-based user interfaces are common, emerging technologies like eye and face-tracked UIs show potential for enhancing usability and user experience in virtual spaces. Understanding the impact of physical-tracking in the context of navigating and viewing exhibits and individual artworks is crucial. Although these technologies are becoming slowly more prevalent in VR applications, there is limited knowledge regarding their effectiveness in mobile virtual environments.

With specific context, new questions arise:
1) what are major factors influencing viewing experiences?
2) what are viewer's intuitive reactions to those factors?

Our initial observations and research suggest that visual distance and image complexity are two major factors influencing virtual art-viewing experiences in 3D environments. Therefore, we design the first study to examine participants' facial and physical expressions when exposed to visually distant and highly complex artworks.

We hypothesize that certain expressions, such as squinting and leaning, will naturally occur as viewers attempt to gain a clearer view or understanding of the displayed art.The insights from this preliminary study will inform the design of interactions for the second stage of our research.

Image Matrices
- Considering the two variables (distance and complexity), we developed two matrices, each containing eight base images.
- We established four levels to quantitatively measure the degree of these variables (distance and complexity).
- Consequently, we generated a total of 64 images for the study.

Randomized Image Sets
- For each participant, we generated a image set of 16, covering all degrees of the variables
- Each base image appear only once
- Each degree of the variable appear twice with different base images

Image size vs. word count
- Word count: how many words were said about each image
- Word count increased with image size; the higher the image size, the higher the word count
- On average, participants had more to say about larger images

Image size vs. viewing time
- Viewing time: how long someone viewed a specific image (until they specified they were finished)
- Viewing time increased with image size; the higher the image size, the more viewing time
- On average, participants spent more time viewing larger images

Image size vs. Lean Percentage
- Lean Percentage: the percentage change of how much a participant leaned into the computer screen
- Lean percentage decreased with image size; the higher the image size, the lower lean percentage
- On average, participants leaned in significantly more for smaller images

Overlay percentage vs. word count
- Word count: how many words were said about each image
-Word count increased with a certain overlay percentage (0.25 to 0.75) in comparison to full-size images

Overlay percentage vs. viewing time
- Viewing time: how long someone viewed a specific image (until they specified they were finished)
- Viewing time increased with a certain overlay percentage (0.25 to 0.75)

Overlay percentage vs. Lean Percentage
- Lean Percentage: the percentage change of how much a participant leaned into the computer screen
- Lean percentage slightly decreased with a certain overlay percentage (0.25 to 0.75)

Average Normalized Lean Percentage
- Image size relatively predicted whether someone leaned in towards the computer screen. The smaller the image, the more likely the participant will lean in farther.
- Generally, overlay percentages show that participants leaned in slightly more for high range confusion values (0.5). Smaller image sizes displayed higher values of leaning in than overlay images.

Squint occurrence
- Image size was a clear predictor for whether participants squinted when viewing a certain image. The smaller the image, the more likely a user squinted when attempting to describe the respective image.
- Generally, overlay percentages show that participants squinted more for high range confusion values (0.5). Smaller image sizes displayed higher values of squinting than overlay images.

General Pre-Study Conclusions
Visually distant artworks (smaller image sizes) majorly provoked reactions of squinting, leaning in, and frowning.

For the purpose of integrating these into a user interface and balancing technical abilities, we went forward with utilizing squinting and leaning as interface triggers.

Existing work on eye expression and VR
Squinting is feasible for 100% participants in this study, suggesting that it is a common and feasible facial expression.Squinting is ranked the second after blinking for preference, suggesting that this gesture has the potential being used more often.

Common Interaction Modes for Mobile Device
The common interaction modes all have references to physical interaction, suggesting that interaction design grounded with physical habits or intuitions are more acceptable.The paper did not identify any best interaction mode, suggesting a need for exploration and potential for personalization.

- Overall, participants found the facial-tracking experience more enjoyable and appealing.
- However, on average, it was perceived as less intuitive and efficient, potentially due to bugs and facial-recognition failures.
- The greater variance in the second experience suggests that personal preferences for modality choice can vary significantly.