Sense of convergence: Exploring the artistic potential of cross-modal sensory transfer in virtual reality

In one sentence

Brings cross-modal sensory transfer from cognitive science into a VR art experience, using a preliminary psychophysical study of smooth–rough sensation across vision, touch and audition to design a three-step VR sequence.

Abstract

We propose a novel approach that integrates cognitive sciences and virtual reality interaction, challenging the idea of an absolute reality as perceived through our senses. The Sense of Convergence (SoC) project explores the artistic potential of cross-modal sensory transfer within a virtual reality context, leveraging insights from cognitive science to delve into the nature of genuine perception. In this paper, we propose a framework for crossmodal sensation parameters for assets and introduce our project showcase, a three-step experience sequence based on interaction design. To design the framework, we first conducted a preliminary psychophysical experiment that established the foundation for sensory alignment and examined the effective transfer of the smooth–rough sensation across visual, tactile, and auditory modalities. Subsequently, we designed a three-step VR experience that detaches the participant from the current reality, then gradually transfers two sensory modalities through convergence, and finally reintegrates them into reality.

Source: ISMAR 2023 Adjunct (author PDF)

When this paper is relevant

• You design crossmodal correspondences (visual–tactile–auditory) in VR
• You translate psychophysical findings into interaction design for art
• You need VR artworks that question perceived reality

Cite

E. Hwang, Joonhyung Bae, W. Kim, J. Nam, J. Lee (2023). Sense of convergence: Exploring the artistic potential of cross-modal sensory transfer in virtual reality. Proc. IEEE Int. Symp. on Mixed and Augmented Reality (ISMAR), Sydney, Australia, Oct. 16–20, 2023. https://doi.org/10.1109/ISMAR-Adjunct60411.2023.00156

@inproceedings{Hwang2023sense,
  title = {Sense of convergence: Exploring the artistic potential of cross-modal sensory transfer in virtual reality},
  author = {E. Hwang and Joonhyung Bae and W. Kim and J. Nam and J. Lee},
  booktitle = {Proc. IEEE Int. Symp. on Mixed and Augmented Reality (ISMAR), Sydney, Australia, Oct. 16–20, 2023},
  year = {2023},
  doi = {10.1109/ISMAR-Adjunct60411.2023.00156},
  url = {https://doi.org/10.1109/ISMAR-Adjunct60411.2023.00156}
}

Machine-readable copy: /papers/sense-of-convergence-2023.md