# Sense of convergence: Exploring the artistic potential of cross-modal sensory transfer in virtual reality

- Authors: E. Hwang, Joonhyung Bae, W. Kim, J. Nam, J. Lee
- Venue: Proc. IEEE Int. Symp. on Mixed and Augmented Reality (ISMAR), Sydney, Australia, Oct. 16–20, 2023
- Year: 2023
- Type: Conference Paper
- DOI: https://doi.org/10.1109/ISMAR-Adjunct60411.2023.00156
- Publisher page: https://ieeexplore.ieee.org/abstract/document/10322287?casa_token=0nyQTo76F1YAAAAA:JdsmmS57I2ITWW0IFhHMybnCJYsA37EMGxYZUFbK7cgvdnQJ3fxsG5lrRU69vOoPwzDPIMId
- PDF: https://drive.google.com/file/d/1iOrMz7tUvWcMcmOIRVYacy58ezYED3-H/view?usp=drive_link
- Keywords: cross-modal sensory transfer, virtual reality, artistic potential, sensory convergence, mixed reality
- Page: https://joonhyungbae.com/publication/sense-of-convergence-2023

## In one sentence

Brings cross-modal sensory transfer from cognitive science into a VR art experience, using a preliminary psychophysical study of smooth–rough sensation across vision, touch and audition to design a three-step VR sequence.

인지과학의 교차감각(cross-modal) 전이를 VR 예술 경험으로 옮긴 작업. 매끄러움–거침 감각의 시각·촉각·청각 간 전이를 예비 심리물리 실험으로 정렬한 뒤 3단계 VR 체험으로 설계했다.

## Abstract

We propose a novel approach that integrates cognitive sciences and virtual reality interaction, challenging the idea of an absolute reality as perceived through our senses. The Sense of Convergence (SoC) project explores the artistic potential of cross-modal sensory transfer within a virtual reality context, leveraging insights from cognitive science to delve into the nature of genuine perception. In this paper, we propose a framework for crossmodal sensation parameters for assets and introduce our project showcase, a three-step experience sequence based on interaction design. To design the framework, we first conducted a preliminary psychophysical experiment that established the foundation for sensory alignment and examined the effective transfer of the smooth–rough sensation across visual, tactile, and auditory modalities. Subsequently, we designed a three-step VR experience that detaches the participant from the current reality, then gradually transfers two sensory modalities through convergence, and finally reintegrates them into reality.

(Source: ISMAR 2023 Adjunct (author PDF))

## When this paper is relevant

- You design crossmodal correspondences (visual–tactile–auditory) in VR / VR에서 시각·촉각·청각 간 교차감각 대응(crossmodal correspondence)을 설계할 때
- You translate psychophysical findings into interaction design for art / 심리물리 실험 결과를 예술 작품의 인터랙션 설계로 옮기는 방법을 찾을 때
- You need VR artworks that question perceived reality / 지각의 실재성을 묻는 VR 아트 사례가 필요할 때

## How to cite

E. Hwang, Joonhyung Bae, W. Kim, J. Nam, J. Lee (2023). Sense of convergence: Exploring the artistic potential of cross-modal sensory transfer in virtual reality. Proc. IEEE Int. Symp. on Mixed and Augmented Reality (ISMAR), Sydney, Australia, Oct. 16–20, 2023. https://doi.org/10.1109/ISMAR-Adjunct60411.2023.00156

```bibtex
@inproceedings{Hwang2023sense,
  title = {Sense of convergence: Exploring the artistic potential of cross-modal sensory transfer in virtual reality},
  author = {E. Hwang and Joonhyung Bae and W. Kim and J. Nam and J. Lee},
  booktitle = {Proc. IEEE Int. Symp. on Mixed and Augmented Reality (ISMAR), Sydney, Australia, Oct. 16–20, 2023},
  year = {2023},
  doi = {10.1109/ISMAR-Adjunct60411.2023.00156},
  url = {https://doi.org/10.1109/ISMAR-Adjunct60411.2023.00156}
}
```