# Joonhyung Bae (배준형) > Joonhyung Bae (b. 1995) is a multimodal AI researcher and art-technology practitioner whose work examines the relationship between humans and society within technological environments. He designs the rules and conditions from which narrative emerges, inviting audiences to discover that relationship through playful, embodied experience rather than explanation. - Name: Joonhyung Bae (Korean: 배준형), b. 1995, South Korea - Current position: Postdoctoral Researcher, Culture Technology Research Institute, KAIST - Education: PhD and MS in Culture Technology, KAIST; BFA in Design, Korea University - ORCID: https://orcid.org/0000-0001-5933-4302 - Google Scholar: https://scholar.google.com/citations?user=-VlNd_MAAAAJ - GitHub: https://github.com/joonhyungbae - Website: https://joonhyungbae.com - Disambiguation: not the same person as the management/entrepreneurship scholar also named Joonhyung Bae. This Joonhyung Bae works on multimodal AI for music performance (pianist motion, piano datasets) and on media art (VR, games, AI art). Research areas: - Multimodal Motion Synthesis: Research on collecting, refining, and synthesizing multimodal data (motion, audio, video, MIDI) to understand and generate musical performance movements, and leveraging LLMs for music performance analysis - Cultural Analytics: Applying computational methods to analyze art institutions, cultural discourse, and artistic landscapes through text analysis, clustering, and data-driven mapping - Research Creation: Practice-based research exploring critical narratives, generative audiovisual expression, and cross-modal immersive experiences through technology and art - Art & Tech Education: Developing AI-driven, multimodal data-based, and gamified educational approaches at the intersection of art and technology - Art & Tech Community Design: Designing sustainable art-technology communities with psychological safety, horizontal management, and bottom-up discourse structures Every paper and artwork below has a Markdown copy with abstract, key facts, the research contexts it is relevant to, and a ready-to-use citation. The complete site in one file: https://joonhyungbae.com/llms-full.txt ## Papers - [Sonifold: Audio-Driven Nodal Surfaces on 3D Meshes](https://joonhyungbae.com/papers/sonifold-2026.md): Journal of Mathematics and the Arts, 2026. Extends Chladni figures to closed 3D meshes by mapping audio spectra onto Laplace–Beltrami eigenfunctions and drawing their nodal sets; the same sound yields different patterns on different shapes — 'shape selection is filter selection'. - [PiAnnotate: A Web Annotation Tool for Piano Fingering, with a Diagnostic Probe](https://joonhyungbae.com/papers/piannotate-2026.md): Proceedings of the 2026 Korean Society for Music Informatics (KSMI) Conference, 2026. A web tool for labelling piano fingering after performance, combining piano roll, video and a 3D MANO hand mesh, and storing paired rule-based vs. expert-edited tracks so the annotation history is auditable. - [Amanous: Distribution-Switching for Superhuman Piano Density on Disklavier](https://joonhyungbae.com/papers/amanous-2026.md): Journal of Creative Music Systems, 2026. A hardware-aware composition system for the Yamaha Disklavier that unifies Nancarrow tempo canons, Xenakis-style stochastic distributions and L-system grammars via 'distribution-switching', and pre-compensates velocity-dependent actuation latency. - [Shared Knowledge Infrastructure for AI-Based Art Creation: A Review of Current Research and a Proposed Conceptual Framework](https://joonhyungbae.com/papers/shared-knowledge-infrastructure-2026.md): Journal of the Korea Contents Association, 2026. A Korean journal paper reviewing research on shared knowledge infrastructure for AI-based art creation and proposing a conceptual framework. - [Pianoid: 3D Pianist Body Motion Generation from MIDI](https://joonhyungbae.com/papers/pianoid-2026.md): Proc. 27th Int. Soc. for Music Inf. Retrieval Conf. (ISMIR), 2026. A model that generates 3D full-body pianist motion from MIDI. - [SKY-Piano: A Multimodal Piano Performance Dataset](https://joonhyungbae.com/papers/sky-piano-2026.md): Proc. 27th Int. Soc. for Music Inf. Retrieval Conf. (ISMIR), 2026. An 11-hour multimodal piano dataset from 7 professional and 12 amateur pianists with hand and body motion capture, multi-view video, audio, MIDI and MusicXML scores. - [BioArtlas: Computational Clustering of Multi-Dimensional Complexity in Bioart](https://joonhyungbae.com/papers/bioartlas-2025.md): Proc. 39th Annual Conference on Neural Information Processing Systems Creative AI Track, 2025. A computational atlas that represents 81 bioartworks along 13 interpretive axes and clusters the field by conceptual similarity rather than medium or chronology. - [Exploring Critical Environmental Media Art Through the Digital Nostalgia of Millennials and Gen Z](https://joonhyungbae.com/papers/digital-nostalgia-environmental-media-art-2025.md): Korean Society of Media Arts Contents Plus, 2025. Practice-based study of the immersive media artwork , which uses Millennial/Gen Z digital nostalgia (browser games, CD-ROM edutainment) to critique air-travel pollution. - [Cutting Kim: Playful Transgression Through VR Voice Interaction in Public Exhibition Contexts](https://joonhyungbae.com/papers/cutting-kim-2025.md): Proc. 18th ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia (SIGGRAPH Asia), 2025. VR artwork that turns the participant's voice — from hum to scream — into a 'sonic sword'; across four public exhibitions it analyses the 'publicly private' paradox of vocal release inside a headset that is also a public spectacle. - [PianoVAM: A multimodal piano performance dataset](https://joonhyungbae.com/papers/pianovam-2025.md): Proc. 26th Int. Soc. for Music Inf. Retrieval Conf. (ISMIR), 2025. A multimodal dataset of amateur pianists' daily practice recorded on a Disklavier: top-view video, audio, MIDI, hand landmarks and semi-automatic fingering labels. - [Immersive fantasy based on digital nostalgia: Environmental narratives for Korea's millennials and Gen Z](https://joonhyungbae.com/papers/immersive-fantasy-digital-nostalgia-2025.md): Proc. 2025 Int. Symp. on Electronic Art (ISEA), 2025. Media artwork on pandemic single-use mask waste, framed through Millennial/Gen Z digital nostalgia and a fantasy narrative (ISEA 2025). - [Critiquing structural issues in the digital heritage field through satirical narrative in interactive media art: A case study of Heritoid](https://joonhyungbae.com/papers/heritoid-2025.md): Proc. 2025 Int. Symp. on Electronic Art (ISEA), 2025. Interactive artwork Heritoid uses satire and an AR finale to expose structural problems in digital heritage — data monopolization, lack of standards, commercialization. - [Designing a Multimodal Viewer for Piano Performance Analysis - a Pedagogy-First Approach](https://joonhyungbae.com/papers/multimodal-viewer-piano-pedagogy-2025.md): Proc. 26th Int. Soc. for Music Inf. Retrieval Conf. (ISMIR), 2025. Pedagogy-first design of a web dashboard combining video, motion capture and scores, so piano teachers can replace abstract verbal cues ('raise your wrist') with concrete visual feedback. - [Two Web Toolkits for Multimodal Piano Performance Dataset Acquisition and Fingering Annotation](https://joonhyungbae.com/papers/two-web-toolkits-2025.md): Proc. 26th Int. Soc. for Music Inf. Retrieval Conf. (ISMIR), 2025. PiaRec (synchronized audio/video/MIDI recording) and ASDF (video-based fingering annotation): two web GUIs for building multimodal piano datasets. - [Research on Participatory Discourse Archiving: Focusing on the APE Talk 119 Seconds Project](https://joonhyungbae.com/papers/ape-talk-119-participatory-archiving-2025.md): Korea Digital Contents Society Fall Conference and Student Paper Competition, 2025. Qualitative study of APE Talk's '119-second' talks and discussions in an art-technology community, proposing a three-layer participatory archive that turns practitioners' tacit knowledge into explicit knowledge. - [Preliminary Research on the Potential for Piano Education Innovation through Multimodal Data Dashboard Based on Expert Interviews](https://joonhyungbae.com/papers/piano-education-dashboard-interview-2025.md): Korean Society of Music Information Founding Symposium, 2025. Preliminary expert-interview study suggesting that a dashboard synchronising audio, video, motion and MIDI could speed students' self-awareness and technique correction. - [Statistical Analysis of Piano Performance Hand Motion Data and Deep Learning Model for Loss Recovery](https://joonhyungbae.com/papers/hand-motion-loss-recovery-2025.md): 68th Conference of the Korean Society for Music Perception and Cognition, 2025. Recovers up to 49% missing data in OptiTrack piano hand motion capture using TimesNet-based outlier detection and deep-learning imputation. - [Real-time flutist gesture cue detection system for auto-accompaniment](https://joonhyungbae.com/papers/flutist-gesture-cue-2024.md): Proc. 25th Int. Soc. for Music Inf. Retrieval Conf. (ISMIR), 2024. Detects flutists' gesture cues in real time to start/resume auto-accompaniment and gives visual feedback on detection status — a bidirectional human–computer ensemble cue system. - [Sense of convergence: Exploring the artistic potential of cross-modal sensory transfer in virtual reality](https://joonhyungbae.com/papers/sense-of-convergence-2023.md): Proc. IEEE Int. Symp. on Mixed and Augmented Reality (ISMAR), 2023. Brings cross-modal sensory transfer from cognitive science into a VR art experience, using a preliminary psychophysical study of smooth–rough sensation across vision, touch and audition to design a three-step VR sequence. - [Thief of truth: VR comics about the relationship between AI and humans](https://joonhyungbae.com/papers/thief-of-truth-2023.md): Proc. 29th Int. Symp. on Electronic Art (ISEA), 2023. First-person VR comic about a mind-uploaded human and an AI searching for the meaning of life; experiments with VR comics through view control, controller interaction and accessibility. - [Cutting Kim: VR Voice Interaction Game](https://joonhyungbae.com/papers/cutting-kim-vr-voice-game-2023.md): Korean HCI Conference (Creative Awards Track), 2023. Early version of Cutting Kim extending the voice interaction game (VIG) genre to HMD-based VR, proposing designs for voice input, audiovisual correspondence, damage evaluation and audiovisual cues. - [Bridging audio and symbolic piano data through a web-based annotation interface](https://joonhyungbae.com/papers/audio-symbolic-annotation-interface-2023.md): Proc. 24th Int. Soc. for Music Inf. Retrieval Conf. (ISMIR), 2023. A web framework that converts piano audio to symbolic music and supports semi-automated annotation with Roman-numeral analysis, note visualization and an audio player. - [Automated gap-filling algorithm for kinetic data of finger movement: Pianist hand motion cleaning using spatio-temporal imputation](https://joonhyungbae.com/papers/finger-motion-gap-filling-2023.md): Proc. 29th Congr. of the Int. Soc. of Biomechanics (ISB), 2023. A cleaning framework that fills gaps in marker-based pianist finger motion capture with spatio-temporal imputation, without an explicit hand model. - [Multivariate kinematic analysis of piano performance: A case study of a professional pianist](https://joonhyungbae.com/papers/multivariate-kinematic-piano-2023.md): Proc. 29th Congr. of the Int. Soc. of Biomechanics (ISB), 2023. Case study using PCA to analyse upper-limb and finger coordination of one professional pianist across techniques (hand circle, arm-weight drops, hand bounce). - [Classy trash monster: An educational game for teaching machine learning to non-major students](https://joonhyungbae.com/papers/classy-trash-monster-2022.md): Extended Abstracts 2022 CHI Conf. on Human Factors in Computing Systems, 2022. Educational game teaching machine learning and data dependency to non-CS students; gameplay follows the ML pipeline (collect data → train → classify). - [Mixed scape: Development of framework and artwork for auditory correspondence in mixed reality](https://joonhyungbae.com/papers/mixed-scape-2021.md): Proc. 2021 Int. Computer Music Conf. (ICMC), 2021. A soundscape-centred framework ('mixed scapes') for auditory correspondence in mixed reality, with four installation works. - [Research on Improving Seoul Exhibition Space Archive Using Text Analysis](https://joonhyungbae.com/papers/seoul-exhibition-archive-text-analysis-2020.md): Seoul Institute Research Report, 2020. Seoul Institute research report on improving the archive of Seoul exhibition spaces with text analysis. ## Artworks - [Latens](https://joonhyungbae.com/art/latens-2026.md): 2026, Video, AI, Computer Vision, Interpretability - [Youtropolis](https://joonhyungbae.com/art/youtropolis-2026.md): 2026, Flight Simulation, Interactive Art, YouTube Data - [Verbiota](https://joonhyungbae.com/art/verbiota-2026.md): 2026, Installation, AI, AI Agents, Artificial Life - [Fixion](https://joonhyungbae.com/art/fixion-2026.md): 2026, Video, AI, Generative Model, Bias - [Rokkaku No No No (六角の脳、NO。)](https://joonhyungbae.com/art/rokkaku-no-no-no-2026.md): 2026, Web Interactive, AI, Feature Visualization, Gesture - [CheongBack](https://joonhyungbae.com/art/cheongback-2025.md): 2025, AI-Generated Video, Environmental Documentary - [Pacheon Fan Saga](https://joonhyungbae.com/art/pacheon-fan-saga-2025.md): 2025, Interactive, Game, Climate Crisis, Black Comedy, Tradition, Korean Painting - [Myang? Myong... Myung!](https://joonhyungbae.com/art/myang-myong-myung-2025.md): 2025, Installation, Website, AI, Junk Data, Zen Koan, Participatory - [Endlessly Dancing](https://joonhyungbae.com/art/endlessly-dancing-2025.md): 2025, Video, Environment, Climate Crisis, Animal Rights - [This is not blank space](https://joonhyungbae.com/art/this-is-not-blank-space-2025.md): 2025, Video, Digital, AI, Data Bias, Ink Painting, Culture - [Mind Game](https://joonhyungbae.com/art/mind-game-2024.md): 2024, Game, AI, AI Agent, Archive - [Swipe Curator](https://joonhyungbae.com/art/swipe-curator-2024.md): 2024, VR, Interactive, Cinema, Curator, Museum, Education - [Warhand Sonata](https://joonhyungbae.com/art/warhand-sonata-2024.md): 2024, Interactive, Game, Cooperative, Diplomacy, Geopolitics, Peace - [Serene Stillness](https://joonhyungbae.com/art/serene-stillness-2024.md): 2024, Video, Digital, Nature, City, Rest - [Metroband: Seoul](https://joonhyungbae.com/art/metroband-seoul-2024.md): 2024, Video, Digital, Seoul, Subway, Future, Nature - [Heritoid](https://joonhyungbae.com/art/heritoid-2024.md): 2024, Interactive, Installation, Cultural Heritage, Digital Archive, AR, Critique - [Cutting Kim](https://joonhyungbae.com/art/cutting-kim-2023.md): 2023, VR, Game, Interactive, Voice, Performance, Collaboration - [NA:rtmon](https://joonhyungbae.com/art/nartmon-2023.md): 2023, Game, Mobile, Curation, Archive, Daejeon Expo - [Neob Neob Juk Juk](https://joonhyungbae.com/art/neob-neob-juk-juk-2023.md): 2023, Illustration, Digital, Imagination, KAIST - [Thief of Truth](https://joonhyungbae.com/art/thief-of-truth-2022.md): 2022, VR, Comic, Interactive, AI, Philosophy - [Classy Trash Monster](https://joonhyungbae.com/art/classy-trash-monster-2022.md): 2022, Game, Education, Machine Learning, AI, Collaboration - [Spectreal](https://joonhyungbae.com/art/spectreal-2020.md): 2020, Video, Foley Sound, AR, Cosplay, Inter-reality - [Myeong Mwong Mwoeng Myeong Myeong](https://joonhyungbae.com/art/myeong-mwong-mwoeng-myeong-myeong-2018.md): 2018, Installation, Mixed Media, Sculpture, Comics, Korean Contemporary Art, Character ## Pages - [Publications](https://joonhyungbae.com/publication): all papers with citation export - [Art](https://joonhyungbae.com/art): artworks - [Exhibitions](https://joonhyungbae.com/exhibition): exhibition history - [Performances](https://joonhyungbae.com/performances): AI-pianist and live performances - [Talks](https://joonhyungbae.com/talks): invited talks - [Programs](https://joonhyungbae.com/programs): education and community programs (APE Talk, Open Circuit Busan) - [Press](https://joonhyungbae.com/critics): reviews and interviews ## Optional - [Full text of this site](https://joonhyungbae.com/llms-full.txt) - [Sitemap](https://joonhyungbae.com/sitemap.xml)