TimToShape
Table of Contents
For more information, also see this article.
About #
Timbre is high-dimensional and sensuous, making it difficult for musical-instrument learners to improve their timbre. Although some systems exist to improve timbre, they require expert labeling for timbre evaluation; however, solely visualizing the results of unsupervised learning lacks the intuitiveness of feedback because human perception is not considered. Therefore, we employ crossmodal correspondences for intuitive visualization of the timbre.
We designed “TimToShape”, a system that visualizes timbre with 2D shapes based on the user’s input of timbre–shape correspondences.
TimToShape generates a shape morphed by linear interpolation according to the timbre’s position in the latent space, which is obtained by unsupervised learning with a variational autoencoder (VAE).
We confirmed that people perceived shapes generated by TimToShape to correspond more to timbre than randomly generated shapes. Furthermore, a user study of six violin players revealed that TimToShape was well-received in terms of visual clarity and interpretability.
Publications #
-
Kota Arai, Yutaro Hirao, Takuji Narumi, Tomohiko Nakamura, Shinnosuke Takamichi, and Shigeo Yoshida. 2023. TimToShape: Supporting Practice of Musical Instruments by Visualizing Timbre with 2D Shapes based on Crossmodal Correspondences. In 28th International Conference on Intelligent User Interfaces (IUI ’23), March 27–31, 2023, Sydney, NSW, Australia. ACM, New York, NY, USA, 16 pages. https://doi.org/10.1145/3581641.3584053
-
Kota Arai, Mone Konno, Yutaro Hirao, Shigeo Yoshida, and Takuji Narumi. 2021. Effect of Visual Feedback on Understanding Timbre with Shapes Based on Crossmodal Correspondences. In Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology (VRST ‘21). Association for Computing Machinery, New York, NY, USA, Article 63, 1–3. https://doi.org/10.1145/3489849.3489912
Downloads #
The paper and videos in this page can be used under the terms of Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0). https://creativecommons.org/licenses/by-nc-nd/4.0/
Related Projects #