Building a Trusted Explorable Recommendation Foundation Technology

To promote a society where people can feel secure in receiving personalized support from AI systems, this interdisciplinary research integrating informatics, neurophysiology, and social psychology aims to establish fundamental technologies that allow users of recommender systems to explore recommendation behaviors. The goal is to provide human-centered, controllable, and transparent recommender systems that can be used sustainably by consumers or producers as a trusted social infrastructure.

• CREST (Core Research for Evolutional Science & Technology) funding program provided by JST (Japan Science and Technology Agency)
• Research Area: Core technologies for trusted quality AI systems (Trusted quality AI systems)
• Research Project: Building a Trusted Explorable Recommendation Foundation Technology
• Project Name: RecMus Project
• Research Period: Five years from December 1, 2020 to March 31, 2026

Members

Research Director

Masataka Goto

Prime Senior Researcher, National Institute of Advanced Industrial Science and Technology (AIST)

Collaborators

Shinichi Furuya

Senior Researcher, Sony Computer Science Laboratories, Inc.

Yoshinori Hijikata

Professor, Kwansei Gakuin University

Major Publications

Journal/Transaction

• Takayuki Nakatsuka, Kento Watanabe, Yuki Koyama, Masahiro Hamasaki, Masataka Goto, and Shigeo Morishima: Vocal-Accompaniment Compatibility Estimation Using Self-Supervised and Joint-Embedding Techniques, IEEE Access, Vol.9, pp.101994-102003, 2021/07.
• Hiromu Yakura, Kento Watanabe, and Masataka Goto: Self-Supervised Contrastive Learning for Singing Voices, IEEE/ACM Transactions on Audio, Speech, and Language Processing, Vol.30, pp.1614-1623, 2022/04.
• Hiromu Yakura, Tomoyasu Nakano, and Masataka Goto: An Automated System Recommending Background Music to Listen to While Working, User Modeling and User-Adapted Interaction (UMUAI), Vol.32, pp.355-388, 2022/05.
• Takanori Oku, Shinichi Furuya: Noncontact and High-Precision Sensing System for Piano Keys Identified Fingerprints of Virtuosity, Sensors, Vol.22, No.13, pp.1-11, 2022/06.
• Kaito Muramatsu, Takanori Oku, Shinichi Furuya : The plyometric activity as a conditioning to enhance strength and precision of the finger movements in pianists, Scientific Reports, Vol.12, No.22267, pp.1-12, 2022/12.
• Hiroshi Sakuma, Ao Hori, Minami Murashita, Chisa Kondo and Yoshinori Hijikata: YouTubers vs. VTubers: Persuasiveness of human and virtual presenters in promotional videos, Frontiers in Computer Science (Sec. Human-Media Interaction), Vol.5, pp.1-12, 2023/03.
• Kosetsu Tsukuda, Masahiro Hamasaki, and Masataka Goto: Why and How People View Lyrics While Listening to Music on a Smartphone, IEICE Transactions on Information and Systems, Vol.E106-D, No.4, pp.556-564, 2023/04.
• Kosetsu Tsukuda, Keisuke Ishida, Masahiro Hamasaki, and Masataka Goto: Kiite Cafe: A Web Service Enabling Users to Listen to the Same Song at the Same Moment While Reacting to the Song, IEICE Transactions on Information and Systems, Vol.E106-D, No.11, 2023/07.
• Kento Watanabe and Masataka Goto: A Method to Detect Chorus Sections in Lyrics Text, IEICE Transactions on Information and Systems, Vol.E106-D, No.9, pp.1600-1609, 2023/09.

International Conference

• Hiromu Yakura, Yuki Koyama, and Masataka Goto: Tool- and Domain-Agnostic Parameterization of Style Transfer Effects Leveraging Pretrained Perceptual Metrics, Proceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI 2021), pp.1208-1216, 2021/08.
• Kosetsu Tsukuda, Masahiro Hamasaki, and Masataka Goto: Toward an Understanding of Lyrics-viewing Behavior While Listening to Music on a Smartphone, Proceedings of the 22nd International Society for Music Information Retrieval Conference (ISMIR 2021), pp.705-713, 2021/11.
• Kosetsu Tsukuda, Keisuke Ishida, Masahiro Hamasaki, and Masataka Goto: Kiite Cafe: A Web Service for Getting Together Virtually to Listen to Music, Proceedings of the 22nd International Society for Music Information Retrieval Conference (ISMIR 2021), pp.697-704, 2021/11.
• Kento Watanabe and Masataka Goto: Atypical Lyrics Completion Considering Musical Audio Signals, Proceedings of the 2nd Workshop on NLP for Music and Spoken Audio (NLP4MuSA 2021), pp.1-5, 2021/11.
• Tian Cheng and Masataka Goto: An Analysis of Using Fuzzy Annotations in CRNN-Based Joint Beat and Downbeat Tracking, Proceedings of the 30th European Signal Processing Conference (EUSIPCO 2022), pp.224-228, 2022/08.
• Yuki Koyama and Masataka Goto: BO as Assistant: Using Bayesian Optimization for Asynchronously Generating Design Suggestions, Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (ACM UIST 2022), No.77, pp.1-14, 2022/11.
• Chisa Kondo, Hiroshi Sakuma, Yoshinori Hijikata: A Study on Agent Expression and User Gaze Behavior in Product Endorsement Videos, Proceedings of the International Workshop on Affective Computing and Emotion Recognition (ACER-EMORE2022), pp.658-665, 2022/11.
• Takayuki Nakatsuka, Masahiro Hamasaki, and Masataka Goto: Content-Based Music-Image Retrieval Using Self- and Cross-Modal Feature Embedding Memory, Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV 2023), pp.2174-2184, 2023/01.
• Erwin Wu, Hayato Nishioka, Shinichi Furuya, Hideki Koike: Marker-removal Networks to Collect Precise 3D Hand Data for RGB-based Estimation and its Application in Piano, Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV 2023), pp.2976-2985, 2023/01.
• Jun Kato and Masataka Goto: Lyric App Framework: A Web-based Framework for Developing Interactive Lyric-driven Musical Applications, Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (ACM CHI 2023), pp.124:1-124:18, 2023/04.
• Tian Cheng and Masataka Goto: U-Beat: A Multi-Scale Beat Tracking Model Based on Wave-U-Net, Proceedings of the 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing (IEEE ICASSP 2023), pp.1-5, 2023/06.
• Tomoyasu Nakano and Masataka Goto: Music Source Separation with MLP Mixing of Time, Frequency, and Channel, Proceedings of the 24th International Society for Music Information Retrieval Conference (ISMIR 2023), 2023/11.
• Kento Watanabe and Masataka Goto: Text-to-lyrics Generation with Image-based Semantics and Reduced Risk of Plagiarism, Proceedings of the 24th International Society for Music Information Retrieval Conference (ISMIR 2023), 2023/11.
• Tian Cheng and Masataka Goto: Transformer-based Beat Tracking with Low-resolution Encoder and High-resolution Decoder, Proceedings of the 24th International Society for Music Information Retrieval Conference (ISMIR 2023), 2023/11.
• Hiromu Yakura and Masataka Goto: IteraTTA: An Interface for Exploring Both Text Prompts and Audio Priors in Generating Music with Text-to-audio Models, Proceedings of the 24th International Society for Music Information Retrieval Conference (ISMIR 2023), 2023/11.
• Vincent Cheung, Lana Okuma, Kazuhisa Shibata, Kosetsu Tsukuda, Masataka Goto, and Shinichi Furuya: Decoding Drums, Instrumentals, Vocals, and Mixed Sources in Music Using Human Brain Activity, Proceedings of the 24th International Society for Music Information Retrieval Conference (ISMIR 2023), 2023/11.
• Haven Kim, Kento Watanabe, Masataka Goto, and Juhan Nam: A Computational Evaluation Framework for Singable Lyric Translation, Proceedings of the 24th International Society for Music Information Retrieval Conference (ISMIR 2023), 2023/11.
• Kosetsu Tsukuda, Masahiro Hamasaki, and Masataka Goto: Chorus-Playlist: Exploring the Impact of Listening to Only Choruses in a Playlist, Proceedings of the 24th International Society for Music Information Retrieval Conference (ISMIR 2023), 2023/11.
• Kosetsu Tsukuda, Tomoyasu Nakano, Masahiro Hamasaki, and Masataka Goto: Unveiling the Impact of Musical Factors in Judging a Song on First Listen: Insights from a User Survey, Proceedings of the 24th International Society for Music Information Retrieval Conference (ISMIR 2023), 2023/11.
• Tomoyasu Nakano, Momoka Sasaki, Mayuko Kishi, Masahiro Hamasaki, Masataka Goto, and Yoshinori Hijikata: A Music Exploration Interface Based on Vocal Timbre and Pitch in Popular Music, Proceedings of the 16th International Symposium on Computer Music Multidisciplinary Research (CMMR 2023), 2023/11.