Yuan Yao


Postdoc Research Fellow
NExT++ Lab,
School of Computing,
National University of Singapore.

Contact

  yaoyuanthu [at] 163 [dot] com
  National University of Singapore
  Google Scholar

About me

Hi! I am a postdoc research fellow at the National University of Singapore, where I am a member of the NExT++ Lab, working with Professor Chua Tat-Seng. I received my Ph.D. degree from the Department of Computer Science and Technology of Tsinghua University, advised by Professor Zhiyuan Liu at the Tsinghua Natural Language Processing Lab. Before I became a Ph.D. student, I also received my bachelor's degree from Tsinghua University. My research interests include multimodal large language models and natural language processing. I lead the MiniCPM-V and MiniCPM-o series of efficient multimodal large language models.

I will join College of AI, Tsinghua University as a Tenure-track Assistant Professor in October, 2025. I'm looking for highly motivated PhD students and research interns interested in multimodal large language models.

PUBLICATIONS

2025

1. Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang,Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Huarong Zhou, Zhihui He, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Jie Zhou, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, Maosong Sun. MiniCPM-V: A GPT-4V Level Multimodal LLM on Your Phone. Nature Communications. [Project: MiniCPM-V]
2. Tianyu Yu, Bo Ji, Shouli Wang, Shu Yao, Zefan Wang, Ganqu Cui, Lifan Yuan, Ning Ding, Yuan Yao†, Zhiyuan Liu, Maosong Sun, Tat-Seng Chua. († indicates corresponding author) RLPR: Scaling RLVR to General Domain without Verifiers. Preprint.
3. Ji Qi, Yuan Yao†, Yushi Bai, Bin Xu, Juanzi Li, Zhiyuan Liu, Tat-Seng Chua. († indicates corresponding author) Quicksviewer: An LMM for Efficient Video Understanding via Reinforced Compression of Video Cubes. Preprint
4. Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, Yuan Yao†, Yankai Lin, Zhiyuan Liu, Maosong Sun. († indicates corresponding author) GUICourse: From General Vision Language Models to Versatile GUI Agents. ACL 2025.
5. Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, Ning Ding. Process Reinforcement through Implicit Rewards. Preprint.
6. Tianyu Yu, Haoye Zhang, Qiming Li, Qixin Xu, Yuan Yao†, Da Chen, Xiaoman Lu, Ganqu Cui, Yunkai Dang, Taiwen He, Xiaocheng Feng, Jun Song, Bo Zheng, Zhiyuan Liu, Tat-Seng Chua, Maosong Sun. († indicates corresponding author) RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness. CVPR 2025. Highlights.
7. Yuan Yao, Tianyu Yu, Chongyi Wang, Junbo Cui, Bokai Xu, Hongji Zhu, Tianchi Cai, Fuwei Huang, Tianran Wang, Wenshuo Ma, etc. MiniCPM-o: A GPT-4o Level MLLM for Vision, Speech, and Multimodal Live Streaming on Your Phone. [Project: MiniCPM-o]

2024

1. Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, Maosong Sun. MiniCPM: Unveiling the Potential of End-side Large Language Models. COLM 2024. [Project: MiniCPM]
2. Zanlin Ni, Yulin Wang, Renping Zhou, Rui Lu, Jinyi Hu, Zhiyuan Liu, Yuan Yao†, Gao Huang. († indicates corresponding author) AdaNAT: Exploring Adaptive Policy for Token-Based Image Generation. ECCV 2024.
3. Ao Zhang, Yuan Yao†, Wei Ji, Zhiyuan Liu, Tat-Seng Chua. († indicates corresponding author) NExT-Chat: An LMM for Chat, Detection and Segmentation. ICML 2024.
4. Tianyu Yu, Yuan Yao†, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, Tat-Seng Chua. († indicates corresponding author) RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback. CVPR 2024. [Project: RLHF-V]
5. Zanlin Ni, Yulin Wang, Renping Zhou, Jiayi Guo, Jinyi Hu, Zhiyuan Liu, Shiji Song, Yuan Yao†, Gao Huang. († indicates corresponding author) Revisiting Non-Autoregressive Transformers for Efficient Image Synthesis. CVPR 2024.
6. Jinyi Hu, Yuan Yao†, Chongyi Wang, Yinxu Pan, Shan Wang, Qianyu Chen, Tianyu Yu, Yue Zhao, Xu Han, Jiao Xue, Dahai Li, Zhiyuan Liu†, Maosong Sun†. († indicates corresponding author) Large Multilingual Models Pivot Zero-shot Multimodal Learning across Languages. ICLR 2024. Spotlight. [Project: VisCPM]
7. Jifan Yu, Xiaozhi Wang, Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv, Hao Peng, Zijun Yao, Xiaohan Zhang, Hanming Li, Chunyang Li, Zheyuan Zhang, Yushi Bai, Yantao Liu, Amy Xin, Kaifeng Yun, Linlu GONG, Nianyi Lin, Jianhui Chen, Zhili Wu, Yunjia Qi, Weikai Li, Yong Guan, Kaisheng Zeng, Ji Qi, Hailong Jin, Jinxin Liu, Yu Gu, Yuan Yao, Ning Ding, Lei Hou, Zhiyuan Liu, Xu Bin, Jie Tang, Juanzi Li. KoLA: Carefully Benchmarking World Knowledge of Large Language Models. ICLR 2024. [Project: KoLA]
8. Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, Maosong Sun. CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models. AI Open. 2024.

2023

1. Ao Zhang, Yuan Yao†, Wei Ji, Zhiyuan Liu, Tat-Seng Chua. († indicates corresponding author) NExT-Chat: An LMM for Chat, Detection and Segmentation. Preprint.
2. Tianyu Yu, Jinyi Hu, Yuan Yao†, Haoye Zhang, Yue Zhao, Chongyi Wang, Shan Wang, Yinxu Pan, Jiao Xue, Dahai Li, Zhiyuan Liu†, Hai-Tao Zheng, Maosong Sun†. († indicates corresponding author) Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants. Preprint.
3. Ao Zhang, Hao Fei†, Yuan Yao†, Wei Ji, Li Li, Zhiyuan Liu, Tat-Seng Chua. († indicates corresponding author) Transfer Visual Prompt Generator across LLMs. NeurIPS 2023.
4. Yuan Yao, Tianyu Yu, Ao Zhang, Mengdi Li, Ruobing Xie, Cornelius Weber, Zhiyuan Liu, Haitao Zheng, Stefan Wermter, Tat-Seng Chua, Maosong Sun. Visually Grounded Commonsense Knowledge Acquisition. AAAI 2023. Oral presentation.

2022

1. Yuan Yao, Qianyu Chen, Ao Zhang, Wei Ji, Zhiyuan Liu, Tat-Seng Chua, Maosong Sun. (* indicates equal contribution) PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models. EMNLP 2022.
2. Ao Zhang*, Yuan Yao*, Qianyu Chen, Wei Ji, Zhiyuan Liu, Maosong Sun, Tat-Seng Chua. (* indicates equal contribution) Fine-Grained Scene Graph Generation with Data Transfer. ECCV 2022. Oral presentation.
3. Yuan Yao, Bowen Dong, Ao Zhang, Zhengyan Zhang, Ruobing Xie, Zhiyuan Liu, Leyu Lin, Maosong Sun, Jianyong Wang. Prompt Tuning for Discriminative Pre-trained Language Models. Findings of ACL 2022.
4. Zheni Zeng*, Yuan Yao*, Zhiyuan Liu and Maosong Sun. (* indicates equal contribution) A Deep-learning System Bridging Molecule Structure and Biomedical Text with Comprehension Comparable to Human Professionals. Nature Communications, 2022. Editors' Highlights.

2021

1. Yuan Yao, Jiaju Du, Yankai Lin , Peng Li, Zhiyuan Liu, Jie Zhou and Maosong Sun. (* indicates equal contribution) CodRED: A Cross-Document Relation Extraction Dataset for Acquiring Knowledge in the Wild. EMNLP 2021.
2. Yuan Yao, Ao Zhang, Xu Han, Mengdi Li, Cornelius Weber, Zhiyuan Liu, Stefan Wermter and Maosong Sun. (* indicates equal contribution) Visual Distant Supervision for Scene Graph Generation. ICCV 2021.
3. Fanchao Qi*, Yuan Yao*, Sophia Xu*, Zhiyuan Liu and Maosong Sun. (* indicates equal contribution) Turn the Combination Lock: Learnable Textual Backdoor Attacks via Word Substitution. ACL 2021.
4. Kai Zhang*, Yuan Yao*, Ruobing Xie, Xu Han, Zhiyuan Liu, Fen Lin, Leyu Lin and Maosong Sun. (* indicates equal contribution) Open Hierarchical Relation Extraction. NAACL-HLT 2021.
5. Yuan Yao, Haoxi Zhong, Zhengyan Zhang, Xu Han, Xiaozhi Wang, Kai Zhang, Chaojun Xiao, Guoyang Zeng, Zhiyuan Liu and Maosong Sun. (* indicates equal contribution) Adversarial Language Games for Advanced Natural Language Intelligence. AAAI 2021.
6. Zheni Zeng, Chaojun Xiao, Yuan Yao, Ruobing Xie, Zhiyuan Liu, Fen Lin, Leyu Lin and Maosong Sun. Knowledge Transfer via Pre-training for Recommendation: A Review and Prospect. Frontiers in Big Data, 2021.

2020

1. Bowen Dong*, Yuan Yao*, Ruobing Xie, Tianyu Gao, Xu Han, Zhiyuan Liu, Fen Lin, Leyu Lin and Maosong Sun. (* indicates equal contribution) Meta-Information Guided Meta-Learning for Few-Shot Relation Classification. COLING 2020.
2. Chaojun Xiao, Yuan Yao, Ruobing Xie, Xu Han, Zhiyuan Liu, Maosong Sun, Fen Lin and Leyu Lin. Denoising Relation Extraction from Document-level Distant Supervision. EMNLP 2020.
3. Stefan Heinrich, Yuan Yao, Tobias Hinz, Zhiyuan Liu, Thomas Hummel, Matthias Kerzel, Cornelius Weber and Stefan Wermter. Crossmodal Language Grounding in an Embodied Neurocognitive Model. Frontiers in Neurorobitics, 2020.

2019

1. Xu Han, Tianyu Gao, Yuan Yao, Demin Ye, Zhiyuan Liu, Maosong Sun. OpenNRE: An Open and Extensible Toolkit for Neural Relation Extraction. EMNLP 2019. Demo paper.
2. Ruidong Wu*, Yuan Yao*, Xu Han, Ruobing Xie, Zhiyuan Liu, Fen Lin, Leyu Lin, Maosong Sun. (* indicates equal contribution) Open Relation Extraction: Relational Knowledge Transfer from Supervised Data to Unsupervised Data. EMNLP 2019.
3. Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, Maosong Sun. (* indicates equal contribution) DocRED: A Large-Scale Document-Level Relation Extraction Dataset. ACL 2019.
4. Jiayuan Mao*, Yuan Yao*, Stefan Heinrich, Tobias Hinz, Cornelius Weber, Stefan Wermter, Zhiyuan Liu, Maosong Sun. (* indicates equal contribution) Bootstrapping Knowledge Graphs From Images and Text. Frontiers in Neurorobitics, 2019.
5. Ruobing Xie, Stefan Heinrich, Zhiyuan Liu, Cornelius Weber, Yuan Yao, Stefan Wermter, Maosong Sun. Integrating Image-based and Knowledge-based Representation Learning. IEEE Transactions on Cognitive and Developmental Systems, 2019.

2018

1. Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, Maosong Sun. FewRel: A Large-Scale Supervised Few-shot Relation Classification Dataset with State-of-the-Art Evaluation. EMNLP 2018.

EXPERIENCE

Postdoc Researcher

School of Computing,
National University of Singapore.
September 2023 - Present

Ph.D. in Computer Science

Department of Computer Science and Technology,
Tsinghua University, Beijing, China.
August 2018 - July 2023

Bachelor of Engineering

Department of Computer Science and Technology,
Tsinghua University, Beijing, China.
August 2014 - July 2018


AWARDS

Outstanding Doctoral Dissertation Award of Wu Wenjun AI Science and Technology Award, Chinese Association for Artificial Intelligence.
10 Major Sci-Tech Achievements, Zhongguancun Forum Annual Conference.
Spotlight Recipient of the WAIC Yunfan Award, World Artificial Intelligence Conference.
Intel China Academic Achievement Award, Intel.
HuggingFace Most Liked and Most Downloaded Open-Source AI Models, HuggingFace.
Outstanding Cases of Jointly Building a Community with a Shared Future in Cyberspace, World Internet Conference.
Tsinghua Shuimu Scholar, Tsinghua University.