Abstract
In recent years, we have witnessed a profound transformation in the learning paradigm of deep neural networks, especially in the applications of large language models and other foundation models. While conventional deep learning methodologies maintain their significance, they are now augmented by emergent model-centric approaches such as transferring knowledge, editing models, fusing models, or leveraging unlabelled data to tune models. Among these advances, deep model fusion techniques have demonstrated particular efficacy in boosting model performance, accelerating training, and mitigating the dependency on annotated datasets. Nevertheless, substantial challenges persist in the research and application of effective fusion methodologies and their scalability to large-scale foundation models. In this talk, we systematically present the recent advances in deep model fusion techniques. We provide a comprehensive taxonomical framework for categorizing existing model fusion approaches, and introduce our recent developments, including (1) weight learning-based model fusion and data-adaptive MoE upscaling, (2) subspace learning approaches to model fusion, and (3) enhanced multi-task model fusion incorporating pre- and post-finetuning to minimize representation bias between the merged model and task-specific models.
About the speaker
Prof. Dacheng Tao is the Distinguished University Professor and the Inaugural Director of the Generative AI Lab in the College of Computing and Data Science at Nanyang Technological University. He was an Australian Laureate Fellow and the founding director of the Sydney AI Centre at the University of Sydney, the inaugural director of JD Explore Academy and senior vice president at JD.com, and the chief AI scientist at UBTECH Robotics. He mainly applies statistics and mathematics to artificial intelligence, and his research is detailed in one monograph and over 300 publications. His publications have been cited over 160K times and he has an h-index 180+ in Google Scholar. He received the 2015 and 2020 Australian Eureka Prize, the 2018 IEEE ICDM Research Contributions Award, 2020 research super star by The Australian, the 2019 Diploma of The Polish Neural Network Society, and the 2021 IEEE Computer Society McCluskey Technical Achievement Award. He is a Fellow of the Australian Academy of Science, ACM and IEEE.
