About me and AIGC Research

I’m a graduate student from School of Information Science and Technology (SIST), ShanghaiTech University. My research interests focus on AI+Art, includes Computer Vision, Computer Graphics, Machine Learning, and Computational Photography. I am forming the AI4C Labs (AI for Creativity Labs), which aims to empower AI to design ,power artistic creation with AI-tech, and study the nature of human creativity. I am running a WeChat Official Account, called AIGC Research, to comprehensively track the academic frontier of AIGC through the PaperDaily series. As of 2024.4.22, it has reached 2200+ followers, and maintains a continuous upward trend, welcome your follow and support!

AI4C Research Plan

Following the idea of “exploring AIGC technology to power the artistic creation and content production, and promote the era of independent creation”, we will take film-level animation as our starting point, because “animation” is a masterpiece of artistic expression, including music, picture, color, story, motion effect, emotion and other all-round elements. It provides rich and comprehensive “learning materials” for AI to fully understand creation. And “film-level” means extreme artistic performance and higher precision technical level (the difference between video, TV and film is the expression of various artistic languages, such as camera language), which will land research results in the industry.

My future research work can be simply described in two aspects: the general level and the application level. The general level aims to give AI the ability of design in creation, which we call “AI with Design Thinking”; The application level aims to apply the current generative AI with design intelligence to the creative workflows, liberate human creativity, improve work efficiency, and thus promote the transformation from “Workflow” to “Creative-flow”, that is, “AI in Creative Workflow”.

1 General-tech Level

General technical level (AI with Design Thinking) starts from the perspective of “understanding the human creative process”. The director’s idea of animation production is to a large extent the overall imagination of sound, painting and story, which means that animation naturally has an explicit relationship to match all elements such as music, picture, color, story, dynamic effect and emotion. Such an ability of multi-modal perception and to match design elements is an important step towards design intelligence.

2 Application-tech Level

Application technical level (AI in Creative Workflow) starts from the perspective of “the production process of digital art assets”. Traditional 2D and 3D animation production still has a lot of repetitive labor, such as 3D modeling from scratch, manual intermediate frame supplement, etc. How to automate these tasks and explore more diverse ways of HCI (human-computer interaction) will become the focus of the application of AIGC technology in film-level animation production. Specifically, from the perspective of data dimension and form, it can be divided into 1D music, 2D images, 3D graphics and 4D video. The current mainstream HCI is text-driven generation, and more input-output objects in HCI will be expanded in the future, according to specific situations.

3 Fundamental Research

So, how do we implement the idea above? This requires us to understand how the current powerful large generative AI-model works, which is called Interpretability research. In my opinion, it can be divided into microscopic interpretability, mesoscopic interpretability and macro interpretability.Research on the interpretability of large models can help us improve current deep learning algorithms and even streamline deep models, thus achieve “more with less”.

In addition to studying the Interpretability of existing technologies and carrying out improved innovation and application, we can also learn from other disciplines. Drawing on the existing great neuroscience theories, we can achieve subversive innovative breakthroughs by building mathematical models and neural network algorithms related to different brain function regions and brain networks (simulation-driven/object-oriented brain-like research).Using AI technology to assist the exploration of neural mechanisms, namely AI for Science (Neuroscience), and expressing neural mechanism as a mathematical model to improve AI algorithms (computational neuroscience).