Jiayi Pan, a senior student of the University of Michigan-Shanghai Jiao Tong University Joint Institute (UM-SJTU JI, JI hereafter) pursuing a dual-degree program at UM, emerged as a first-prize winner of the recent Amazon Alexa SimBot Challenge, after his Team SEAGULL (Situated and Embodied Agents with the ability of GroUnded Language Learning) representing UM defeated the other four finalist teams and clinched the champions of the competition.
Jiayi Pan (second person from right) with his teammates of Team SEAGULL
The UM team, consisting of nine computer science, engineering, and robotics researchers, including Pan and a JI Class of 2021 graduate, Ziqiao Ma, won the $500,000 first-place prize of the competition for their innovative design of the SEAGULL SimBot, demonstrating the potential for interactive embodied agents to take on increasingly complex tasks.
Virtual environment of Alexa Arena
The Amazon-sponsored SimBot Challenge tasked teams with developing virtual robots that can complete complicated, real-world tasks via continuous learning, demonstrated via an interactive puzzle game triggered by the prompt, “Alexa, play with robot.” The challenge included teams from elite institutions across the globe, bringing together the brightest minds in the field to explore and extend the possibilities of virtual assistants. Five university teams were selected to participate in the final live-interactions phase of the global competition.
“As the only undergraduate student in the team, I mainly undertook the development of the SEAGULL’s visual perception module, which involved various levels of object recognition and scene reconstruction based on the images (a series of pictures) captured by SEAGULL. This experience has enhanced my engineering skills and communication abilities. Through my involvement in projects like SimBot, I gradually discovered my true career interest. In the future, I plan to pursue further studies and pursue a Ph.D. in Artificial Intelligence at the University of California, Berkeley campus,” said Pan.
Pan’s teammate, JI alumnus Ma who is currently a second-year doctoral student at UM said:” Participating in the SEAGULL project has strengthened my interest in embodied artificial intelligence and language interaction intelligence, while fostering great anticipation for the future of artificial intelligence. Through the SimBot project, I have had the opportunity to intern at the Alexa AI team and contribute to training leading joint models for visual and language tasks. I plan to further delve into research in the field of artificial intelligence, complete a Ph.D. in computer science, and make more contributions in the areas of language understanding and natural language processing.”
