EgoLife: Towards Egocentric Life Assistant

Jingkang Yang, Shuai Liu, Hongming Guo, Yuhao Dong, Xiamengwei Zhang,
Sicheng Zhang, Pengyun Wang, Zitang Zhou, Binzhu Xie, Ziyue Wang, Bei Ouyang,
Zhengyu Lin, Marco Cominelli, Zhongang Cai, Yuanhan Zhang, Peiyuan Zhang,
Fangzhou Hong, Joerg Widmer, Francesco Gringoli, Lei Yang, Bo Li, Ziwei Liu
1S-Lab, Nanyang Technological University, Singapore       2LMMs-Lab
3IMDEA Networks, Spain       4University of Brescia, Italy
arXiv

🤗

 Dataset Code

🖥

 Gradio Demo

🎞

 EgoLife Movie Mode
Since our arXiv submission is on hold, you can read our local PDF version first.

Abstract

We introduce EgoLife, a project to develop an egocentric life assistant that accompanies and enhances personal efficiency through AI-powered wearable glasses 👓. To lay the foundation for this assistant, we conducted a comprehensive data collection study where six participants lived together for one week, continuously recording their daily activities—including discussions 💬, shopping 🛍️, cooking 🍳, socializing 👥, and entertainment 🎮—using AI glasses for multimodal egocentric video capture, along with synchronized third-person-view video references. This effort resulted in the EgoLife Dataset 📖, a comprehensive 300-hour egocentric, interpersonal, multiview, and multimodal daily life dataset with intensive annotation. Leveraging this dataset, we introduce EgoLifeQA❓, a suite of 3K long-context, life-oriented question-answering tasks designed to provide meaningful assistance in daily life by addressing practical questions such as recalling past relevant events, monitoring health habits, and offering personalized recommendations.

To address the key technical challenges of 1) developing robust visual-audio models for egocentric data, 2) enabling identity recognition, and 3) facilitating long-context question answering over extensive temporal information, we introduce EgoBulter 🫡, an integrated system comprising EgoGPT 🧠 and EgoRAG 🔍. EgoGPT is a vision-language model trained on egocentric datasets, achieving state-of-the-art performance on egocentric video understanding. EgoRAG is a retrieval-based component that supports answering ultra-long-context questions. Our experimental studies verify their working mechanisms and reveal critical factors and bottlenecks, guiding future improvements. By releasing our datasets, models, and benchmarks, we aim to stimulate further research in egocentric AI assistants.

The EgoLife Dataset

The EgoLife dataset captures the lives of six participants over seven days,
recorded through Meta Aria glasses.
Living together with a shared goal of organizing an Earth Day Event,
their experiences are preserved in over 250+ hours of rich, everyday data.
This includes egocentric, interpersonal and multiview moments,
offering a detailed and intimate view of their lives and teamwork.

What EgoLife Captures and Annotates

Through Meta Aria glasses and other sensors, we collect rich multimodal data including first-person video, audio, eye tracking, motion, and third-person videos captured by mounted 15 Exo cameras. The dataset features synchronized egocentric and third-person views, detailed activity timelines, and 3D reconstructions of living spaces - providing unprecedented insight into natural human behavior and interactions in a shared living environment. Annotations include audio transcripts, dense captioning, and a long caption each 30 seconds.

EgoLifeQA: Challenges for Personal Life Assistant

Based on the storyline of EgoLife dataset, we created EgoLifeQA with 3K life-oriented questions requiring ultra-long context understanding. We ensure that 66% of questions need looking back over 2 hours of history and over 15% require reviewing more than 24 hours of past activities. We specifically design the following 5 types of QAs to evaluate the performance of the life assistant.

EgoGPT & EgoRAG for AI Assistants

Other Interesting Info

Join Us!

EgoLife is an evolving initiative that aims to push the boundaries of egocentric AI. We are actively working to enhance every aspect of the project - from expanding our dataset and enriching annotations to advancing our omnimodal models and refining the long-range system II architecture. We warmly welcome researchers who share our vision to join this exciting journey. If you're interested in contributing to the future of egocentric AI assistants, please reach out to us at jingkang001@e.ntu.edu.sg. Together, let's bring truly personalized AI assistance into reality!

BibTeX

@inproceedings{yang2025egolife,
  title={EgoLife: Towards Egocentric Life Assistant},
  author={Yang, Jingkang and Liu, Shuai and Guo, Hongming and Dong, Yuhao and Zhang, Xiamengwei and Zhang, Sicheng and Wang, Pengyun and Zhou, Zitang and Xie, Binzhu and Wang, Ziyue and Ouyang, Bei and Lin, Zhengyu and Cominelli, Marco and Cai, Zhongang and Zhang, Yuanhan and Zhang, Peiyuan and Hong, Fangzhou and Widmer, Joerg and Gringoli, Francesco and Yang, Lei and Li, Bo and Liu, Ziwei},
  booktitle={The IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2025},
}