Egocentric Vision for human behavior understanding

January 2, 2026

Overview

Egocentric vision is a rapidly growing field, in which deep learning models are used to process videos that captures human activities from the perspective of the camera wearer, offering a unique viewpoint on their actions. This field has seen rapid development due to the release of several large-scale egocentric vision datasets, which have rich annotations for tasks like action recognition, action anticipation, next active object prediction, action segmentation, episodic memory and many more. Our research in egocentric vision focuses on the processing of videos collected in first person, with the goal of better understanding humans and learning how to transfer human skills to robot manipulators. To achieve this, we are developing new methods and algorithms that can effectively analyze and interpret egocentric video data.

Key results

NeurIPS26

A General Concept-based Decomposition for Vision–Language EmbeddingsA general way to decompose vision–language embeddings into interpretable concepts, in collaboration with EPFL.

NeurIPS26

Learning to Evolve Scenes: Reasoning about Human Activities with Scene GraphsGiven a scene graph and a description of an activity, GLEN predicts how the scene will change, turning scene graphs into a common language for long-horizon reasoning.

Project page

ICCV25

HiERO: understanding the hierarchy of human behavior enhances reasoning on egocentric videosLearns, without direct supervision, a hierarchical representation of human activities that improves reasoning on long egocentric videos.

Project page

IEEE T-PAMI

Hier-EgoPack: Hierarchical Egocentric Video Understanding with Diverse Task PerspectivesA single model learns many egocentric video tasks and stores its knowledge in a reusable "backpack of skills" that makes new tasks easier to learn.

Paper

Elsevier PRL

Egocentric zone-aware action recognition across environmentsMakes egocentric action recognition robust to new environments by modelling the zones where activities take place.

Project page

ECCV24

AMEGO: Active Memory from long EGOcentric videosBuilds a structured memory of the objects and interactions in long egocentric videos, enabling reasoning over long time spans, in collaboration with the University of Bristol.

Paper

Key Findings

We have made several steps in our research on egocentric vision. One of our primary findings is that we have shown how knowledge sharing and abstraction across different tasks can significantly improve the performance and efficiency of egocentric video processing models, and can provide a better holistic understanding of the human behavior. (Play with our code here.) We have also studied the feasibility to generate - without direct supervision - structured representations of the human activity, which proved to be extremely helpful for a variety of downstream tasks, such as Visual Question Answering, but also as a privileged structured representation for transferring human skills to intelligent manipulators. We are currently focusing on generalizing our findings to more complex and long-horizon activities typical of daily life, either scripted (e.g. procedural) or generic.

Challenges and Learnings

There are several challenges that need to be addressed in order to further advance the field of egocentric vision. First, egocentric vision data is often noisy and cluttered, which can make it difficult to accurately recognize human actions and predict future actions. Second, it can be difficult to learn transferable representations of human actions from egocentric vision data, as the appearance of human actions can vary significantly depending on the viewpoint and the context. Third, it can be difficult to model the environment and the interaction between objects from egocentric vision data, as the viewpoint of the camera wearer is constantly changing. Despite these challenges, we have learned a great deal from our research. First, we have learned that it is important to develop robust methods that can effectively handle the noise and clutter in egocentric vision data. Second, it is important to develop methods that can learn transferable representations of human actions from egocentric vision data, suitably designed to prune all the confouding factors in the scene and preserve only what matters for the current interaction between the human and the environment. We believe that our research in egocentric vision is in a good path towards a better description of human behavior in daily living activities, with the specific goal of generating trasnferrable knowledge from which intelligent manipulators can easily learn. We are confident that our research will eventually lead to the development of more intelligent and capable robots that can safely and effectively interact with humans in the real world.