In recent years, various software and hardware tools have entered the consumer market which enable users to log data about their lives on a continuous basis. Popular examples include self-tracking devices or apps such as Fitbit or Garmin that allow users to keep track of their physical activities or to monitor their biometrics. The process of gathering such multi-modal data from multiple sources is also referred to as lifelogging.

Due to the constant stream of data being captured, lifelogging can result in the creation of large personal archives that are too large for manual organization. Consequently, automated approaches to handle such data are needed. However, due to privacy concerns, advances in the field have been limited by the lack of shared test collections.

Aiming to promote further research on novel approaches to multi-modal personal data analytics and retrieval, we organized a comparative benchmarking exercise, Lifelog, that ran between 2015 and 2022 as part of the evaluation conference NTCIR. Several Lifelog datasets were released and participants could work on various sub-tasks to tackle different challenges related to Lifelog retrieval. In this keynote presentation, I will give an overview of these sub-tasks and reflect on lessons learned.

In this paper, we present Memento 2.0, an improved version of our system which first participated in the Lifelog Search Challenge 2021. Memento 2.0 employs image-text embeddings derived from two CLIP models (ViT-L/14 and ResNet-50x64) and adopts a weighted ensemble approach to derive a combined final ranking. Our approach significantly improves the performance over the baseline LSC'21 system. We additionally make important updates to the system's user interface after analysing the shortcomings to make it more efficient and better suited to the needs of the Lifelog Search Challenge.

Research on retrieving data and analyzing lifelogs revealed to be a very complex task, and the interdisciplinary challenges to be tackled have boosted increasing attention from the scientific community in information retrieval and lifelogging. The Lifelog Search Challenge is an international competition for lifelog retrieval in which researchers propose their approaches and compete to solve lifelog retrieval challenges and evaluate the effectiveness of their systems. In this paper, we present the MEMORIA computational tool to participate for the first time in the Lifelog Search Challenge 2022. The information retrieval is based on the search of keywords and time periods and several computer vision methods are used to process visual lifelogs, from pre-processing algorithms to feature extraction methods, in order to enrich the annotation of the lifelogs. Preliminary experimental results of the user interaction with our retrieval module are presented, confirming the effectiveness of the proposed approach and showing the most relevant functionalities of the system.

In this paper, we introduce LifeSeeker 4.0 - an interactive lifelog retrieval system developed for the fifth annual Lifelog Search Challenge (LSC'22). In LifeSeeker 4.0, we focus on enhancing our previous system to allow users who have little to no knowledge of underlying system functioning and lifelog data to use it with ease by employing a Contrastive Language-Image Pre-training (CLIP) model. Furthermore, we have exploited the music metadata to facilitate searches that may incorporate emotion. Event clustering is also improved in this version to increase user experience by reducing the occurrence of repeated images, and hence decreasing the search time.

Building a retrieval system with lifelogging data is more complicated than with ordinary data due to the redundancies, blurriness, massive amount of data, various sources of information accompanying lifelogging data, and especially the ad-hoc nature of queries. The Lifelog Search Challenge (LSC) is a benchmarking challenge that encourages researchers and developers to push the boundaries in lifelog retrieval. For LSC'22, we develop FIRST 3.0, a novel and flexible system that leverages expressive cross-domain embeddings to enhance the searching process. Our system aims to adaptively capture the semantics of an image at different levels of detail. We also propose to augment our system with an external search engine to help our system with initial visual examples for unfamiliar concepts. Finally, we organize image data in hierarchical clusters based on their visual similarity and location to assist users in data exploration. Experiments show that our system is both fast and effective in handling various retrieval scenarios.

In this paper, we present the iteration of the multimedia retrieval system vitrivr participating at LSC 2022. vitrivr is a general-purpose retrieval system which has previously participated at LSC. We describe the system architecture and functionality, and show initial results based on the test and validation topics.

Developing interactive lifelog retrieval systems is a growing research area. There are many international competitions for lifelog retrieval that encourage researchers to build effective systems that can address the multimodal retrieval challenge of lifelogs. The Lifelog Search Challenge (LSC) was first organised in 2018 and is currently the only interactive benchmarking evaluation for lifelog retrieval systems. Participating systems should have an accurate search engine and a user-friendly interface that can help users to retrieve relevant content. In this paper, we upgrade our previous MyScéal, which was the top performing system in LSC'20 and LSC'21, and present E-MyScéal for LSC'22, which includes a completely different search engine. Instead of using visual concepts for retrieval such as MyScéal, the new E-MyScéal employs an embedding technique that facilitates novice users who are not familiar with the concepts. Our experiments show that the new search engine can find relevant images in the first place in the ranked list, four a quarter of the LSC'21 queries (26%) by using just the first hint from the textual information need. Regarding the user interface, we still keep the simple non-faceted design as in the previous version but improve the event view browsing in order to better support novice users.

The multimodal nature of lifelog data poses unique challenges for analysis, indexing and interactive retrieval. To address these challenges, the Lifelog Search Challenge (LSC) is an annual evaluation campaign allowing interactive retrieval systems to explore new ideas and measure their performance against each other.

This paper describes the virtual reality (VR) multimedia retrieval system vitrivr-VR, with a focus on aspects relevant to the LSC'22, especially the user interaction in VR, the formulation of typical LSC queries, and different options to explore the retrieval results in VR.

Voxento is an interactive voice-based retrieval system for lifelogs which has been redeveloped and optimised to participate in the fifth Lifelog Search Challenge LSC'22, at ACM ICMR'22. Based on the previous experience in the LSC competition and ranked in the top 4 in the last LSC'21 competition among 17 participants, we present a revised version of Voxento to address the critical points to improve the efficiency of retrieval tasks in lifelog datasets. Basically, Voxento provides a spoken interface to the lifelog data, which facilitates an expert and novice user to interact with a personal lifelog using a range of vocal commands and interactions. Briefly, we made some important improvements to support both the retrieval of content and system interaction. This latest version has been enhanced with the addition of a text-based search feature, new filters based on new metadata provided in lifelog data, rich visual information and features and enhanced speech query. Also, the data preparation tasks comprised a new function to reduce the number of non-relevant images and the latest CLIP model version used to derive features from images. The long term development of Voxento includes a lifelog retrieval that supports speech and conversation interaction with less physical actions required by users such as using a mouse. The system presented here uses a desktop computer in order to participate in the LSC'22 competition with the option to use voice interaction or standard text-based retrieval.

Lifelogging creates substantial data archives that are challenging to manage and search. The annual Lifelog Search Challenge (LSC) aims at improving this situation by encouraging international teams to create interactive retrieval systems for searching large lifelog databases. The LSC challenge is held as a live event co-located at the ACM International Conference on Multimedia Retrieval (ICMR), where participating teams compete against each other by solving time-based retrieval tasks. In this paper, we present an improved version of lifeXplore -- our system already participating since LSC2018. For LSC2022, we focus on improving the result presentation as well as the system's interface.