MediaEval Multimedia Benchmark

MediaEval Multimedia Benchmark: Looking back on 2010 and looking forward to 2011

MediaEval http://www.multimediaeval.org is a benchmarking initiative dedicated to evaluating new algorithms for multimedia access and retrieval. MediaEval puts emphasis on context and topic of video and aims at moving beyond what is explicitly depicted in the visual channel towards overall meaning and relevance to users. It attracts participants who are interested in multimodal approaches to multimedia involving, e.g., speech recognition, multimedia content analysis, user-contributed information (tags, tweets), viewer affective response, social networks, temporal and geo-coordinates. MediaEval was established in 2008 as VideoCLEF, a track of CLEF, and in 2010 became an independent benchmarking initiative.

Group photo from MediaEval 2010 (photo by Jaeyoung Choi)

Gerald Friedland (photo by Jaeyoung Choi)

The MediaEval 2010 workshop was held just before ACM Multimedia 2010 on Sunday, October 24, 2010 in a medieval convent in Pisa, Italy. The workshop was attended by 29 participants from nine countries and three continents. The day was dedicated to talks and discussion of the results that were achieved on MediaEval 2010 tasks. These results comprised algorithms that were developed for automatic annotation of video: prediction of geo-location, of user-contributed tags, of subject labels and of viewer-reported boredom.

The talks revealed the value of user-contributed metadata for automatic tag prediction. Co-occurrence information and information derived from external sources such as Wikipedia proved productive. Features derived from the visual and audio channel are also very useful, but remain more challenging to exploit. It will be the continued focus of future MediaEval benchmarks to encourage development of the most useful combinations of diverse and multimodal information sources.

Highlights of MediaEval 2010 results will be presented in the special session "Automatic Tagging and Geo-Tagging in Video Collections and Communities" that will be held at ICMR 2011 (http://www.icmr2011.org), the First ACM International Conference on Multimedia Retrieval. ICMR 2011 will be held 17-20 April 2011 in Trento, Italy.

Sign-up for MediaEval 2011 is open now through 31 May 2011 on the MediaEval website http://www.multimediaeval.org The website also gives the full schedule for MediaEval 2011 and descriptions of the tasks that are offered this year. Sign-up for MediaEval is open to any group that wishes to participate. Participants receive task descriptions, data sets and supporting resources and develop algorithms that generate task results. When they receive the result evaluation from the MediaEval organizers, they write up a working notes paper and attend the MediaEval 2011 workshop. The MediaEval 2011 workshop is an official satellite event of Interspeech 2011. http://www.interspeech2011.org and will be held 1&2 September 2011, again in Pisa, Italy.

MediaEval is coordinated by the EU FP7 PetaMedia Network of Excellence http://www.petamedia.eu and also by the ICT Labs of EIT http://eit.ictlabs.eu/ and is made possible by the many projects, institutions and researchers that contribute to the organization of the individual tasks.