Evaluation Showcase
Three live evaluation events will take place at the ACM CIVR 2007: Video Retrieval, Image Retrieval and Video Copy Detection.
The evaluation showcases will follow the following general plan:
- For the Image Retrieval and Copy Detection showcase image databases will be made available to participants beforehand so that they can be loaded into their retrieval systems. Some example queries will also be provided. For the VideOlympics it is assumed that participants bring their own copy of the NIST TRECVID 2006 test set.
- At the demo session, groups will bring their retrieval systems and set them up.
- A set of queries will be handed out at the event. The authors of the retrieval systems as well as conference attendees will be given the task of finding the target videos/images/copies for the queries on the various search engines.
- For evaluation, metrics such as the amount of time required to find the target videos/images/copies and the
percentage of the correct targets found will be measured. We will also attempt to evaluate the ease of use of each system based on user feedback.
- Wireless internet connection will be provided by the local CIVR organization.
VideOlympics: Video Retrieval Showcase
Contact: Cees Snoek, University of Amsterdam, The Netherlands.
Please see our separate website on the VideOlympics Showcase.
Image Retrieval Showcase
Contact: Allan Hanbury, Vienna University of Technology, Austria.
Database: IAPR-TC12 dataset of 20,000 vacation images with annotations in two languages. Some of the annotations will be edited or removed to make the task more realistic.
Types of Queries: Visual and text queries will be provided. Examples range from the classic "Find similar images to the one shown" to questions such as "What is the name of the monument in this image?" and requests such as "Find images of snowy mountains".
Please see our separate website on the Image Retrieval Showcase.
Video Copy Detection Showcase
Contact: Alexis Joly, INRIA - Rocquencourt, France.
Database: About 150 hours of video materials comming from different sources: web video clips, TV archives, movies.
Types of Queries: The queries will be clustered in two level of difficulties:
- Movie copies like queries: Real copies of entire long videos (from 1 minute to 3 hours). The data can be reencoded, noised, or slightly retouched. Most difficult queries could be movies re-acquired by a cam-corder.
- TV stream like queries: Several sequences will be choosen in the database and post-processed by professional audiovisual archivists with a professional video editing software. Applied transformations can be very diversified: cropping; fade cuts; flips; insertion of logos, borders, texts, moving texts, moving characters, etc.; cholorimetric manipulations: contrast, gamma, etc.; Queries include parts of several videos belonging or not to the database. Sequences belonging to the database must be identified and localized by their start and end time codes.
Please see our separate website on the Video Copy Detection Showcase.
Sponsor
The showcase events are sponsored by the MUSCLE Network of Excellence.
