INEX 2011 Snippet Retrieval Track
Final results available.
The goal of the snippet retrieval track is to determine how best to generate informative snippets for search results. Such snippets should provide sufficient information to allow the user to determine the relevance of each document, without needing to view the document itself. Participating organisations will compare both the effectiveness of their focused retrieval systems as well as the effectiveness of their snippet generation systems, to others.
The snippet retrieval track will use the INEX Wikipedia collection. Topics will be recycled from previous ad hoc tracks.
Participating organisations will submit a ranked list of documents, and corresponding snippets. For those organisations uninterested in developing their own focused retrieval system, a reference run will be provided, consisting of a ranked list of documents, for which snippets should be created.
Each run will be submitted in the form of an XML file (format described below). Each submission should contain 500 snippets per topic, with a maximum of 300 characters per snippet. The snippets themselves may be created in any way you wish – they may consist of summaries, passages from the document, or any other text at all.
Participants may submit more than one submission; however submissions must be ranked in order of importance, as not all submissions may be evaluated.
The DTD for the submission format is as follows.
Each submission must contain the following:
<!ELEMENT inex-snippet-submission (description,topic+)>
participant-id CDATA #REQUIRED
run-id CDATA #REQUIRED
<!ELEMENT description (#PCDATA)>
<!ELEMENT topic (snippet+)>
topic-id CDATA #REQUIRED
<!ELEMENT snippet (#PCDATA)>
doc-id CDATA #REQUIRED
rsv CDATA #REQUIRED
Every run should contain the results for each topic, conforming to the following:
- participant-id: The participant number of the submitting institution (available here)
- run-id: A run ID, which must be unique across all submissions sent from a single participating organisation.
- description: A description of the approach used.
An example submission in the correct format is given below.
- topic: Contains a ranked list of snippets, ordered by decreasing level of relevance of the document which they represent.
- topic-id: The ID number of the topic.
- snippet: A snippet representing a document (defined by doc-id).
- doc-id: The document ID.
- rsv: The document's score.
<!DOCTYPE inex-snippet-submission SYSTEM "inex-snippet-submission.dtd">
<inex-snippet-submission participant-id="20" run-id="Qut_Snippet_Run_01">
<description>A description of the approach used.</description>
<snippet doc-id="16080300" rsv="0.9999">...</snippet>
<snippet doc-id="16371300" rsv="0.9998">...</snippet>
<snippet doc-id="1686300" rsv="0.9999">...</snippet>
<snippet doc-id="1751300" rsv="0.9997">...</snippet>
Documents for each topic will be manually assessed for relevance based on the snippets alone, as the goal is to determine the effectiveness of the snippet’s ability to provide sufficient information about the document behind the snippet. Each topic within a submission will be assigned an assessor, who will assess each document within the ranked list, based on the snippet alone, until a predetermined number of relevant documents are found, or until there are no documents remaining. The assignment of assessors to topics will be shuffled for each submission to avoid bias introduced by assessors judging the same topic twice, and to allow each submission to be judged by multiple assessors.
It is hoped that crowd-sourcing may be used for assessment; however
participants should also expect to perform a number of assessments, depending on the number of runs they submit. As the assessment is based on short snippets rather than whole documents, the assessment load will be relatively light.
Download the assessment tool (with instructions)
The primary evaluation metric, and the one which determines the ranking, is the geometric mean of recall and negative recall (GM), averaged over all topics: sqrt(TN/(TN+FP) * TP/(TP+FN)).
Other metrics include:
- Mean prediction accuracy (MPA) – the percentage of results the assessor correctly assessed, averaged over all topics: (TP+TN)/(TP+FN+TN+FP)
- Mean normalised prediction accuracy (MNPA) – the average of the relevant results correctly assessed, and the irrelevant results correctly assessed, averaged over all topics: 0.5*TP/(TP+FN) + 0.5*TN/(TN+FP)
- Recall – the percentage of relevant documents correctly assessed, averaged over all topics: TP/(TP+FN)
- Negative recall (NR) or specificity – the percentage of irrelevant documents correctly assessed, averaged over all topics: TN/(TN+FP)
- Positive agreement (PA) – the conditional probability of agreement between snippet assessor and document assessor (i.e. ground truth), given that one of the two judged relevant. Equivalent to F1 score: 2*TP/(2*TP+FP+FN)
- Negative agreement (NA) – the conditional probability of agreement between snippet assessor and document assessor (i.e. ground truth), given that one of the two judged irrelevant: 2*TN/(2*TN+FP+FN)
- July 15: Release of training topics.
- Aug 13: Release of final topics.
- Oct 7: Submission deadline.
- Nov 11: Relevance assessments due.
- Nov 18: Release of evaluation results.
University of Otago