INEX 2012 Relevance Feedback Track

Overview

The goal of the Relevance Feedback track is to determine how to best apply (focused) relevance feedback provided by users to provide the user with better results within the same search session.

To participate in the Relevance Feedback track, organisations will implement software solutions, then run them using the supplied Evaluation Platform. The Evaluation Platform will then automatically submit the results online. Unlike in previous versions of the Relevance Feedback track, there is no need to submit the software solution. It does not have to be portable to other systems and does not have to be a single program. The document collection can be indexed in advance so that it does not have to be rerun each time.

Use Case

The use-case of this track is a single user searching with a particular query in an information retrieval system that supports relevance feedback. The user highlights relevant passages of text in returned documents (if any) and provides this feedback to the information retrieval system. The IR system re-ranks the remainder of the unseen results list to provide more relevant results to the user. The exact manner in which this is implemented is not of concern in this evaluation; here we test the ability of the system to use focused relevance feedback to improve the ranking of previously unseen results.

Test Collection

The relevance feedback track will use the Wikipedia XML Corpus as the test collection. Evaluation will be based on a collection of relevance assessments gathered from assessments used to judge previous INEX Ad Hoc tracks. Two topic sets (a training set and an evaluation set) will be provided with the evaluation platform. The evaluation set should not be used for testing as the results will be submitted online, although multiple submissions may be made.

Resources

The evaluation software for the Relevance Feedback track, complete with documentation and sample submissions, is available here.

Task

Participating organisations will create one or more Relevance Feedback Modules (RFMs) intended to rank a collection of documents with a query while incrementally responding to explicit user feedback on the relevance of the results presented to the user. These RFMs will be implemented as executable files that the Evaluation Platform (EP) will interact with through input/output stream pipes, simulating a user search session. To evaluate the RFM, participants will run the (Java) application and provide it with the path to the RFM.

The EP will run the RFM and provide it with a topic, as a single line of text ending with a newline. The RFM will respond with the document ID of the first document to present to the user. The EP will then respond with the number of relevant passages within the document, again on a single line. If there is at least 1, the EP will provide the RFM with the text from those passages, on one line each. The RFM will then present the document ID of the next document. When the RFM has presented all the documents it wishes to present, it will supply the text EOF on a line instead of a document ID. The RFM will then present the RFM with a new topic or the text EOF if there are no more topics to evaluate.

The EP will retain the presentation order of documents as generated by the RFM. This order will then be evaluated as a submission using the trec_eval evaluation tool.

In previous versions of the Relevance Feedback track, a reduced collection was used to avoid long running times. As the evaluation platform does not require the document collection to be present for evaluation to work, the full Wikipedia collection can be used. It may be advisable to pre-index the document collection so that the RFM can load the index for the evaluation instead of having to re-index all the documents each time.

Protocol

The evaluation platform and the participating relevance feedback module will communicate using a pipe, a standard feature of all modern operating systems. Hence, any programming language capable of creating an executable that can read from standard input and write to standard output would be suitable for creating a relevance feedback module for the task.

Each 'message' from the evaluation platform or the relevance feedback module will be in the form of a single line of text ending in a linefeed character. The meaning of the line of text will be derived from the context in which it is submitted.

The evaluation platform communicates first, providing a topic line. This line will either contain the text of the topic or the text EOF, signalling to the RFM that the evaluation is over and it may exit. The RFM will respond with a document line. This line will contain either a document ID or the text EOF, signalling to the EP that the RFM has finished presenting documents for the current topic and is ready to move on to the next topic. If a document ID is presented, the EP will respond with feedback.

Feedback will be provided in the form of a line with a number indicating the number of passages of relevant text found in the document. If that number was 0, the document was not relevant and the RFM should provide the next document ID. Otherwise, the EP will immediately follow up the number with that many passages of feedback text, each on a single line. The feedback text will be stripped of characters outside the ASCII printable range of 32-127. Note that these lines potentially be as large as the largest document in the collection. After all the lines of feedback have been sent, the RFM is expected to respond with another document.

A sample submission (with source code) will be provided with the evaluation platform.

Schedule

The submission deadline for the track is the 29th of July, 2012, at 11:59pm (any timezone). Submissions will be generated and submitted automatically by using the evaluation tool.

Organizers

Shlomo Geva
QUT
s.geva@qut.edu.au

Timothy Chappell
QUT
t.chappell@connect.qut.edu.au
Imprint | Contact someone about INEX