The PIR-CLEF workshop will provide a forum for the proposal and analysis of methodologies for the comparative evaluation of methods for personalised information retrieval (PIR). AT CLEF 2017, PIR-CLEF will consist of a workshop to be held at the CLEF 2017 conference, and a pilot Lab task to be run during the CLEF 2017 campaign. The pilot task is intended allow groups to experiment with a common collection intended to enable the Crandfield style evaluation of PIR. The outcomes of this pilot activity are intended to provide inputs for the final design of a full Lab at CLEF 2018.
The pilot test collection will provide all the traditional components in a laboratory-based evaluation experiment, including topics and relevance judgments and access to a text documents. To support the personalisation element of the task, it will be accompanied by a set of user-related information for modelling and introducing profiles in the evaluation experiment. Specifically, this personal information will consist of:
- user personal information: including gender, age range, native language, and occupation.
- search logs: which contain the history of the user’s interactions with a search engine.
- the user’s documents of interest: provided as useful and raw sources to extract topical user preferences.
- basic user profile representations in the form of bag-of-words will be also provided with the aim of offering a basic model of the user’s topical interests.
- user satisfaction: a satisfaction grade decided by the user and providing a feedback on the ranking of documents.
We will use ClueWeb12, which contains over 730 million Web pages, as the basic repository to extract the above mentioned user-related information. Recognising that operating a ClueWeb12 service is a significant undertaking, API access to an existing ClueWeb12 search service will be made available to participants.
The 2017 edition of PIR-CLEF Lab will consist of:
- A one day workshop, to include discuss of comparative evaluation of PIR, including a report on the PIR-CLEF pilot with participant presentations, invited presentations, and a discussion of potential PIR tasks for CLEF 2018.
- A pilot PIR task intended to enable practical exploration of our proposed PIR evaluation methodology, with the intention of offering a fully tuned Lab at CLEF 2018.