Call for Participation in the PIR-CLEF 2017 Pilot Task
The goal of the PIR-CLEF 2017 Pilot Task is to investigate the use of an alternative method to enable comparative laboratory-based evaluation of PIR.
For the PIR-CLEF 2017 Pilot Task we are making available user profile data and raw search data produced by guided search sessions undertaken by 10 volunteer users. The data provided include the submitted queries, baseline ranked lists of documents retrieved using a standard search system in response to each query, the items clicked by the user in response to this list, and document relevance information provided by the user on a 4-grade scale. Each session was performed by the users on a topic of their choosing, and search was carried out over a subset of the ClueWeb12 web collection.
The aim of the task is to use the available personal information to use the provided personal and prior search information to improve over a baseline ranking of documents judged relevant to the query by the user who entered the query.
Since this is a pilot activity we encourage participants to attempt the task using existing algorithms and to explore new ideas. We also welcome contributions examining details of the task and dataset.
Our objective is to use the outputs from the Pilot Task as the basis of one or more formal PIR-CLEF Lab tasks at CLEF 2018.
Participants in any aspect of the PIR-TASK are encouraged to submit a paper up to 6 pages in lenght describing their work for inclusion in the workshop proceedings as part of the CLEF 2017 Labs. The workshop is open to the submission of papers reporting on feedback on the pilot task. Further details of paper submission instructions and deadlines are available at submission page.
Participants are encouraged to attend the PIR-CLEF workshop to be held at CLEF 2017. This will form an important venue for reporting results and findings from the Pilot Task and providing feedback towards the specification of the PIR-CLEF task at CLEF 2018.
The Pilot Task data will be provided in csv format. Access to the indexed ClueWeb12 collection will be provided by the Dublin CIty University via API.
The links for ClueWeb API are:
http://clueweb.adaptcentre.ie/WebSearcher/search?query=”query String”&page=pagenumber