C-TEST: CSIRO Toolkit for the Evaluation of Search Technology
David Hawking and Tom Rowlands
version 1.0.1; 5 August 2009C-TEST implements a simple but generalised test-collection framework for evaluating the effectiveness of search systems. It is designed primarily for the evaluation of web and enterprise web search but is more generally applicable. It comprises:
- two XML file formats specifying:
- queries, interpretations, weighted answers
- results of a `run'
- a collection of tools to evaluate based on various metrics, sample from existing files, pool results for evaluation and use a web based tool for evaluation
- a library, written in Perl, to work with the files from your own tools
It is capable of appropriately evaluating the Homepage Finding, Named Page Finding and Topic Distillation tasks from the TREC Web Track. The framework effectively penalises the return of duplicate results and has a prominent recognition of the one query text having many potential, variably-weighted, interpretations.
C-TEST is distributed free under the Mozilla Public Licence. We genuinely welcome your suggestions, criticism and patches at c-test at wirrapoi dot com. We do, however, ask that if you use C-TEST in the development of an academic publication, you cite our C-TEST paper (BibTeX):
David Hawking, Tom Rowlands, Paul Thomas. C-TEST: Supporting novelty and diversity in testfiles for search evaluation. Proceedings of the SIGIR workshop on redundancy, diversity and interdependent document relevance, Boston 2009.
The manual is automatically produced by the code and is also available here:
Requirements and installation
C-TEST is tested on Ubuntu and Mac OS X. It should work without fault on Windows XP systems with Perl installed. It almost certainly runs without modification on other Unix-like systems. Packages that are required are Perl, LibXML::XML, Statistics::Distributions and File::Slurp. Directions for installing the latter two are included in the download.
Download
DTDs
XML DTDs are available for checking the conformance of files to the two standard C-TEST formats.