Robustness and reliability evaluations of image annotation

Abstract

The semantic gap problem in image retrieval has motivated much work focusing on automatic image annotation, aimed at facilitating computers to automatically assign keywords to images. The basic measure for evaluating the annotation performance is usually to examine the annotation accuracy. To do this, the fraction of the relevant images, which have been correctly classified by a specific classifier or image annotation system, is measured. Consequently, the evaluation result can be thought of as a surrogate for the judgment of real users. However, the ability of this kind of quantitative evaluation measure to fully evaluate the performance and value of image annotation systems is limited. This paper introduces two complementary metrics related to the rates of annotation accuracy, which can help to further assess the robustness and stability of image annotation systems. They are: (i) the number of annotated keywords with zero-rate accuracy and (ii) the coefficient of variation of annotation accuracy. The evaluation results based on three datasets show that these two metrics are very useful to make a more reliable conclusion for image annotation systems.

Keywords:

• Annotation accuracy
• Coefficient of variation
• Image annotation
• Performance evaluation

Notes

* http://archive.ics.uci.edu/ml/databases/CorelFeatures/CorelFeatures.html.

^†The Weka software which is available at: http://www.cs.waikato.ac.nz/ml/weka/ was used to develop the SVM and k-NN classifiers.

^‡ http://www.vision.caltech.edu/Image_Datasets/Caltech101/.

^§ http://www.vision.caltech.edu/Image_Datasets/Caltech256/.

^¶ http://www.cs.ubc.ca/~lowe/keypoints/.