Extreme Value Theory for Visual Recognition

Keywords: calibration, meta-recognition, score analysis, object recognition, face recognition, attributes
Spring 2008 - Present
teaser image


For any recognition system, maximizing the accuracy of recognition is a primary goal. In the case of general object recognition, we do not want an object of a class unknown to the system to be recognized as being part of a known class, nor do we want an object that should be recognized by the system to be rejected as being unknown. In the case of biometric recognition, the stakes are sometimes higher: we never want a misidentification in the case of a watch-list security or surveillance application. With these scenarios in mind, the ability to predict the performance of a recognition system on a per instance match basis is desirable for a number of important reasons, including automatic threshold selection for determining matches and non-matches, automatic algorithm selection for multi-algorithm fusion, and further data acquisition signaling — all ways we can improve the basic recognition accuracy.

Meta-recognition is inspired by the multidisciplinary field of meta-cognition study. In the most basic sense, meta-cognition is "knowing about knowing". For decades, psychologists and cognitive scientists have explored the notion that the human mind has knowledge of its own cognitive processes, and can use it to develop strategies to improve cognitive performance. For example, if a student notices that she has more trouble learning history than mathematics, she "knows" something about her learning ability and can take corrective action to improve her academic performance. In this sense, a meta-recognition system analyzes the score output of a recognition system, and if necessary, adjusts the recognition decisions or signals for a specific response action. In this work, we have developed a statistical theory of meta-recognition for post-recognition score analysis derived from the extreme value theory. This theory generalizes to all recognition systems producing distance or similarity scores over a gallery of known images. In conjunction with the theory of meta-recognition, we have developed a new statistical classifier based upon the Weibull distribution that produces accurate predictions of recognition success or failure on a per instance recognition basis.

Making a prediction isn't the only thing we can do with meta-recognition. Recognition problems in computer vision often benefit from a fusion of different algorithms and/or sensors, with score level fusion being among the most widely used fusion approaches. Choosing an appropriate score normalization technique before fusion is a fundamentally difficult problem because of the disparate nature of the underlying distributions of scores for different sources of data. Further complications are introduced when fusion inputs outright fail or have adversarial content, which we find in the fields of biometrics and forgery detection. Ideally, a score normalization should be robust to modeling assumptions, modeling errors, and parameter estimation errors, as well as algorithm failure. To address these issues, we have introduced the w-score, a new technique for robust recognition score normalization. We do not assume a match or non-match distribution, but instead suggest that the top scores of a recognition system's non-match scores follow the statistical Extreme Value Theory, and show how to use this observation to provide consistent robust normalization with a strong statistical basis.

This work was supported by ONR SBIR Award No. N00014-11-C-0243, ONR STTR Award No. N000-14-07-M-0421, ONR MURI Award No. N00014-08-1-0638, and DHS SBIR Award No. NBCHC080054


  • "Probability Models for Open Set Recognition,"
    Walter J. Scheirer, Lalit P. Jain, Terrance E. Boult,
    IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI),
    November 2014.
  • "Multi-Class Open Set Recognition Using Probability of Inclusion,"
    Lalit P. Jain, Walter J. Scheirer, Terrance E. Boult,
    Proceedings of the European Conference on Computer Vision (ECCV),
    September 2014.
  • "How Reliable are Your Visual Attributes?"
    Walter J. Scheirer, Neeraj Kumar, Vijay Iyer, Peter N. Belhumeur, Terrance E. Boult,
    Proceedings of the SPIE Defense, Security + Sensing Symposium
    May 2013.
  • "Learning for Meta-Recognition,"
    Walter J. Scheirer, Anderson Rocha, Jonathan Parris, Terrance E. Boult,
    IEEE Transactions on Information Forensics and Security (T-IFS),
    August 2012.
  • "Multi-Attribute Spaces: Calibration for Attribute Fusion and Similarity Search,"
    Walter J. Scheirer, Neeraj Kumar, Peter N. Belhumeur, Terrance E. Boult,
    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
    June 2012.
  • "Meta-Recognition: The Theory and Practice of Recognition Score Analysis,"
    Walter J. Scheirer, Anderson Rocha, Ross Micheals, Terrance E. Boult,
    IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI),
    August 2011.
  • "Robust Fusion: Extreme Value Theory for Recognition Score Normalization,"
    Walter J. Scheirer, Anderson Rocha, Ross Micheals, Terrance E. Boult,
    Proceedings of the 11th European Conference on Computer Vision (ECCV),
    September 2010.
  • "A Fusion Based Approach to Enhancing Multi-Modal Biometric Recognition System Failure and Overall Performance,"
    Walter J. Scheirer, Terrance E. Boult,
    Proceedings of the IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS),
    September 2008.
  • "Predicting Biometric Facial Recognition Failure With Similarity Surfaces and Support Vector Machines,"
    Walter J. Scheirer, Abhijit Bendale, Terrance E. Boult,
    Proceedings of the IEEE Computer Society Workshop on Biometrics,
    June 2008.


Attribute Calibration Software

Attribute Calibration Software:

C/C++/Python code to calibrate attribute classifier outputs using the w-score formulation, based on the Extreme Value Theory. For non-commercial uses only.


MugHunt Online Face Search Engine

MugHunt Online Face Search Engine:

An online face search engine built using meta-recognition fusion technology. This search engine is a joint collaboration with Automatic Face Systems, Inc. and Columbia University.