Extreme Value Theory for Visual Recognition

Keywords: calibration, meta-recognition, score analysis, object recognition, face recognition, attributes
Spring 2008 - Present
teaser image

Description

For any recognition system, maximizing the accuracy of recognition is a primary goal. In the case of general object recognition, we do not want an object of a class unknown to the system to be recognized as being part of a known class, nor do we want an object that should be recognized by the system to be rejected as being unknown. In the case of biometric recognition, the stakes are sometimes higher: we never want a misidentification in the case of a watch-list security or surveillance application. With these scenarios in mind, the ability to predict the performance of a recognition system on a per instance match basis is desirable for a number of important reasons, including automatic threshold selection for determining matches and non-matches, automatic algorithm selection for multi-algorithm fusion, and further data acquisition signaling — all ways we can improve the basic recognition accuracy.

Meta-recognition is inspired by the multidisciplinary field of meta-cognition study. In the most basic sense, meta-cognition is "knowing about knowing". For decades, psychologists and cognitive scientists have explored the notion that the human mind has knowledge of its own cognitive processes, and can use it to develop strategies to improve cognitive performance. For example, if a student notices that she has more trouble learning history than mathematics, she "knows" something about her learning ability and can take corrective action to improve her academic performance. In this sense, a meta-recognition system analyzes the score output of a recognition system, and if necessary, adjusts the recognition decisions or signals for a specific response action. In this work, we have developed a statistical theory of meta-recognition for post-recognition score analysis derived from the extreme value theory. This theory generalizes to all recognition systems producing distance or similarity scores over a gallery of known images. In conjunction with the theory of meta-recognition, we have developed a new statistical classifier based upon the Weibull distribution that produces accurate predictions of recognition success or failure on a per instance recognition basis.

Making a prediction isn't the only thing we can do with meta-recognition. Recognition problems in computer vision often benefit from a fusion of different algorithms and/or sensors, with score level fusion being among the most widely used fusion approaches. Choosing an appropriate score normalization technique before fusion is a fundamentally difficult problem because of the disparate nature of the underlying distributions of scores for different sources of data. Further complications are introduced when fusion inputs outright fail or have adversarial content, which we find in the fields of biometrics and forgery detection. Ideally, a score normalization should be robust to modeling assumptions, modeling errors, and parameter estimation errors, as well as algorithm failure. To address these issues, we have introduced the w-score, a new technique for robust recognition score normalization. We do not assume a match or non-match distribution, but instead suggest that the top scores of a recognition system's non-match scores follow the statistical Extreme Value Theory, and show how to use this observation to provide consistent robust normalization with a strong statistical basis.

This work was supported by ONR SBIR Award No. N00014-11-C-0243, ONR STTR Award No. N000-14-07-M-0421, ONR MURI Award No. N00014-08-1-0638, and DHS SBIR Award No. NBCHC080054

Publications

  • "The Extreme Value Machine,"
    Ethan Rudd, Lalit P. Jain, Walter J. Scheirer, Terrance Boult,
    IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI),
    March 2018.
    [pdf] [supp. material] [code]
    [bibtex]
    @article{Rudd_2018_TPAMI,
    author = {Ethan Rudd and Lalit P. Jain and Walter J. Scheirer and Terrance Boult},
    title = {The Extreme Value Machine},
    journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI)},
    volume = {40},
    number = {3},
    month = {March},
    year = {2018}
    }
  • "Extreme Value Theory-Based Methods for Visual Recognition,"
    Walter J. Scheirer,
    Morgan & Claypool Publishers,
    February 2017.
    [buy]
    [bibtex]
    @book{Scheirer_2017_MC,
    author = {Walter J. Scheirer},
    title = {Extreme Value Theory-Based Methods for Visual Recognition},
    publisher = {Morgan & Claypool Publishers},
    month = {February},
    year = {2017}
    }
  • "Probability Models for Open Set Recognition,"
    Walter J. Scheirer, Lalit P. Jain, Terrance E. Boult,
    IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI),
    November 2014.
    [pdf] [supp. material] [errata]
    [bibtex]
    @article{Scheirer_2014_TPAMIb,
    author = {Walter J. Scheirer and Lalit P. Jain and Terrance E. Boult},
    title = {Probability Models for Open Set Recognition},
    journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI)},
    volume = {36},
    issue = {11},
    month = {November},
    year = {2014}
    }
  • "Multi-Class Open Set Recognition Using Probability of Inclusion,"
    Lalit P. Jain, Walter J. Scheirer, Terrance E. Boult,
    Proceedings of the European Conference on Computer Vision (ECCV),
    September 2014.
    [pdf] [supp. material]
    [bibtex]
    @InProceedings{Jain_2014_ECCV,
    author = {Lalit P. Jain and Walter J. Scheirer and Terrance E. Boult},
    title = {Multi-Class Open Set Recognition Using Probability of Inclusion},
    booktitle = {The European Conference on Computer Vision (ECCV)},
    month = {September},
    year = {2014}
    }
  • "How Reliable are Your Visual Attributes?"
    Walter J. Scheirer, Neeraj Kumar, Vijay Iyer, Peter N. Belhumeur, Terrance E. Boult,
    Proceedings of the SPIE Defense, Security + Sensing Symposium
    May 2013.
    [pdf]
    [bibtex]
    @InProceedings{Scheirer_2013_SPIE,
    author = {Walter J. Scheirer and Neeraj Kumar and Vijay Iyer and Peter N. Belhumeur and Terrance E. Boult},
    title = {How Reliable are Your Visual Attributes?},
    booktitle = {The SPIE Defense, Security + Sensing Symposium},
    month = {May},
    year = {2013}
    }
  • "Learning for Meta-Recognition,"
    Walter J. Scheirer, Anderson Rocha, Jonathan Parris, Terrance E. Boult,
    IEEE Transactions on Information Forensics and Security (T-IFS),
    August 2012.
    [pdf]
    [bibtex]
    @article{Scheirer_2012_TIFS,
    author = {Walter J. Scheirer and Anderson Rocha and Jonathan Parris and Terrance E. Boult},
    title = {Learning for Meta-Recognition},
    journal = {IEEE Transactions on Information Forensics and Security (TIFS)},
    volume = {7},
    issues = {4},
    month = {August},
    year = {2012}
    }
  • "Multi-Attribute Spaces: Calibration for Attribute Fusion and Similarity Search,"
    Walter J. Scheirer, Neeraj Kumar, Peter N. Belhumeur, Terrance E. Boult,
    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
    June 2012.
    [pdf] [code]
    [bibtex]
    @InProceedings{Scheirer_2012_CVPR,
    author = {Walter J. Scheirer and Neeraj Kumar and Peter N. Belhumeur and Terrance E. Boult},
    title = {Multi-Attribute Spaces: Calibration for Attribute Fusion and Similarity Search},
    booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    month = {June},
    year = {2012}
    }
  • "Meta-Recognition: The Theory and Practice of Recognition Score Analysis,"
    Walter J. Scheirer, Anderson Rocha, Ross Micheals, Terrance E. Boult,
    IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI),
    August 2011.
    [pdf] [errata] [code]
    [bibtex]
    @article{Scheirer_2011_TPAMI,
    author = {Walter J. Scheirer and Anderson Rocha and Ross Michaels and Terrance E. Boult},
    title = {Meta-Recognition: The Theory and Practice of Recognition Score Analysis},
    journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI)},
    volume = {33},
    issue = {8},
    pages = {1689--1695},
    year = {2011}
    }
  • "Robust Fusion: Extreme Value Theory for Recognition Score Normalization,"
    Walter J. Scheirer, Anderson Rocha, Ross Micheals, Terrance E. Boult,
    Proceedings of the 11th European Conference on Computer Vision (ECCV),
    September 2010.
    [pdf] [code]
    [bibtex]
    @InProceedings{Scheirer_2011_ECCV,
    author = {Walter J. Scheirer and Anderson Rocha and Ross Michaels and Terrance E. Boult},
    title = {Robust Fusion: Extreme Value Theory for Recognition Score Normalization},
    booktitle = {The 11th European Conference on Computer Vision (ECCV)},
    month = {September},
    year = {2010}
    }
  • "A Fusion Based Approach to Enhancing Multi-Modal Biometric Recognition System Failure and Overall Performance,"
    Walter J. Scheirer, Terrance E. Boult,
    Proceedings of the IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS),
    September 2008.
    [pdf]
    [bibtex]
    @InProceedings{Scheirer_2008_BTAS,
    author = {Walter J. Scheirer and Terrance E. Boult},
    title = {A Fusion Based Approach to Enhancing Multi-Modal Biometric Recognition System Failure and Overall Performance},
    booktitle = {The IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS)},
    month = {September},
    year = {2008}
    }
  • "Predicting Biometric Facial Recognition Failure With Similarity Surfaces and Support Vector Machines,"
    Walter J. Scheirer, Abhijit Bendale, Terrance E. Boult,
    Proceedings of the IEEE Computer Society Workshop on Biometrics,
    June 2008.
    [pdf]
    [bibtex]
    @InProceedings{Scheirer_2008_ICSWB,
    author = {Walter J. Scheirer and Abhijit Bendale and Terrance E. Boult},
    title = {Predicting Biometric Facial Recognition Failure With Similarity Surfaces and Support Vector Machines},
    booktitle = {The IEEE Computer Society Workshop on Biometrics},
    month = {June},
    year = {2008}
    }