Both recognition and classification are common terms in computer vision. What is the difference? In classification, one assumes there is a given set of classes between which we must discriminate. For recognition, we assume there are some classes we can recognize in a much larger space of things we do not recognize. A motivating question for our work here is: What is the general object recognition problem? This question, of course, is a central theme in vision. How one should approach multi-class recognition is still an open issue. Should it be performed as a series of binary classifications, or by detection, where a search is performed for each of the possible classes? What happens when some classes are ill-sampled, not sampled at all or undefined?

The general term recognition suggests that the representation can handle different patterns often defined by discriminating features. It also suggests that the patterns to be recognized will be in general settings, visually mixed with many classes. For some problems, however, we do not need, and often cannot have, knowledge of the entire set of possible classes. For instance, in a recognition application for biologists, a single species of fish might be of interest. However, the classifier must consider the set of all other possible objects in relevant settings as potential negatives. Similarly, verification problems for security-oriented face matching constrain the target of interest to a single claimed identity, while considering the set of all other possible people as potential impostors. In addressing general object recognition, there is a finite set of known objects in myriad unknown objects, combinations and configurations - labeling something new, novel or unknown should always be a valid outcome. This leads to what is sometimes called "open set" recognition, in comparison to systems that make closed world assumptions or use "closed set" evaluation.

This work explores the nature of open set recognition, and formalizes its definition as a constrained minimization problem. The open set recognition problem is not well addressed by existing algorithms because it requires strong generalization. As steps towards a solution, we introduce the novel "1-vs-Set Machine", "W-SVM" and "EVM" learning formulations. The overall methodology applies to several different applications in computer vision where open set recognition is a challenging problem, including object recognition and face verification. Subsequent work has developed novel loss formulations that take into consideration the human perception of novelty, as well as a theoretical framework for studying novelty in visual recognition. Current work is developing more comprehensive open world learning formulations, which involve not just detecting novelty, but characterizing it and incorporating it back into a trained model. These latter explorations have included work on human activity recognition and handwritten document transcription.

This work was supported by ONR MURI Award No. N00014-08-1-0638, NSF IIS-1320956, FAPESP 2010/05647-4, Army SBIR W15P7T-12-C-A210, DARPA and the Army Research Office under contract HR001120C0055, and Microsoft