To build a visual recognition system for any application these days, one's first inclination is to turn to the most recent machine learning breakthrough from the area of deep learning, which no doubt has been enabled by access to millions of training images from the Internet. But there are many circumstances where such an approach cannot be used as an off-the-shelf component to assemble the system we desire, because even the largest training dataset does not take into account all of the artifacts that can be experienced in the environment. As computer vision pushes further into real-world applications, what should a software system that can interpret images from sensors placed in any unrestrictedsetting actually look like?

First, it must incorporate a set of algorithms, drawn from the areas of computational photography and machine learning, into a processing pipeline that corrects and subsequently classifies images across time and space. Image restoration and enhancement algorithms that remove corruptions like blur, noise, and mis-focus, or manipulate images to gain resolution, change perspective, and compensate for lens distortion are now commonplace in photo editing tools. Such operations are necessary to improve the quality of raw images that are otherwise unacceptable for recognition purposes. But they must be compatible with the recognition process itself, and not adversely affect feature extraction or classification.

Remarkably, little thought has been given to image restoration and enhancement algorithms for visual recognition - the goal of computational photography thus far has simply been to make images look appealing after correction. It remains unknown what impact many transformations have on visual recognition algorithms. To begin to answer that question, exploratory work is needed to find out which image pre-processing algorithms, in combination with the strongest features and supervised machine learning approaches, are promising candidates for different problem domains.

This work was supported by IARPA contract #2016-16070500002 and the NVIDIA Corporation