Walter J. Scheirer

walter.scheirer@nd.edu, @wjscheirer

Dennis O. Doughty Collegiate Associate Professor in the Department of Computer Science and Engineering,
University of Notre Dame

Research Overview

My research is primarily focused around the problem of recognition, including the representations and algorithms supporting solutions to it. I am particularly interested in features and learning-based methods that apply to both vision and language, thus breaking away from the persistent compartmentalization of recognition tasks (something hinted at by David Marr over 30 years ago). This has led to some intriguing and often unconventional approaches that can be applied to a broad set of areas within artificial intelligence including computer vision, machine learning, and human biometrics. Specifically, my work is looking at open set recognition, extreme value theory models for visual recognition, and biologically-inspired learning algorithms.

Curriculum Vitae

PDF Format

Postdoctoral Fellows

Megan Levis
Zachary Miksis (Co-Advised with Jonathan Hauenstein)

Graduate Students

Samuel Grieggs

Jin Huang

Jeffery Kinnison

Derek Prijatelj

Arturo Russell (Co-Advised with Jane Cleland-Huang)

Abigail Swenor

Rosaura Vidal Mata (Co-Advised with Kevin Bowyer)

William Theisen

Teaching

Spring 2023: CSE 20176 The Archeology of Hacking

Fall 2022: CSE 40175 Ethical & Professional Issues

Spring 2022: CSE 40175 Ethical & Professional Issues

Fall 2021: CSE 40175 Ethical & Professional Issues

Spring 2020: CSE 40567/60567 Computer Security

Fall 2019: CSE 40171 Artificial Intelligence

Spring 2019: CSE 40567/60567 Computer Security

Fall 2018: CSE 40171 Artificial Intelligence

Spring 2018: CSE 40567/44567/60567 Computer Security

Spring 2017: CSE 40567/60567 Computer Security

Spring 2016: CSE 40567/60567 Computer Security

Fall 2015: CSE 40537/60537 Biometrics

Books

A History of Fake Things on the Internet (Stanford University Press, Coming Soon)

Quantitative Intertextuality: Analyzing the Markers of Information Reuse (Springer-Nature) with Christopher W. Forstall

Extreme Value Theory-based Methods for Visual Recognition (Morgan & Claypool Publishers)

Recent Publications (see all)

"Unfooling Perturbation-Based Post Hoc Explainers,"
Zachariah Carmichael, Walter J. Scheirer,
Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 2023),

February 2023.

[pdf]
[bibtex]
@inproceedings{Carmichael2023,
url = {https://arxiv.org/abs/2205.14772},
author = {Carmichael, Zachariah and
Scheirer, Walter},
title = {Unfooling Perturbation-Based Post Hoc Explainers},
booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence
(AAAI 2023)},
year = {2023}
}
"Motif Mining: Finding and Summarizing Remixed Image Content,"
William Theisen, Daniel Gonzalez, Zachariah Carmichael, Daniel Moreira,
Tim Weninger, Walter Scheirer,
Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision
(WACV),

January 2023.

[pdf]
[bibtex]
@inproceedings{Theisen_2023,
author = {Theisen, William and
Cedre, Daniel Gonzalez and
Carmichael, Zachariah and
Moreira, Daniel and
Weninger, Tim and
Scheirer, Walter},
title = {Motif Mining: Finding and Summarizing Remixed Image Content},
publisher = {arXiv},
booktitle = {Proceedings of the IEEE/CVF Winter Conference on
Applications of Computer Vision (WACV)},
year = {2023},
}
"SILA: A System For Scientific Image Analysis,"
Daniel Moreira, Joao P. Cardenuto, Ruiting Shao, Sriram Baireddy, Davide Cozzolino,
Diego Gragnaniello, Wael Abd-Almageed, Paolo Bestagini, Stefano Tubaro,
Anderson Rocha, Walter J. Scheirer, Luisa Verdoliva, Edward J. Delp,
Scientific Reports,

October 2022.

[pdf]
[bibtex]
@article{moreira2022sila,
title={SILA: a system for scientific image analysis},
author={Moreira, Daniel and
Cardenuto, Jo{\~a}o Phillipe and
Shao, Ruiting and
Baireddy, Sriram and
Cozzolino, Davide and
Gragnaniello, Diego and
Abd-Almageed, Wael and
Bestagini, Paolo and
Tubaro, Stefano and
Rocha, Anderson and
Scheirer, Walter and
Verdoliva, Luisa and
Delp, Edward},
journal={Scientific Reports},
volume={12},
number={1},
pages={1--15},
year={2022},
publisher={Nature Publishing Group}
}
"The Tesserae Intertext Service,"
Nozomu Okuda, Jeffery Kinnison, Patrick Burns, Neil Coffee, Walter J. Scheirer,
Digital Humanities Quarterly,

April 2022.

[pdf]
[bibtex]
@article{Okuda_DHQ2022,
author = {Nozomu Okuda and
Jeffery Kinnison and
Patrick Burns and
Neil Coffee and
Walter J. Scheirer},
title = {The Tesserae Intertext Service},
journal = {Digital Humanities Quarterly},
volume = {16},
number = {1},
month = {April},
year = {2022}
}
"Bridging the Gap Between Computational Photography and Visual
Recognition,"
Rosaura G. VidalMata, Sreya Banerjee, Brandon RichardWebster, Michael Albright,
Pedro Davalos, Scott McCloskey, Ben Miller, Asong Tambo, Sushobhan Ghosh,
Sudarshan Nagesh, Ye Yuan, Yueyu Hu, Junru Wu, Wenhan Yang, Xiaoshuai Zhang,
Jiaying Liu, Zhangyang Wang, Hwann-Tzong Chen, Tzu-Wei Huang, Wen-Chi Chin,
Yi-Chun Li, Mahmoud Lababidi, Charles Otto, Walter J. Scheirer,
IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI),

December 2021.

[pdf]
[bibtex]
@article{vidalmata2021bridging,
title = {Bridging the Gap Between Computational Photography and Visual
Recognition},
author = {Rosaura G. VidalMata and
Sreya Banerjee and
Brandon RichardWebster and
Michael Albright and
Pedro Davalos and
Scott McCloskey and
Ben Miller and
Asong Tambo and
Sushobhan Ghosh and
Sudarshan Nagesh and
Ye Yuan and
Yueyu Hu and
Junru Wu and
Wenhan Yang and
Xiaoshuai Zhang and
Jiaying Liu and
Zhangyang Wang and
Hwann-Tzong Chen and
Tzu-Wei Huang and
Wen-Chi Chin and
Yi-Chun Li and
Mahmoud Lababidi and
Charles Otto and
Walter J. Scheirer},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence
(T-PAMI)},
volume = {43},
number = {12},
month = {December},
year = {2021}
}
"A Study of the Human Perception of Synthetic Faces,"
Bingyu Shen, Brandon RichardWebster, Alice O'Toole, Kevin Bowyer,
Walter J. Scheirer,
Proceedings of the IEEE Conference on Automatic Face and Gesture Recognition (FG),

December 2021.

[pdf]
[bibtex]
@inproceedings{Shen_FG2021,
author = {Bingyu Shen and
Brandon RichardWebster and
Alice O'Toole and
Kevin Bowyer and
Walter J. Scheirer},
title = {A Study of the Human Perception of Synthetic Faces},
booktitle = {Proceedings of the IEEE Conference on Automatic Face
and Gesture Recognition (FG)},
year = {2021}
}
"Handwriting Recognition with Novelty,"
Derek Prijatelj, Samuel Grieggs, Futoshi Yumoto, Eric Robertson, Walter J. Scheirer,
International Conference on Document Analysis and Recognition (ICDAR),

September 2021.

[pdf]
[bibtex]
@inproceedings{Prijatelj_ICDAR2021,
author = {Derek Prijatelj and
Samuel Grieggs and
Futoshi Yumoto and
Eric Robertson and
Walter J. Scheirer},
title = {Handwriting Recognition with Novelty},
booktitle = {International Conference on Document Analysis and Recognition
(ICDAR},
year = {2021}
}
"Measuring Human Perception to Improve Handwritten Document Transcription,"
Samuel Grieggs, Bingyu Shen, Greta Rauch, Pei Li, Jiaqi Ma, David Chiang,
Brian Price, Walter J. Scheirer,
IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI),

Accepted for Publication in June 2021.

[pdf]
[bibtex]
@article{DBLP:journals/corr/abs-1904-03734,
author = {Samuel Grieggs and
Bingyu Shen and
Greta Rauch and
Pei Li and
Jiaqi Ma and
David Chiang and
Brian Price and
Walter J. Scheirer},
title = {Measuring Human Perception to Improve Handwritten Document
Transcription},
journal = {CoRR},
volume = {abs/1904.03734},
year = {2020},
url = {http://arxiv.org/abs/1904.03734},
archivePrefix = {arXiv},
eprint = {1904.03734},
}
"Automatic Discovery of Political Meme Genres with Diverse Appearances,"
William Theisen, Joel Brogan, Pamela Bilo Thomas, Daniel Moreira, Pascal Phoa,
Tim Weninger, Walter J. Scheirer,
Proceedings of the International AAAI Conference on Web and Social Media (ICWSM),

June 2021.

[pdf] [code] [data]
[bibtex]
@inproceedings{theisen2021automatic,
title={Automatic Discovery of Political Meme Genres with Diverse Appearances},
author={William Theisen and Joel Brogan and Pamela Bilo Thomas and
Daniel Moreira and Pascal Phoa and Tim Weninger and Walter Scheirer},
booktitle={International AAAI Conference on Web and Social Media (ICWSM)}
year={2021},
}
"Towards a Unifying Framework for Formal Theories of Novelty,"
Terrance Boult, Przemyslaw Grabowicz, Derek Prijatelj, Roni Stern, Lawrence Holder,
Joshua Alspector, Mohsen Jafarzadeh, Touqeer Ahmad, Akshay Dhamija, Chunchun Li,
Steve Cruz, Abhinav Shrivastava, Carl Vondrick, Walter J. Scheirer,
AAAI Conference on Artificial Intelligence (AAAI 2021), Senior Member Track,

February 2021.

[pdf]
[bibtex]
@inproceedings{Boult21,
author = {Terrance Boult and
Przemyslaw Grabowicz and
Derek Prijatelj and
Lawrence Holder and
Joshua Alspector and
Mohsen Jafarzadeh and
Touqeer Ahmad and
Akshay Dhamija and
Chunchun Li and
Steve Cruz and
Abhinav Shrivastava and
Carl Vondrick and
Walter J. Scheirer},
title = {Towards a Unifying Framework for Formal Theories of Novelty},
booktitle = {AAAI Conference on Artificial Intelligence (AAAI 2021)},
year = {2021},
}

New Research in arXiv / bioRxiv

"Human Activity Recognition in an Open World,"
Derek Prijatelj, Samuel Grieggs, Jin Huang, Dawei Du, Ameya Shringi, Christopher Funk,
Adam Kaufman, Eric Robertson, Walter Scheirer,
December 2022.

[pdf]
[bibtex]
@misc{https://doi.org/10.48550/arxiv.2212.12141,
author = {Prijatelj, Derek S. and
Grieggs, Samuel and
Huang, Jin andi
Du, Dawei and
Shringi, Ameya and
Funk, Christopher and
Kaufman, Adam and
Robertson, Eric and
Scheirer, Walter J.},
title = {Human Activity Recognition in an Open World},
publisher = {arXiv},
year = {2022}
}
"Using Human Perception to Regularize Transfer Learning,"
Justin Dulay, Walter Scheirer,
November 2022.

[pdf]
[bibtex]
@misc{Dulay2022b,
doi = {10.48550/ARXIV.2211.07885},
url = {https://arxiv.org/abs/2211.07885},
author = {Dulay, Justin and Scheirer, Walter J.},
title = {Using Human Perception to Regularize Transfer Learning},
publisher = {arXiv},
year = {2022},
copyright = {Creative Commons Attribution Share Alike 4.0 International}
}
"Measuring Human Perception to Improve Open Set Recognition,"
Jin Huang, Derek Prijatelj, Justin Dulay, Walter Scheirer,
September 2022.

[pdf]
[bibtex]
@misc{Huang2022,
doi = {10.48550/ARXIV.2209.03519},
url = {https://arxiv.org/abs/2209.03519},
author = {Huang, Jin and
Prijatelj, Derek and
Dulay, Justin and
Scheirer, Walter},
title = {Measuring Human Perception to Improve Open Set Recognition},
publisher = {arXiv},
year = {2022},
copyright = {Creative Commons Attribution Share Alike 4.0 International}
}
"Guiding Machine Perception with Psychophysics,"
Justin Dulay, Sonia Poltoratski, Till Hartmann, Samuel Anthony,
Walter Scheirer,
July 2022.

[pdf]
[bibtex]
@misc{Dulay2022,
url = {https://arxiv.org/abs/2207.02241},
author = {Dulay, Justin and
Poltoratski, Sonia and
Hartmann, Till S. and
Anthony, Samuel E.
and Scheirer, Walter J.},
title = {Guiding Machine Perception with Psychophysics},
publisher = {arXiv},
year = {2022}
}
"On the Objective Evaluation of Post Hoc Explainers,"
Zachariah Carmichael, Walter J. Scheirer,
June 2021.

[pdf]
[bibtex]
@misc{carmichael2021objective,
title={On the Objective Evaluation of Post Hoc Explainers},
author={Zachariah Carmichael and Walter J. Scheirer},
year={2021},
eprint={2106.08376},
archivePrefix={arXiv},
primaryClass={cs.LG}
}

Selected Projects (see all)

Practical Technology Ethics

Fall 2019 - Present

Constructive criticism of emerging technologies, scientific trends, and policies

Keywords: machine learning, facial analysis, election integrity, scientific integrity, peace studies

Human-Drone Partnerships for Emergency Response

Fall 2019 - Present

Semi-autonomous UAVs to support emergency response scenarios, such as fire surveillance and search and rescue

Keywords: drones, software engineering, participatory design, object detection, first responders

Unconventional Approaches to Hacking Neural Networks

Spring 2018 - Present

Uncovering new vulnerabilities in neural network-based applications where security is a consideration

Keywords: machine learning, neural networks, face recognition, security, backdoors, exploits

Hardware-Aware Distributed Hyperparameter Optimization

Spring 2017 - Present

Intelligent mapping of hardware resources to model search tasks

Keywords: machine learning, hyperparameters, optimization, distributed computing, connectomics

Computational Models of Zebrafish Sensory Integration

Fall 2016 - Fall 2020

Modeling neural circuits to explain fish behavior and apply to generalized recognition problems in AI

Keywords: computational neuroscience, zebrafish, sensory integration, data fusion, extreme value theory

Image Restoration and Enhancement for Visual Recognition

Fall 2016 - Present

Work to bridge the gap between computational photography and visual recognition

Keywords: deblurring, denoising, super-resolution, deep learning, object recognition, video

Face Synthesis

Fall 2015 - Present

Fast and accurate photo-quality face synthesis for use in face recognition work

Keywords: face synthesis, face recognition, face analysis, computer graphics, data augmentation

Learning-free Image Segmentation

Fall 2015 - Spring 2019

Image segmentation for settings where machine learning-based approaches are slow and do not generalize

Keywords: image filtering, volumetric segmentation, connectomics, medical imaging, iris recognition

Tools for Neuroscience

Fall 2015 - Fall 2020

Techniques to help neuroscientists understand neuroanatomy and function

Keywords: connectomics, electron microscopy, two-photon imaging, x-ray tomography, behavior

Brain-informed Machine Learning

Fall 2014 - Spring 2019

Applying measurements of neural activity to regularize machine learning models

Keywords: fMRI, psychophysics, classification, deep learning, neural-weights, object recognition

Visual Place Recognition

Spring 2013 - Fall 2019

Condition invariant visual place recognition for robot navigation inspired by vision science

Keywords: robotics, navigation, place recognition, whole image matching, patch matching, calibration

Good Recognition is Non-metric

Fall 2012 - Spring 2014

A new look at a fundamental question in computer vision: is recognition metric?

Keywords: machine learning, metric learning, recognition, face recognition, object recognition

Psychophysics for Computer Vision

Fall 2012 - Present

Measuring exemplar-by-exemplar difﬁculty and the pattern of errors of humans for supervised learning

Keywords: machine learning, psychology, citizen science, psychophysics, face detection, attributes

Open Set Recognition

Spring 2011 - Present

Theory and algorithms that address the difficult problem of training without complete class knowledge

Keywords: machine learning, object recognition, face recognition, support vector machines, 1-vs-set machine

Extreme Value Theory for Visual Recognition

Spring 2008 - Present

The theory and practice of recognition score analysis for prediction and fusion

Keywords: calibration, meta-recognition, score analysis, object recognition, face recognition, attributes

Language & Literature

Spring 2009 - Present

Machine learning and related statistical methods to improve the process by which intertextuality is studied

Keywords: computational linguistics, intertextuality, sound, stylistics, classics, authorship attribution

Digital Image and Video Forensics

Spring 2008 - Present

Approaches to steganalysis, forgery detection and sensor fingerprinting

Keywords: forensics, forgery detection, data hiding, source identification, sensor fingerprinting

Selected Talks (see all)

Invited talk at the CVPR 2022 Workshop Open World Vision: A Unifying Framework for Formal Theories of Novelty in Visual Perception
Invited talk at the WACV 2022 Workshop xAI4Biometrics: Visual Psychophysics for Making Face Recognition Algorithms More Explainable
Invited talk at the Computational Research Integrity Conference (CRI-CONF 21): Understanding the Provenance of Visual Disinformation Targeting Science
Invited talk at the IEEE International Symposium on Technology and Society Conference (ISTAS 20): The 'Criminality from Face' Illusion
Invited talk at the WACV 2020 Workshop Neural Architecture Search for Computer Vision in the Wild: Representational Dissimilarity Analysis as a Tool for Neural Network Model Search
Talk at the 2019 Beijing Forum Panel on Humanity and Legal Governance in the Digital Era: New Perspectives on Digital Media Integrity
Keynote talk at the 18th International Conference of the Biometrics Special Interest Group (BIOSIG 2019): The Limits and Potentials of Deep Learning for Facial Analysis
Talk at the DEF CON 27 AI Village: Backdooring Convolutional Neural Networks with Targeted Weight Perturbations
Invited talk at the CVPR 2018 Workshop "Real-World Challenges and New Benchmarks for Deep Learning in Robotic Vision":
A Psychophysics Driven Evaluation Framework for Visual Recognition
Invited talk at Cultural Analytics 2017: Quantitative Intertextuality: Analyzing the Markers of Information Reuse
Invited talk at Argonne National Laboratory: Scalable Strategies for Image Analysis in Neuroscience
CVPR 2016 Tutorial: Statistical Methods for Open Set Recognition
Invited talk at the RSS 2016 Workshop "Are the Sceptics Right? Limits and Potentials of Deep Learning in Robotics":
The Impact of the Open Set Recognition Problem on Deep Learning (Video)

Activities

WACV 2023 and FG 2023 will be co-located as a hybrid meeting online and in Waikoloa, HI, January 3rd - 8th
CVPR 2022 was held as a hybrid meeting online and in New Orleans, June 19th - 24th
WACV 2022 was held as a hybrid meeting online and in Waikoloa, HI, January 4th - 8th (yes we actually made it to Hawaii)
The UG^2+ Challenge prize challenge workshop on image restoration and enhancement was held at CVPR 2019
The Interactive and Adaptive Learning in an Open World workshop was co-located with ECCV 2018
The UG^2 Challenge prize challenge workshop on image restoration and enhancement was held at CVPR 2018
Join the PAMI-TC (now on LinkedIn) and the Computer Vision Foundation