An underappreciated aspect of supervised machine learning is the scenario where classes that were not seen at training time appear at testing time. This is known as the open set recognition problem. It turns out that open set recognition is a problem that humans negotiate with ease (i.e., we have no problem identifying unknown objects as being new or novel instead of mistakenly grouping them with known classes), but machine learning algorithms struggle with. In this assignment, you will use the LIBSVM machine learning package to investigate a possible solution to open set recognition. LIBSVM can be built from source on multiple platforms, and it should compile just fine on the department's student machines. An Ubuntu virtual machine is also an option to get up and running.
For this assignment, record your responses to the following activities in the
README.md file in the
homework06 folder of your assignments GitLab
repository and push your work by 11:59 PM Wednesday, November 7. Any code you produce should be pushed to your repository as well.
To create a
homework06 branch in your local repository, follow the
$ cd path/to/cse-40171-fa18-assignments # Go to assignments repository $ git remote add upstream https://gitlab.com/wscheirer/cse-40171-fa18-assignments # Switch back over to the main class repository $ git fetch upstream # Toggle the upstream branch $ git pull upstream master # Pull the files for homework06 $ git checkout -b homework06 # Create homework06 branch and check it out $ cd homework06 # Go into homework06 folder
The first step is to download the LIBSVM source archive and build the package. For this assignment, use Version 3.23. After unzipping the downloaded file archive, enter the
libsvm-3.23 directory and type
make to build the software. If the build process is successful, you will see two binaries that we will use in this assignment:
The MNIST database of handwritten digits is a popular dataset used in computer vision as a basic sanity check for new algorithms. It consists of small images of the digits 0 through 9, where the goal of a learning system is to train models that can recognize new images of each digit based on a set of training examples. The dataset is considered solved, meaning even basic learning algorithms achieve near perfect performance on it these days. It is commonly used for testing new algorithms to make sure that they work. That is to say, if performance is poor on MNIST, the algorithm in question needs more work before it can be applied to state-of-the-art datasets and real-world problems. Your first task will be to generate a baseline accuracy result on this dataset using LIBSVM.
Download the training features file for MNIST. This file contains the pixel-level features of the dataset arranged in multi-dimensional feature vectors, each of which begins with a class label from "0" to "9". Using these features, train a multi-class RBF SVM model using the
svm-train program. You can leave the hyperparameters set to their defaults for this. Training may take some time, so be patient. Once it completes, you will have a model that can be used to make predictions about new handwritten digits that correspond to the classes that were seen at training time.
Download the testing features file for MNIST. This file is organized in a similar fashion to the training features file, but has new examples of digits to test the trained model with. Using the
svm-predict program, evaluate the model you just trained using the testing features. Record the accuracy given to you by
svm-predict in your
The baseline accuracy you just calculated reflects the closed set performance on the MNIST dataset. Now let's turn our attention to the open set recognition problem.
Download the training features file for an open set partition of MNIST that has been created for this assignment. This file is organized in a similar manner to the training features file from Activity 1, except that it only contains feature vectors for the digits 0 through 5. The remaining digits have been left out, but will be brought back at testing time, where they are used as "unknown" examples. Again using
svm-train, train a multi-class RBF SVM with the default hyperparameters using this more limited training data. Download the testing features file for the open set partition of MNIST. This file is identical to the testing features file from Activity 1, except that it labels the digits 6 through 9 with "-1", signifying that they are unknown. Use
svm-predict to establish an accuracy baseline for your new model. Record the accuracy given to you by
svm-predict in your
README.md. Note that if you have done this correctly, there should be a very large gap in performance between the closed set scenario in Activity 1 and this open set scenario.
The challenge you now face is improving the baseline performance on this open set task. Using LIBSVM, design a training and testing methodology for open set recognition that is able to achieve higher accuracy than the open set baseline you just calculated. Here is some information that you may find helpful for doing this:
svm-predictwill be more useful to you if you can have it output decision scores along with predicted labels.
README.md, including instructions to run it from a base installation of LIBSVM and considering the training and testing files from this assignment. Make sure to document the training and testing parameters you provided to the LIBSVM software. Turn in any code you have written along with your
If you have any questions, comments, or concerns regarding the course, please
provide your feedback at the end of your
To submit your assignment, please commit your R scripts and
README.md to the
homework06 branch in your assignment's GitLab repository:
$ cd path/to/cse-40171-fa18-assignments # Go to assignments repository $ git checkout master # Make sure we are in master branch $ git pull --rebase # Make sure we are up-to-date with GitLab $ git checkout -b homework06 # Create homework06 branch and check it out $ cd homework06 # Go to homework06 directory ... $ $EDITOR README.md # Edit appropriate README.md $ git add README.md # Mark changes for commit $ git commit -m "homework06: complete" # Record changes ... $ git push -u origin homework06 # Push branch to GitLab
Procedure for submitting your work: create a merge request by the process that is described here, but make sure to change the target branch from wscheirer/cse-40171-fa18-assignments to your personal fork's master branch so that your code is not visible to other students. Additionally, assign this merge request to your TA and add wscheirer, agraese, and AndroidKitKat as approvers (so all class staff can track your submission). Your assigned TA is agraese if you have a last name starting with A through Ki, or AndroidKitKat if you have a last name starting with Kl through W.