An underappreciated aspect of supervised machine learning is the scenario where classes that were not seen at training time appear at testing time. This is known as the open set recognition problem. It turns out that open set recognition is a problem that humans negotiate with ease (i.e., we have no problem identifying unknown objects as being new or novel instead of mistakenly grouping them with known classes), but machine learning algorithms struggle with. In this assignment, you will use the LIBSVM machine learning package to investigate a possible solution to open set recognition. LIBSVM can be built from source on multiple platforms, and it should compile just fine on the department's student machines. An Ubuntu virtual machine is also an option to get up and running.

For this assignment, record your responses to the following activities in the README.md file in the homework06 folder of your assignments GitLab repository and push your work by 11:59 PM Wednesday, November 7. Any code you produce should be pushed to your repository as well.

Activity 0: Branching

As discussed in class, each homework assignment must be completed in its own git branch; this will allow you to separate the work of each assignment and for you to use the merge request workflow.

To create a homework06 branch in your local repository, follow the instructions below:

$ cd path/to/cse-40171-fa18-assignments                                                     # Go to assignments repository

$ git remote add upstream https://gitlab.com/wscheirer/cse-40171-fa18-assignments           # Switch back over to the main class repository

$ git fetch upstream                                                                        # Toggle the upstream branch

$ git pull upstream master                                                                  # Pull the files for homework06

$ git checkout -b homework06                                                                # Create homework06 branch and check it out

$ cd homework06                                                                             # Go into homework06 folder

Once these commands have been successfully performed, you are now ready to add, commit, and push any work required for this assignment.

Activity 1: Train and Test a Supervised SVM Model for the MNIST Dataset (25 Points)

The first step is to download the LIBSVM source archive and build the package. For this assignment, use Version 3.23. After unzipping the downloaded file archive, enter the libsvm-3.23 directory and type make to build the software. If the build process is successful, you will see two binaries that we will use in this assignment: svm-train and svm-predict.





The MNIST database of handwritten digits is a popular dataset used in computer vision as a basic sanity check for new algorithms. It consists of small images of the digits 0 through 9, where the goal of a learning system is to train models that can recognize new images of each digit based on a set of training examples. The dataset is considered solved, meaning even basic learning algorithms achieve near perfect performance on it these days. It is commonly used for testing new algorithms to make sure that they work. That is to say, if performance is poor on MNIST, the algorithm in question needs more work before it can be applied to state-of-the-art datasets and real-world problems. Your first task will be to generate a baseline accuracy result on this dataset using LIBSVM.

Download the training features file for MNIST. This file contains the pixel-level features of the dataset arranged in multi-dimensional feature vectors, each of which begins with a class label from "0" to "9". Using these features, train a multi-class RBF SVM model using the svm-train program. You can leave the hyperparameters set to their defaults for this. Training may take some time, so be patient. Once it completes, you will have a model that can be used to make predictions about new handwritten digits that correspond to the classes that were seen at training time.

Download the testing features file for MNIST. This file is organized in a similar fashion to the training features file, but has new examples of digits to test the trained model with. Using the svm-predict program, evaluate the model you just trained using the testing features. Record the accuracy given to you by svm-predict in your README.md.

The baseline accuracy you just calculated reflects the closed set performance on the MNIST dataset. Now let's turn our attention to the open set recognition problem.

Activity 2: Design an SVM Training and Testing Methodology for Open Set Recognition Using MNIST (75 Points)

Download the training features file for an open set partition of MNIST that has been created for this assignment. This file is organized in a similar manner to the training features file from Activity 1, except that it only contains feature vectors for the digits 0 through 5. The remaining digits have been left out, but will be brought back at testing time, where they are used as "unknown" examples. Again using svm-train, train a multi-class RBF SVM with the default hyperparameters using this more limited training data. Download the testing features file for the open set partition of MNIST. This file is identical to the testing features file from Activity 1, except that it labels the digits 6 through 9 with "-1", signifying that they are unknown. Use svm-predict to establish an accuracy baseline for your new model. Record the accuracy given to you by svm-predict in your README.md. Note that if you have done this correctly, there should be a very large gap in performance between the closed set scenario in Activity 1 and this open set scenario.

The challenge you now face is improving the baseline performance on this open set task. Using LIBSVM, design a training and testing methodology for open set recognition that is able to achieve higher accuracy than the open set baseline you just calculated. Here is some information that you may find helpful for doing this:

After applying your solution, what is the new accuracy? Add a brief description of your solution to your README.md, including instructions to run it from a base installation of LIBSVM and considering the training and testing files from this assignment. Make sure to document the training and testing parameters you provided to the LIBSVM software. Turn in any code you have written along with your README.md file.

Feedback

If you have any questions, comments, or concerns regarding the course, please provide your feedback at the end of your README.md.

Submission

To submit your assignment, please commit your R scripts and README.md to the homework06 folder of your homework06 branch in your assignment's GitLab repository:

$ cd path/to/cse-40171-fa18-assignments   # Go to assignments repository
$ git checkout master                     # Make sure we are in master branch
$ git pull --rebase                       # Make sure we are up-to-date with GitLab
$ git checkout -b homework06              # Create homework06 branch and check it out
$ cd homework06                           # Go to homework06 directory
...
$ $EDITOR README.md                       # Edit appropriate README.md
$ git add README.md                       # Mark changes for commit
$ git commit -m "homework06: complete"    # Record changes
...
$ git push -u origin homework06           # Push branch to GitLab

Procedure for submitting your work: create a merge request by the process that is described here, but make sure to change the target branch from wscheirer/cse-40171-fa18-assignments to your personal fork's master branch so that your code is not visible to other students. Additionally, assign this merge request to your TA and add wscheirer, agraese, and AndroidKitKat as approvers (so all class staff can track your submission). Your assigned TA is agraese if you have a last name starting with A through Ki, or AndroidKitKat if you have a last name starting with Kl through W.