This sketch was an idea that came about from following an Image Classification tutorial from Daniel Shiffman. In this video he shows you how to make a sketch that uses MobileNet, a lightweight model trained on a subset of the ImageNet dataset.

Because MobileNet can only classify images into 1000 pre-defined categories, most things you put in front of the camera will be wrong. Interacting with the sketch feels a bit like being in a room of people shouting out bad guesses in a game of charades. Which is admittedly, quite fun. But I also found it interesting because of what it can teach us about the relationship between data and classification results.

Thinking About Data

You can look at (some of) the original training data here, and start to make guesses as to what qualities the images from your camera feed might share with the images in the dataset. i.e. in what ways is my screaming rubber chicken like a balance beam? Or more accurately—out of 1000 everyday things, why would a balance beam the best possible match to my screaming rubber chicken?

Thinking about Confidence

The raw confidence score that you get from the model is a big long number between 0 and 1. It’s not very human readable.

This made me think about the human ways we communicate confidence. In speech, we tend to signal a lack of confidence in our claims by finishing our sentences with an upward inflection? To signal others that we might need a little validation?? I think that question marks are a playful way to visualise confidence (or lack thereof), and this project made me think of different possibilities for communicating machine confidence in more human ways.

You can play with the sketch here.