The Perceptron / Neuron

I. Background

We've had the opportunity to discuss with some of the "big ideas" in machine learning: supervised learning, classification models, features and labels in training data.

We've also used a pre-configured, web-based, machine-learning library, Tensorflow, to train a data set and test it.

Machine-learning libraries are wonderfully powerful, and can be complex to configure. We'll be in a better position to be able to do that if we have a solid understanding of the foundations of machine-learning: the Perceptron/Neuron and the Neural Network.

II. The Perceptron

A. The simplest perceptron

The perceptron model was proposed by Rosenblatt in 1958. In its simplest form, a series of binary (0, 1) input values x_i are added together and compared with some threshold value. If the sum of the input values is greater than the threshold, you get one result, and if the sum of values is less than the threshold, you get a different result.

Let's say we want to have the perceptron identify whether or not we should study this evening. We'll have x₁ be 1 (true) if we're reasonably awake and 0 (false) if we're exhausted. Also, x₂ is true if we have a test tomorrow, or false if we don't. Finally, x₃ is true if we need this class for college admission, and false if we don't.

Our threshold for studying will be the value 2—if the sum of our inputs is 2 or greater, we'll study.

Under what conditions will the perceptron suggest we should study?

B. Improvements to the perceptron

We're very quickly going to iterate on this model.

1. We're going to add a weight for each of the input values, a multiplier that we can use to give more or less importance to each of our inputs.
2. Rather than try to achieve some threshold value, we're going to add a bias to our result, which will adjust the sum of our input-weight results by some amount.
3. Finally, we're going to formally define an activation function that will give us more information about our perceptron's calculation.

Our improved perceptron can be drawn like this:

III. The Neuron

The neuron is similar to the perceptron but it has two significant improvements that will be critical to creating a machine-learning program:

Inputs can be non-binary, and the output can be non-binary.

With this brief introduction, take a look at the Perceptron-Neuron Worksheet.

Homework

Watch the video from 3 Blue, 1 Brown, But What is a Neural Network? [YouTube, 18 min]