Perceptron

Created

May 14, 2025

Last Modified

2 weeks ago

Perceptron

Think of the Perceptron as a simple decision-maker—it looks at inputs, weighs their importance, and makes a yes-or-no judgment. It was one of the first steps toward building intelligent systems and still plays a crucial role in understanding how neural networks work today. By learning how to draw a boundary between different classes of data, the perceptron forms the foundation for more advanced models used in real-world applications.

📘 Definition:

A Perceptron is an artificial neuron in which the activation function is a threshold function.

Perceptron model diagram showing input features, weights, summation, and step activation function producing binary output

📌 Terminology:

$x_{1}, x_{2}, \dots, x_{n}$ = input signals
$w_{1}, w_{2}, \dots, w_{n}$ = associated weights
$x_{0}$ = Bias input (typically set to 1).
$w_{0}$ = Bias weight (adjusts activation threshold).
$\sum w_{i} x_{i}$ = Weighted sum of inputs.
$f (\sum w_{i} x_{i})$ = Activation function (determines output).
$y$ = Final output signal.

The neuron is called as perceptron if the output of the neuron is given by the following functions

O (x_{1}, x_{2}, .. x_{n})

- 1 i f w_{0} + w_{1} x_{1} + w_{n} x_{n} \leq 0

This is a step function — if the weighted sum exceeds the threshold, the neuron "fires" (outputs 1), else it outputs 0.

Perceptron Learning Algorithm

In the algorithm, we use the following notations

Symbol	Description
$n$	Number of input variables
$y = f (z)$	output for input vector $z$
$x_{j} = (x_{j 0}, x_{j 1}, ..., x_{j n})$	The `n`-dimensional input vector (j-th training sample)
$d_{j}$	Desired output for input $x_{j}$
$x_{j i}$	Value of `i-th` feature in `j-th` input
$x_{j 0}$	Bias input (always 1)
$w_{i}$	Weight for the `i-th` input variable
$w_{i} (l)$	Weight for the `i-th` input at `l-th` iteration

🔷 Algorithm Steps

🔹 Step 1: Initialization

Initialize the weights:

w_{0}, w_{1}, ..., w_{n}

Can be initialized to 0 or small random values.
Also, initialize a threshold (bias term).

🔹 Step 2: Training (for each training sample)

For each example $J$ in the training set $D$ .
perform the following step over the input $x_{j}$ and desired output $d_{j}$

Compute the output $y_{j} (t)$
This is the predicted output of the perceptron after processing input $x_{j}$ using the current weights at iteration $t$ . The perceptron computes this using:
$y_{j} (t) = i = 0 \sum n w_{i} (t) \cdot x_{j i}$
$y_{j} (t) = f (w_{0} (t) \cdot x_{j 0} + w_{1} (t) \cdot x_{j 1} + ... + w_{n} (t) \cdot x_{j n})$

Update weights for each $w_{i}$ using:

w i (t + 1) = w_{i} (t) + (d_{j} - y_{j} (t)) \cdot x_{j i}

for i = 0, 1, ..., n

🔹 Step 3: Repeat Until Convergence

Repeat Step 2 until either:
- The average error per iteration is less than a predefined threshold:
$\frac{1}{s} j = 1 \sum s ∣ d^{j} - y^{j} (t) ∣ < error threshold$
OR
- A maximum number of iterations is reached.