Artificial Neural Network

Brain and Perceptrons:

Biological learning systems are built from a very complex web of interconnected neurons.
The human brain has an immensely connected network of approximately 100 billion $(10^{11})$ neurons, each connected on average to 1,000 $(10^4)$ other neurons.
Even though the neuron transmission speeds are much slower than computer processing speeds, the brain’s massive parallelism makes it extremely powerful for learning and decision-making tasks.

Artificial Neurons:

An artificial neuron is a mathematical function conceived as a model of a biological neuron.
Artificial neurons are the elementary units of an artificial neural network (ANN).
This artificial neuron receives one or more inputs and sums them to produce an output.
Each input is individually weighted, and the sum is passed through a function known as the activation function or transfer function.

📌 Terminology:

• $x$ : Input signal

• $w$ : Weights associated with inputs

• $x_0$ : Input signal taking a constant value of 1

• $w_0$ : Weight associated with x₀ (called bias)

• $\sum wx$ : Summation of input signals

• $f\left(\sum wx\right)$ : Activation function (produces the output)

• $y$ : Output signal

⚡ Activation Function

In artificial neural networks, the activation function takes the incoming signals as input and produces the output signal. It determines whether a neuron should be activated or not by calculating a weighted sum and applying a transformation.

🧠 Common Activation Functions:

1. Threshold activation function

The threshold activation function is defined as

2. Unit step function

Some time threshold activation function is also defined as unite step function in which case it is called unit step activation functions.

Defined as

3. Sigmoid activation function (logistic function)

one of the most commonly used activation functions is the sigmoidal activation function.
it is defined as:

4. Linear activation function

The Linear activation function is defined by

5. Piecewise saturated linear activation function

It is defined as:

$f(x)=0\text{if x > }{x}_{max}$$

6. Gaussian activation function

This is defined as

Explanation:

• $\sigma$ is the standard deviation.

• $\pi$ is the mathematical constant Pi.

• $e^{-\frac{x^2}{2{\sigma }^2}}$ ensures the Gaussian shape.

This function is commonly used in radial basis function networks and probabilistic models.

7. Hyperbolic tangent (tanh) activation function

This function is defined as