Linear Regression

Abhijeet Singh Rajput

@abhijeetsingh

Created

Feb 25, 2025

Last Modified

2 weeks ago

Regression

linear regression
polynomial regression
logistic regression

Linear regression is a method in machine learning where we fit a straight line to a set of data points that appear to be concentrated around the line. This model helps to predict the outcome for future values.

Equation of line: $\overset{y}{^} = α + β x$

where $α$ is scaler used to transform the line.
and $β$ (coefficient of x) is slope of the line

Linear regression graph showing best fit line with actual data points, predicted values, slope (beta), and y-intercept

$x_{i}$ (Input)	$x_{1}$	$x_{2}$	$x_{3}$	.	.	$x_{n}$
$y_{i}$ (Outcome)	$y_{1}$	$y_{2}$	$y_{3}$	.	.	$y_{n}$

In the figure, n data points are plotted. We are trying to fit the line $Y = α + β X$ to this set of discrete data points.

Given Data:
- $y_{i}$ (Actual Data): The value of the output variable $y$ where $x = x_{i}$ , as provided by the data set.
Line Equation:
- $\overset{y_{i}}{^}$ (Predicted Data): The value of the outcome $y_{i}$ as predicted by the line $Y = α + β X$ , corresponding to $X = x_{i}$ .
Error Calculation:
- Error: The difference between $y_{i}$ (actual data point) and $\overset{y_{i}}{^}$ (predicted value from the line).
- Square of Errors: $e_{i}^{2} = (y_{i} - \overset{y_{i}}{^})^{2}$
- Sum of Square of Errors: $E = Σ_{i = 1}^{n} e_{i}^{2}$

Where: $\overset{y_{i}}{^} = α + β x_{i}$

$E = Σ_{i = 1}^{n} e_{i}^{2}$
$E = Σ_{i = 1}^{n} (y_{i} - \overset{y_{i}}{^})^{2}$
$E = Σ_{i = 1}^{n} {y_{i} - (α + β x_{i})}^{2}$

Goal

Now, we are trying to find out those value of $α$ and $β$ for which $E$ is minimum.

Partial Derivative with respect to $α$ :

using the power rule:
$\frac{\partial}{\partial x} (x^{n}) = n x^{n - 1}$

Differentiate with respect to $α$

$\frac{\partial E}{\partial α} = \frac{\partial}{\partial α} Σ_{i = 1}^{n} {y_{i} - (α + β)}^{2}$
Apply the Chain Rule:

$\frac{\partial E}{\partial α} = \sum_{i = 1}^{n} 2 (y_{i} - (α + β x_{i})) \cdot \frac{\partial}{\partial α} (y_{i} - (α + β x_{i}))$
Differentiate the Inner Term:

$\frac{\partial}{\partial α} (y_{i} - (α + β x_{i}))$

$\frac{\partial}{\partial α} (y_{i}) - \frac{\partial}{\partial α} (α) - \frac{\partial}{\partial α} (β x_{i})$

$0 - 1 - 0$

$- 1$
Put the inner term:

$\frac{\partial E}{\partial α} = \sum_{i = 1}^{n} 2 {y_{i} - (α + β x_{i})} . - 1$

$\frac{\partial E}{\partial α} = \sum_{i = 1}^{n} - 2 {y_{i} - (α + β x_{i})}$

$\frac{\partial E}{\partial α} = - 2 \sum_{i = 1}^{n} {y_{i} - (α + β x_{i})}$

$\frac{\partial E}{\partial α} = - 2 \sum_{i = 1}^{n} (y_{i} - α - β x_{i})$

$\frac{\partial E}{\partial α} = - 2 \sum_{i = 1}^{n} (y_{i} - α - β x_{i}) = 0$
Simplify:

$\sum_{i = 1}^{n} (y_{i} - α - β x_{i}) = 0$

$\sum_{i = 1}^{n} y_{i} - \sum_{i = 1}^{n} α - β \sum_{i = 1}^{n} x_{i} = 0$
$\sum_{i = 1}^{n} y_{i} = n α + β \sum_{i = 1}^{n} x_{i}$

Divide both sides by $n$ :

$\frac{1}{n} \sum_{i = 1}^{n} y_{i} = \frac{1}{n} (n α + β \sum_{i = 1}^{n} x_{i})$

$\frac{1}{n} \sum_{i = 1}^{n} y_{i} = α + β \frac{1}{n} \sum_{i = 1}^{n} x_{i}$

$\overset{y}{ˉ} = α + β \overset{x}{ˉ}$

E q u a t i o n^{1} : \overset{y}{ˉ} = α + β \overset{x}{ˉ}

Partial Derivative with respect to $β$ :

Differentiate the Error Function with respect to $β$

$\frac{\partial E}{\partial β} = \frac{\partial}{\partial β} \sum_{i = 1}^{n} (y_{i} - (α + β x_{i}))^{2}$
Apply the Chain Rule:

$\frac{\partial E}{\partial β} = \sum_{i = 1}^{n} 2 (y_{i} - (α + β x_{i})) \cdot \frac{\partial}{\partial β} (y_{i} - (α + β x_{i}))$
Differentiate the Inner Term:

$\frac{\partial}{\partial β} (y_{i} - (α + β x_{i}))$

$\frac{\partial}{\partial β} (y_{i}) - \frac{\partial}{\partial β} (α) - \frac{\partial}{\partial β} (β x_{i})$

$0 - 0 - x_{i}$

$- x_{i}$
Put the inner term:

$\frac{\partial E}{\partial β} = \sum_{i = 1}^{n} 2 (y_{i} - (α + β x_{i})) (- x_{i})$

$\frac{\partial E}{\partial β} = - 2 \sum_{i = 1}^{n} x_{i} (y_{i} - (α + β x_{i}))$

$\frac{\partial E}{\partial β} = - 2 \sum_{i = 1}^{n} x_{i} (y_{i} - α - β x_{i}) = 0$
Simplify:

$\sum_{i = 1}^{n} x_{i} (y_{i} - α - β x_{i}) = 0$

$\sum_{i = 1}^{n} x_{i} y_{i} - α \sum_{i = 1}^{n} x_{i} - β \sum_{i = 1}^{n} x_{i}^{2} = 0$

$\sum_{i = 1}^{n} x_{i} y_{i} = α \sum_{i = 1}^{n} x_{i} + β \sum_{i = 1}^{n} x_{i}^{2}$

Divide both sides by $n$ :

$\frac{1}{n} \sum_{i = 1}^{n} x_{i} y_{i} = α \frac{1}{n} \sum_{i = 1}^{n} x_{i} + β \frac{1}{n} \sum_{i = 1}^{n} x_{i}^{2}$

$\overline{(x y)} = α \overline{x} + β \overline{(x^{2})}$

E q u a t i o n^{2} : \overline{(x y)} = α \overline{x} + β \overline{(x^{2})}

Equation 1: $\overset{y}{ˉ} = α + β \overset{x}{ˉ}$

Equation 2: $\overline{(x y)} = α \overline{x} + β \overline{(x^{2})}$

Multiply Equation 1 by $\overline{x}$ :

$\overline{y} \cdot \overline{x} = α \overline{x} + β \overline{x}^{2}$

Subtracting from Equation 2:

${\overline{(x y)} = α \overline{x} + β \overline{(x^{2})}} - {\overline{y} \cdot \overline{x} = α \overline{x} + β \overline{x}^{2}}$

$\overline{(x y)} - \overline{y} . \overline{x} = β \overline{(x^{2})} - β \overline{x}^{2}$

$\overline{(x y)} - \overline{x} . \overline{y} = β (\overline{(x^{2})} - \overline{x}^{2})$

$β = \frac{( x y ) - x . y}{( ( x ^{2} ) - x ^{2} )}$

E q u a t i o n^{3} : β = \frac{( x y ) - x . y}{( ( x ^{2} ) - x ^{2} )}

Variance

$V a r (x) = \frac{1}{n} \sum_{i = 1}^{n} (x_{i} - \overset{x}{ˉ})^{2}$

$V a r (x) = \frac{1}{n} {\sum_{i = 1}^{n} (x_{i}^{2} + \overset{x}{ˉ}^{2} - 2 x_{i} \overset{x}{ˉ})}$

$V a r (x) = \frac{1}{n} {\sum_{i = 1}^{n} x_{i}^{2} + \overset{x}{ˉ}^{2} \sum_{i = 1}^{n} .1 - 2 \overset{x}{ˉ} \sum_{i = 1}^{n} x_{i}}$

$V a r (x) = \frac{1}{n} {\sum_{i = 1}^{n} x_{i}^{2} + \overset{x}{ˉ}^{2} . n - 2 \overset{x}{ˉ} \sum_{i = 1}^{n} x_{i}}$

$V a r (x) = \frac{1}{n} \sum_{i = 1}^{n} x_{i}^{2} + \overset{x}{ˉ}^{2} - 2 \overset{x}{ˉ} . \overset{x}{ˉ}$

$V a r (x) = \overline{(x^{2})} + \overset{x}{ˉ}^{2} - 2 \overset{x}{ˉ}^{2}$

$V a r (x) = \overline{(x^{2})} - \overset{x}{ˉ}^{2}$

E q u a t i o n^{4} : V a r (x) = \overline{(x^{2})} - \overset{x}{ˉ}^{2}

Co-variance

co v (x, y) = \frac{1}{n} i = 1 \sum n (x_{i} - \overline{x}) (y_{i} - \overline{y})

co v (x, y) = \frac{1}{n} i = 1 \sum n (x_{i} y_{i} - x_{i} \overline{y} - \overline{x} y_{i} + \overline{x} . \overline{y})

co v (x, y) = \frac{1}{n} (i = 1 \sum n x_{i} y_{i} - i = 1 \sum n x_{i} \overline{y} - i = 1 \sum n \overline{x} y_{i} + i = 1 \sum n \overline{x} . \overline{y}))

co v (x, y) = \frac{1}{n} (i = 1 \sum n x_{i} y_{i} - \overline{y} i = 1 \sum n x_{i} - \overline{x} i = 1 \sum n y_{i} + \overline{x} . \overline{y} i = 1 \sum n .1)

co v (x, y) = \overline{(x y)} - \overline{y} . \overline{x} - \overline{x} . \overline{y} + \overline{x} . \overline{y}

co v (x, y) = \overline{(x y)} - \overline{x} . \overline{y}

E q u a t i o n^{5} : co v (x, y) = \overline{(x y)} - \overline{x} . \overline{y}

E q u a t i o n^{1} : \overset{y}{ˉ} = α + β \overset{x}{ˉ}

E q u a t i o n^{2} : \overline{(x y)} = α \overline{x} + β \overline{(x^{2})}

E q u a t i o n^{3} : β = \frac{( x y ) - x . y}{( ( x ^{2} ) - x ^{2} )}

E q u a t i o n^{4} : V a r (x) = \overline{(x^{2})} - \overset{x}{ˉ}^{2}

E q u a t i o n^{5} : co v (x, y) = \overline{(x y)} - \overline{x} . \overline{y}

β = \frac{( x y ) - y . y}{( ( x ^{2} ) - x ^{2} )}

β = \frac{co v ( x , y )}{v a r ( x )}

Put the value of $β$ in $e q^{1}$ :

$\overline{y} = α + β \overline{x}$

$α = \overline{y} - β \overline{x}$

α = \overline{y} - \frac{co v ( x , y )}{v a r ( x )} \overline{x}

α = \overline{y} - \frac{co v ( x , y )}{( x ^{2} ) - x ^{2}} \times \overline{x}

α = \overline{y} - \frac{co v ( x , y )}{( x ^{2} ) - x}

Final Value

β = \frac{co v ( x , y )}{v a r ( x )}

α = \overline{y} - β \overline{x}