Polynomial Regression

Created

Mar 3, 2025

Last Modified

2 weeks ago

Polynomial Regression

Not all relationships in data are straight lines—and that’s where polynomial regression becomes powerful. When patterns start to curve, bend, or grow in unexpected ways, a simple linear model isn’t enough. Polynomial regression allows us to fit a smooth curve through the data, helping us capture real-world complexity more accurately and make better predictions.

A polynomial is a kind of expression:

$y = f (x) = a_{0} + a_{1} x + a_{2} x^{2} + ... + a_{n} x^{n}$

if a ≠ 0 then $f (x)$ is a polynomial in $x$ of degree $n$

$a_{0}, a_{1}, a_{2}, ... a_{n}$ all are constant.

fitting of this line on given data points we have n data points given.
$(x_{1}, y_{1}) (x_{2}, y_{2}) (x_{3}, y_{3}) ... (x_{n}, y_{n})$

Sometime data points are distributed in the space in such a way that it is not possible to fit a line on the data point H so in the figure observing the distribution pattern of the data points we feed suitable curve on the given data points polynomial regression.

$f (x) = a + b x + c x^{2}$
if c ≠ 0 second degree polynomial.

Error Calculation:

Square of Errors: $e_{i}^{2} = (y_{i} - \overset{y_{i}}{^})^{2}$
Sum of Square of Errors: $E = \sum_{i = 1}^{n} e_{i}^{2}$

Where: $\overset{y_{i}}{^} = a + b x_{i} + c x_{i}^{2}$

$E = \sum_{i = 1}^{n} e_{i}^{2}$
$E = \sum_{i = 1}^{n} (y_{i} - \overset{y_{i}}{^})^{2}$
$E = \sum_{i = 1}^{n} (y_{i} - (a + b x_{i} + c x_{i}^{2}))^{2}$
$E = \sum_{i = 1}^{n} (y_{i} - a - b x_{i} - c x_{i}^{2}))^{2}$

method of least sum of square of error to fit the polynomial.

Goal

The expression on the parameter $a, b, c$ to minimize $E$ we need to differentiate $E$ partially with respect to $a, b$ and $c$ and then equating them with zero.

Partial Derivative with respect to $α$ :

\frac{\partial E}{\partial α} = i = 1 \sum n \frac{\partial}{\partial α} (y_{i} - a - b x_{i} - c x_{i}^{2})^{2}

$\frac{\partial E}{\partial α} = \sum_{i = 1}^{n} \frac{\partial}{\partial α} (y_{i} - a - b x_{i} - c x_{i}^{2})^{2}$

$\frac{\partial E}{\partial α} = 2 \sum_{i = 1}^{n} (y_{i} - a - b x_{i} - c x_{i}^{2})^{2} \times \frac{\partial}{\partial α} (y_{i} - a - b x_{i} - c x_{i}^{2})$

$\frac{\partial E}{\partial α} = 2 \sum_{i = 1}^{n} (y_{i} - a - b x_{i} - c x_{i}^{2})^{2} \times - 1$

$\frac{\partial E}{\partial α} = - 2 \sum_{i = 1}^{n} (y_{i} - a - b x_{i} - c x_{i}^{2})^{2}$

Equating $\frac{\partial E}{\partial α}$ with 0

$\frac{\partial E}{\partial α} = 0$

$- 2 \sum_{i = 1}^{n} (y_{i} - a - b x_{i} - c x_{i}^{2})^{2} = 0$

$\sum_{i = 1}^{n} (y_{i} - a - b x_{i} - c x_{i}^{2})^{2} = 0$

$\sum_{i = 1}^{n} y_{i} - a \sum_{i = 1}^{n} .1 - b \sum_{i = 1}^{n} x_{i} - c \sum_{i = 1}^{n} x_{i}^{2})^{2} = 0$

Divide both sides by $n$ :

$\frac{1}{n} \sum_{i = 1}^{n} y_{i} - \frac{1}{n} a \sum_{i = 1}^{n} .1 - \frac{1}{n} b \sum_{i = 1}^{n} x_{i} - \frac{1}{n} c \sum_{i = 1}^{n} x_{i}^{2})^{2} = 0$

$\overline{y} - a - b \overline{x} - c \overline{(x^{2})} = 0$

Partial Derivative with respect to $b$ :

\frac{\partial E}{\partial b} = i = 1 \sum n \frac{\partial}{\partial α} (y_{i} - a - b x_{i} - c x_{i}^{2})^{2}

$\frac{\partial E}{\partial b} = \sum_{i = 1}^{n} \frac{\partial}{\partial α} (y_{i} - a - b x_{i} - c x_{i}^{2})^{2}$

$\frac{\partial E}{\partial b} = 2 \sum_{i = 1}^{n} (y_{i} - a - b x_{i} - c x_{i}^{2}) \times \frac{\partial}{\partial b} (y_{i} - a - b x_{i} - c x_{i}^{2})$

$\frac{\partial E}{\partial b} = 2 \sum_{i = 1}^{n} (y_{i} - a - b x_{i} - c x_{i}^{2}) \times (- x_{i})$

$\frac{\partial E}{\partial b} = - 2 \sum_{i = 1}^{n} x_{i} (y_{i} - a - b x_{i} - c x_{i}^{2})$

Equating $\frac{\partial E}{\partial b}$ with 0

$- 2 \sum_{i = 1}^{n} x_{i} (y_{i} - a - b x_{i} - c x_{i}^{2}) = 0$

$\sum_{i = 1}^{n} x_{i} (y_{i} - a - b x_{i} - c x_{i}^{2}) = 0$

$\sum_{i = 1}^{n} x_{i} y_{i} - a \sum_{i = 1}^{n} x_{i} - b \sum_{i = 1}^{n} x_{i}^{2} - c \sum_{i = 1}^{n} x_{i}^{3} = 0$

Divide both sides by $n$ :

$\frac{1}{n} \sum_{i = 1}^{n} x_{i} y_{i} - \frac{a}{n} \sum_{i = 1}^{n} x_{i} - \frac{b}{n} \sum_{i = 1}^{n} x_{i}^{2} - \frac{c}{n} \sum_{i = 1}^{n} x_{i}^{3} = 0$

$(\overline{x y}) - a \overline{x} - b \overline{x^{2}} - c \overline{x^{3}} = 0$

Partial Derivative with respect to $c$ :

\frac{\partial E}{\partial c} = i = 1 \sum n \frac{\partial}{\partial c} (y_{i} - a - b x_{i} - c x_{i}^{2})^{2}

$\frac{\partial E}{\partial c} = \sum_{i = 1}^{n} \frac{\partial}{\partial c} (y_{i} - a - b x_{i} - c x_{i}^{2})^{2}$

$\frac{\partial E}{\partial c} = 2 \sum_{i = 1}^{n} (y_{i} - a - b x_{i} - c x_{i}^{2}) \times \frac{\partial}{\partial c} (y_{i} - a - b x_{i} - c x_{i}^{2})$

$\frac{\partial E}{\partial c} = 2 \sum_{i = 1}^{n} (y_{i} - a - b x_{i} - c x_{i}^{2}) \times (- x_{i}^{2})$

$\frac{\partial E}{\partial c} = - 2 \sum_{i = 1}^{n} x_{i}^{2} (y_{i} - a - b x_{i} - c x_{i}^{2})$

Equating $\frac{\partial E}{\partial c}$ with 0

$- 2 \sum_{i = 1}^{n} x_{i}^{2} (y_{i} - a - b x_{i} - c x_{i}^{2}) = 0$

$\sum_{i = 1}^{n} x_{i}^{2} (y_{i} - a - b x_{i} - c x_{i}^{2}) = 0$

$\sum_{i = 1}^{n} x_{i}^{2} y_{i} - a \sum_{i = 1}^{n} x_{i}^{2} - b \sum_{i = 1}^{n} x_{i}^{3} - c \sum_{i = 1}^{n} x_{i}^{4} = 0$

Divide both sides by $n$ :

$\frac{1}{n} \sum_{i = 1}^{n} x_{i}^{2} y_{i} - \frac{a}{n} \sum_{i = 1}^{n} x_{i}^{2} - \frac{b}{n} \sum_{i = 1}^{n} x_{i}^{3} - \frac{c}{n} \sum_{i = 1}^{n} x_{i}^{4} = 0$

$(\overline{x^{2} y}) - a \overline{x^{2}} - b \overline{x^{3}} - c \overline{x^{4}} = 0$

\overset{y_{i}}{^} = a + b x_{i} + c x_{i}^{2}

\overline{y} - a - b \overline{x} - c \overline{x^{2}} = 0

(\overline{x y}) - a \overline{x} - b \overline{x^{2}} - c \overline{x^{3}} = 0

(\overline{x^{2} y}) - a \overline{x^{2}} - b \overline{x^{3}} - c \overline{x^{4}} = 0

Polynomial Regression

Polynomial Regression

Goal

Partial Derivative with respect to α:

Partial Derivative with respect to b:

Partial Derivative with respect to c:

Partial Derivative with respect to $α$ :

Partial Derivative with respect to $b$ :

Partial Derivative with respect to $c$ :