SVM

Support Vector Machine (SVM)

Support Vector Machine (SVM) is one of the most powerful supervised machine learning algorithms used for classification and regression tasks. The core idea behind SVM is simple yet elegant: it tries to find the best possible boundary that separates data points of different classes with maximum margin.

At the heart of SVM lies the concept of a hyperplane, which acts as a decision boundary in the feature space. Depending on the number of dimensions, this boundary takes different forms. In two dimensions, it is a straight line, while in three dimensions, it becomes a plane. In higher dimensions, it generalizes into what we call a hyperplane.

Equation of line

$\overline{w}.\overline{x}+b=0$
$\overline{w}=(w_1,w_2,w_3,....w_n)$
$\overline{x}=(x_1,x_2,x_3,....x_n)$

🔹 Support Vectors

The data points that are closest to the hyperplane.
These points directly influence the position and orientation of the hyperplane.

🔹 Hyperplane

A decision boundary that separates different classes in the feature space.
in $2D:ax+by+c=0$
in $3D:ax+by+cz+d=0$

in $nD:w_1{x}_1+w_2{x}_2+\cdots +w_n{x}_n+b=0$
In higher dimensions → It's a hyperplane

This generalization allows SVM to work effectively even in high-dimensional datasets, which is one of its biggest strengths in real-world machine learning applications.

🔹 Equation of a Hyperplane (Line)

$w_1{x}_1+w_2{x}_2+\cdots +w_n{x}_n+b=0$

$\overline{w}.\overline{x}+b=0$

$\overline{w}=\left(w_1,w_2,\cdots w_n\right)$

$\overline{x}=\left(x_1,x_2,\cdots x_n\right)$

Projection of Vectors

📝 From the triangle OLB:

So,

🔷 Using Dot Product Formula:

So scalar projection becomes:

$cos(\theta )=\frac{OL}{OB}$

$cos(\theta )=\frac{OL}{b}$

$OL=bcos\theta$

$\vec{a}\cdot \vec{b}=|\vec{a}||\vec{b}|\cos (\theta )$

$\vec{a}\cdot \vec{b}=a\cdot b\cos (\theta )$

$bcos(\theta )=\frac{\vec{a}\cdot \vec{b}}{|\vec{a}|}$