KNN Classification

KNN Classification / Classifier

Picture this: you’re at a crowded Indian wedding, trying to figure out which side of the family a guest belongs to. You don’t ask for their ID. you just look at who they’re sitting with. If they’re surrounded by the groom’s cousins cracking jokes, they’re probably part of the Barat. If they’re huddled with the aunts discussing the catering, they’re likely from the bride’s side. Judging someone by their immediate neighbors—is exactly how the K-Nearest Neighbors (KNN) algorithm works. It is perhaps the most "human" algorithm in Data Science, relying on the simple, intuition that similar things always stay close together.

Euclidean Distance

$Distance(AB)=\sqrt{(x_2-x_1{)}^2+(y_2-y_1{)}^2}$

Euclidean distance in K-dimension

$A(x_1,x_2,\cdots ,x_k)$

$B(y_1,y_2,\cdots ,y_k)$

$Distance(A,B)=\sqrt{(y_1-x_1{)}^2+(y_2-x_2{)}^2+\cdots +(y_k-x_k{)}^2}$

Setting K data points which are nearest to X where K is generally an odd number then checking the class of all those K data points which are in neighborhood of X and then use both of majority concept to decide the class of X this is called normal KNN classifier.

Let’s say we have the following data points:

Now we want to classify point X = (4, 4) using K = 5

📏 Step 1: Calculate Euclidean Distances from X(4,4)

🔍 Step 2: Choose the K (5) Nearest Neighbors

Sorted by distance:

1. G (1.0) 🟦

2. H (1.0) 🟦

3. B (2.24) 🟥

4. D (2.24) 🟥

5. I (2.24) 🟦

🧠 Step 3: Voting (Majority Class)

From 5 nearest neighbors:

• 🟥 Red → 2 (B, D)

• 🟦 Blue → 3 (G, H, I)

✅ Majority = Blue, so
👉 X(4,4) is classified as 🟦 Blue

🎯 Weighted KNN Classifier

Unlike the normal KNN where all neighbors have equal vote, in Weighted KNN, closer neighbors have more influence in deciding the class of the new point.

💡 Weight is usually inverse of the distance

$\left(e.g.w_i=\frac{1}{{\left(d_i\right)}^2}\right)$ — smaller distance → higher weight.

🧪 Example: Weighted KNN

👉 Classify point X = (4, 2) with K = 3

📏 Step 1: Compute Euclidean Distance to X(4,2)

Step 2: Pick K = 3 Nearest Neighbors

Sorted by distance:

1. B (1.0) 🟦

2. C (1.0) 🟥

3. D (1.41) 🟦

Step 3: Assign Weights

Step 4: Weighted Voting

• 🟦 Blue: 1.00 (B) + 0.5 (D) = 1.50

• 🟥 Red: 1.00 (C) = 1.00

✅ Weighted majority = Blue

👉 So, X(4,2) is classified as 🟦 Blue