> For the complete documentation index, see [llms.txt](https://alham-rizvi.gitbook.io/alhamrizvi/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://alham-rizvi.gitbook.io/alhamrizvi/ai-college-notes/practical-3-decision-tree-classifier.md).

# Practical 3 — Decision Tree Classifier

**Language used:** Python 3 with the **scikit-learn** (`sklearn`) machine learning library. Run this practical in Google Colab.

**Aim:** Build a Decision Tree Classifier that predicts whether a customer will **Buy** or **Not Buy** a product, based on their **Age** and **Gender**.

***

### 3.1 Before the code: what IS a Decision Tree? 

A decision tree is basically a **flowchart of yes/no questions** that ends in a decision. You already build these in your head all the time:

> "Is it raining?" → Yes → "Take umbrella" | No → "Don't take umbrella"

A Decision Tree Classifier in ML does the same thing, except **the computer figures out which questions to ask, and in what order**, by looking at training data.

```mermaid
flowchart TD
    A[Root: Is Age <= 31.5?] -->|Yes| B[Don't Buy]
    A -->|No| C[Buy]
```

#### How does the computer decide *which question* to ask first?

It uses a measure of "messiness" in the data called **Entropy**.

* **Entropy = 0** → the group is perfectly "pure" (everyone in it has the same label, e.g. all "Buy").
* **Entropy = 1** (max, for 2 classes) → the group is a perfect 50/50 mix — completely uncertain.

The formula (for 2 classes, Buy `p` and Don't Buy `1-p`):

```
Entropy = - p*log2(p) - (1-p)*log2(1-p)
```

The tree tries every possible question (e.g. "Age ≤ 20?", "Age ≤ 21?", "Age ≤ 22?"... "Gender == 0?") and picks the one that gives the **biggest drop in entropy** after splitting, this drop is called **Information Gain**. That's exactly why our code says:

```python
model = DecisionTreeClassifier(criterion="entropy")
```

We're explicitly telling scikit-learn: *"use entropy/information gain to decide your splits"* (the other common option is `"gini"`, a slightly different but similar messiness measure).

#### Key vocabulary

| Term               | Meaning                                                             |
| ------------------ | ------------------------------------------------------------------- |
| **Root node**      | The very first question at the top of the tree                      |
| **Leaf node**      | An end point of the tree — a final decision, no more questions      |
| **Feature**        | An input column used to ask questions (here: `Age`, `Gender`)       |
| **Label / Target** | What we're trying to predict (here: `Buy` = 1, `Don't Buy` = 0)     |
| **Training data**  | Examples with known answers, used to build/"grow" the tree          |
| **Test data**      | Examples kept aside to check if the tree actually learned correctly |

***

### 3.2 The full code

```python
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Sample dataset
X = [
    [22, 0],
    [25, 1],
    [47, 1],
    [52, 0],
    [46, 1],
    [56, 0],
    [23, 1],
    [40, 0]
]

# Target labels
# 1 = Buy, 0 = Don't Buy
y = [0, 0, 1, 1, 1, 1, 0, 1]

# Split the dataset
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42
)

# Create Decision Tree model
model = DecisionTreeClassifier(criterion="entropy")

# Train the model
model.fit(X_train, y_train)

# Predict
y_pred = model.predict(X_test)

# Accuracy
print("Accuracy:", accuracy_score(y_test, y_pred))

# Predict a new sample
new_data = [[30, 1]]
prediction = model.predict(new_data)

if prediction[0] == 1:
    print("Prediction: Buy")
else:
    print("Prediction: Don't Buy")
```

***

### 3.3 Line-by-line explanation

#### Imports

```python
from sklearn.tree import DecisionTreeClassifier
```

Brings in the actual Decision Tree algorithm, pre-built inside scikit-learn — you don't have to code entropy calculations yourself.

```python
from sklearn.model_selection import train_test_split
```

A ready-made function to randomly split your dataset into a **training set** (to build the tree) and a **testing set** (to check if it actually works on unseen data).

```python
from sklearn.metrics import accuracy_score
```

A ready-made function that compares predicted labels vs actual labels and gives you a percentage correct.

#### The dataset

```python
X = [
    [22, 0],
    [25, 1],
    ...
]
```

`X` is our **feature matrix** — each inner list is one customer: `[Age, Gender]`. Here `Gender` is encoded as a number (`0` and `1`) because ML models only work with numbers, not text like `"Male"`/`"Female"`.

| Age | Gender | Meaning    |
| --- | ------ | ---------- |
| 22  | 0      | customer 1 |
| 25  | 1      | customer 2 |
| 47  | 1      | customer 3 |
| 52  | 0      | customer 4 |
| 46  | 1      | customer 5 |
| 56  | 0      | customer 6 |
| 23  | 1      | customer 7 |
| 40  | 0      | customer 8 |

```python
y = [0, 0, 1, 1, 1, 1, 0, 1]
```

`y` is the **label/target** for each corresponding row in `X` — whether that customer bought (`1`) or didn't (`0`). Row 1 in `X` (`[22, 0]`) maps to `y[0] = 0` → that 22-year-old did **not** buy.

#### Splitting the data

```python
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42
)
```

* `test_size=0.25` → set aside **25%** of the data (2 out of 8 rows) for testing, keep **75%** (6 rows) for training.
* `random_state=42` → a "seed" number that makes the random split **reproducible** — anyone who runs this exact code gets the exact same split every time (this is why we can show you the *exact* real output below, not a guess).
* The function returns **4 things at once**: training features, testing features, training labels, testing labels — unpacked into 4 variables in one line.

#### Building and training the model

```python
model = DecisionTreeClassifier(criterion="entropy")
```

Creates an (empty, untrained) decision tree object, configured to use entropy/information gain for choosing splits.

```python
model.fit(X_train, y_train)
```

This is where the actual "learning" happens — the model looks at `X_train` and `y_train` together and **builds the tree structure** (decides the questions and thresholds) that best separates buyers from non-buyers.

#### Predicting and measuring accuracy

```python
y_pred = model.predict(X_test)
```

Feeds the **test features** (which the model has never seen labels for) through the now-trained tree, and gets back predicted labels.

```python
print("Accuracy:", accuracy_score(y_test, y_pred))
```

Compares `y_pred` (what the model guessed) against `y_test` (the real, correct answers) and prints what fraction it got right.

#### Predicting a brand-new customer

```python
new_data = [[30, 1]]
prediction = model.predict(new_data)
```

Simulates a real-world use case: a new customer who is **30 years old, Gender = 1** walks in will they buy?

```python
if prediction[0] == 1:
    print("Prediction: Buy")
else:
    print("Prediction: Don't Buy")
```

`prediction` is a list (because `.predict()` can handle many rows at once) — `prediction[0]` gets the answer for our one new customer, and we translate the `1`/`0` number back into a human-readable label.

***

### 3.4 What the tree actually learned (real diagram, generated by running your exact code)

![Decision Tree Diagram](https://claude.ai/chat/decision_tree.png)

Reading this diagram:

* **Top box (root):** the tree asks `Age <= 31.5?` — this was the single best question it could ask to separate buyers from non-buyers, based on entropy.
* **Left branch (Age ≤ 31.5 → True):** entropy = 0, meaning this group is **100% pure** — everyone here is "Don't Buy" (class 0).
* **Right branch (Age ≤ 31.5 → False):** entropy = 0 too — everyone here is "Buy" (class 1).
* Interestingly, the tree **never even used `Gender`** — it found that `Age` alone perfectly separates this particular tiny dataset, so Gender added no extra information gain.

#### Same tree, shown as text:

```
|--- Age <= 31.50
|   |--- class: 0        (Don't Buy)
|--- Age >  31.50
|   |--- class: 1        (Buy)
```

***

### 3.5 Actual output (verified by running the exact code)

```
Accuracy: 1.0
Prediction: Don't Buy
```

**Why Accuracy = 1.0 (100%)?** With `random_state=42`, the test set happened to be `[25, 1]` (true label: Don't Buy) and `[56, 0]` (true label: Buy). Since Age 25 ≤ 31.5 → tree predicts "Don't Buy" ✅ correct, and Age 56 > 31.5 → tree predicts "Buy" ✅ correct. Both test predictions matched reality, so accuracy = 2/2 = **1.0**.

**Why "Don't Buy" for `new_data = [[30, 1]]`?** Age = 30, and 30 ≤ 31.5 → the tree routes this customer down the **left** branch → predicted class = 0 → **"Don't Buy."** Note that Gender (`1`) played no role at all in this decision, since the tree only split on Age.

***

### 3.6 Quick recap for viva/oral exam

* A **Decision Tree** splits data repeatedly using yes/no questions on feature values, chosen to reduce **entropy** (uncertainty) the most at each step — this is called **Information Gain**.
* `criterion="entropy"` tells scikit-learn to use entropy-based information gain (alternative: `"gini"`).
* `train_test_split` divides data into training (build the model) and testing (evaluate the model) sets.
* `.fit()` = train the model. `.predict()` = use the trained model to guess labels for new/unseen data.
* `accuracy_score` = fraction of test predictions that matched the true labels.
* This is a **supervised learning** technique — it needs labeled data (`y`) to learn from, unlike unsupervised methods like clustering.
* This dataset is tiny (8 rows) — for real exams they may ask you what would happen with more data, or if the tree could **overfit** (become overly complex and memorize training data instead of generalizing) — a common Decision Tree weakness, usually controlled using `max_depth`.

Output: