What's the idea of Decision Tree Classifier?

The basic intuition behind a decision tree is to map out all possible decision paths in the form of a tree. It can be used for classification and regression (Decision Tree Regression ). In this post, let's try to understand the classifier.

Suppose that we have a dataset $S$ like in the figure below (ref)

An example of dataset $S$.

A decision tree we want.

There are many algorithms which can help us make a tree like above, in Machine Learning, we usually use:

• ID3 (Iterative Dichotomiser): uses information gain / entropy.
• CART (Classification And Regression Tree): uses Gini impurity.

Some basic concepts

• Splitting: It is a process of dividing a node into two or more sub-nodes.
• Pruning: When we remove sub-nodes of a decision node, this process is called pruning.
• Parent node and Child Node: A node, which is divided into sub-nodes is called parent node of sub-nodes where as sub-nodes are the child of parent node.

ID3 algorithm

• ID3 algorithm (TL;DR;)
• ID3 algorithm in detail

CART algorithm

• CART algorithm (TL;DR;)
• CART algorithm in detail

Gini Impurity or Entropy?

Some points:(ref)