Gradient Boosting Machine (GBM) Simplified ! #11

A classroom teaching analogy to understand GBM!

May 31, 2024

GBM is one of the most popular machine learning algorithms. It comes under the category of ensembling techniques. It combines several weak learners(known as base models) to generate a final strong model.

GBM starts by building a simple base model and identify the errors made in predictions by that model. Once those errors are identified, a second model is built to correct the errors from the first model. Then a third model is built to correct the errors from the second model. This process is continued for several iterations with the aim of improving prediction quality with each passing iteration.

Thus, GBM build several sequential models in a way that each model focuses on correcting the errors made by the previous model. By combining all these models, we create a strong model.

Simple Process of GBM is given below:

Start with a simple model (most commonly used is a decision tree) to make initial predictions.
Calculate the residuals (difference between actual and predicted values).
Train a new model on these residuals.
Adjust the predictions by adding new model’s predictions to the previous predictions.
Repeat the process for several iterations. Keep adding models, each one improving on the mistakes of the previous models.

Let’s understand GBM with a classroom teaching analogy!

Imagine you are a new teacher who is assigned a Mathematics class which is not performing well in the exams. Your aim is to improve overall test scores of your students. Instead of spending too much time in making a perfect lesson plan from the start, you decide to use a step-by-step approach to identify and fix specific mistakes they are making.

Here’s how you can think about Gradient Boosting Machine (GBM) in this context:

Step 1: Initial Lesson Plan

You start with a basic lesson plan. You teach the students and announce a test for them based on whatever was taught in the class. After the first test, you notice that while some students did well, many still made mistakes. This initial lesson plan is like the first weak model in GBM. It is not perfect but gives you a starting point.

Step 2: Analyzing Mistakes

Next, you analyze carefully the mistakes made by the students on the test. You find out that some students consistently made mistakes on specific topics. This is similar to the residual errors or the mistakes made by the initial weak model.

Step 3: Targeted Teaching

Instead of completely redesigning your lesson plan, you focus specifically on the topics where students made the most mistakes. You provide targeted tutoring sessions on these topics. The objective is to improve the performance of the students in these topics. This is same as creating a new model that tries to correct the errors made by the initial model.

Step 4: Combining Efforts

You give another test after the targeted teaching. The students’ overall performance improves, but there are still some mistakes. You repeat the process: analyze the new mistakes, provide additional targeted teaching, and test again. Each round of teaching builds on the previous one, addressing specific shortcomings each time. This iterative process of adding new models to correct the mistakes of previous ones is the essence of boosting.

Step 5: Final Performance

After several rounds of targeted teaching and testing, the students’ test scores significantly improve. Each teaching session contributed to a better overall understanding, just as each new model in GBM reduces the errors of the combined model ensemble. The final performance is a result of all the incremental improvements.

Quote of the day

“We cannot solve problems with the kind of thinking we employed when we came up with them.” —Albert Einstein

P.S. Let’s grow our tribe. Know someone who is curious to dive into ML and AI? Share this newsletter with them and invite them to be a part of this exciting learning journey.

ML & AI Cupcakes

Discussion about this post