5 Confusion Matrix Mistakes Even Seasoned Professionals Make! #38
Make sure you are learning to avoid them!
Hello Everyone,
Welcome to the 38th edition of my newsletter ML & AI Cupcakes!
Confusion matrix is one of the most discussed topics during AI/ML interviews. Not only that, it plays a significant role while building models on your job. So, make sure you are mastering this topic to the core.
In today’s newsletter, I have curated a list of mistakes practitioners often make while using confusion matrix. They must be avoided if you want to derive reliable insights regarding the performance of your model.
1. Not checking target variable class distribution
In case of classification problems, it may happen that some classes have much higher frequency as compared to other classes in the target variable. This leads to the problem of imbalance in the dataset. If class distribution is not determined (balanced or imbalanced) before evaluating the performance of the model, it may lead to misleading metrics and poor real-world performance.
Let’s understand this with a binary classification problem. For a fraud detection model (imbalanced classes with 98% data belonging to “non-fraud” class and 2% data belonging to “fraud” class), it will give an accuracy of 98% which is quite high and declare your model as excellent. But this model is not able to capture “fraud” cases which is more crucial for a financial institution.
So, always check the target variable class distribution before rushing towards evaluating the performance of the model.
Image credit: Author
2. Depending only upon accuracy
In case of binary classification problems, accuracy is often considered a primary evaluation metric. But relying solely on it can be misleading (as seen in the above example).
In case of imbalanced datasets, other evaluation metrics like precision, recall, F1-score etc. can be more useful.
Image credit: Author
3. Confused between the interpretation of precision and recall
Practitioners often get confused between these two metrics. If you don’t know their proper definition and difference, it will be difficult for you to decide which one should be paid more attention while solving a particular problem.
Precision tells out of all the predicted positives; how many are correct?
Recall tells out of all the actual positives; how many are predicted positive?
4. Assuming that the places for TP, TN, FP, FN are fixed
Practitioners often think that the place for the four components (TP, TN, FP, FN) in a confusion matrix is fixed. Mostly, they think that the first cell (upper left) is always TP. But this is not the case. In fact, when you compute the confusion matrix in python, the first cell is TN. If you don’t know about this, your results interpretation will be wrong.
So, always check the arrangement of actual and predicted values before interpreting TP, TN, FP and FN in a confusion matrix.
Image credit: Author
5. Assuming that TypeI error is more critical than TypeII error or vice versa.
Which error is more critical? The answer is, it depends upon the problem you are solving.
For example, in a medical diagnosis, TypeII error is more critical especially in the case of a deadly disease like cancer. Because if disease is not detected on time, it may lead to delay in patient’s treatment and consequently, increases the chances of death.
On the other hand, in case of selecting a candidate for a corporate job, TypeI error is more critical. The reason is, a company spends a lot of money in training employees. A wrong candidate not only wastes that, but also impacts the performance of the projects he is involved in.
Image credit: Author
What else would you like to add to the list?
Quick Knowledge Check!
What is the formula for recall?
A) TP/(TP+FP)
B) TP/(TP+FN)
C) TN/(TP+FP)
D) TN/(TP+FN)
Some helpful resources
https://kavitagupta.substack.com/p/how-to-construct-a-confusion-matrix
https://kavitagupta.substack.com/p/mcqs-on-confusion-matrix-24
Writing each newsletter takes a lot of research, time and effort. Just want to make sure it reaches maximum people to help them grow in their AI/ML journey.
It would be great if you could share this newsletter with your network.
Also, please let me know your feedbacks and suggestions in the comments section. That will help me keep going. Even a “like” on my posts will tell me that my posts are helpful to you.
See you soon!
-Kavita