Confusion matrix, Roc curve and AUC: the foundations of the machine assessment learning
To assess the performance of our fraud detection tools, we rely on the confusion matrix , from which we build and analyze the ROC (Receiver Operating Characteristic) curve and its associated indicator, AUC ( Area Under the Curve).
The confusion matrix allows us, at first, to view the four possible scenarios in a binary classification task.
What is a binary classification task?
This amounts to building a model capable of classifying a situation in two possible categories: "yes" or "no".
Take the case of COVVI-19 screening tests at the time, fortunately over, the pandemic. The tests could give four possible scenarios:
- True positive: the person had the covid and the test was positive.
- True negative: the person did not have the covid and the test was negative.
- False positive: the person did not have the covid, but the test was positive.
- False negative: the person had the covid, but the test was negative.
Transposed to the detection of fraud, it is exactly the same principle. We seek to store individuals in the categories: "" fraudsters "or" non-fraudsters ", as above with the" sick "or" not sick "categories. And this, by making the least possible classification errors.
At Meelo, our goal is to :
- detect a maximum of real positives - proven fraudsters;
- Seek to minimize false positives - customers wrongly identified as fraudsters;
- And above all do not let false negatives pass - fraudsters unidentified as such.
From the confusion matrix, we can calculate key indicators, such as the ROC curve (Receiver Operating Charactristic), which graphically illustrates compromise between fraud detection (the rate of real positives or properly detected fraud) and alert errors (the rate of false positives, in other words the legitimate transactions reported on different thresholds.
By varying the decision thresholds or "alert thresholds" (as we could vary the sensitivity of the COVVID test to detect the virus), we obtain different points on the curve, reflecting the performance of the model in various scenarios.
The AUC (Area Under the Curve), or " area under the curve " in French, measures the surface under the Roc curve. Being able to give value between 0.5 and 1, the AUC represents the overall capacity of the model to distinguish the fraud of normal transactions, taking into account all the decision thresholds. The closer the AUC, the better the performance of the model. An AUC of 0.5 would indicate that the model would not do better than chance, like a simple battery or face draw.
From AUC to Gini, there is only one step ...
Gini's coefficient is derived from AUC. While the AUC takes a value between 0.5 and 1, the Gini normalizes this measure to vary it between 0 and 1, which is often more intuitive. Mathematically, the Gini is calculated, from the AUC, according to the formula:
Gini = 2 × AUC - 1
The Gini therefore varies between 0 and 1 (or 0 % and 100 %). The higher it is, the better the power of discrimination of the model. Like the AUC, a Gini coefficient of 0.5 (50 %) corresponds to a random model (battery or face), while a Gini of 1 (100 %) represents a perfect model.
The Gini coefficient is a largely adopted metric, especially in the banking sector, to assess the overall efficiency of a scoring model.
However, despite its popularity, it has a crucial limit. It focuses mainly on the overall capacity of the model to be discriminated against between "good" and "bad" profiles, without taking into account, neither the imbalance between classes, nor the distribution of probability scores, nor the confidence that can be given to each individual prediction.
In cases where classes are strongly unbalanced (for example, 99 % negative and 1 % positive), the GINI can overestimate the performance of the model. This is precisely what we observe in most of the issues encountered with our customers. Fortunately, fraud rates are low: often around 1 to 2 % of transactions.
The Gini therefore gives a good overall indication of the effectiveness of a model, but is limited to a gross vision of performance, without taking into account the explanability of predictions.
This is why Meelo has chosen to integrate another metric into his tools in addition to the Gini: the Brier Score.
Towards the Gini and beyond ... thanks to the Brier Score!
Unlike the Gini, which measures the model's ability to separate good and bad profiles, the brier score assesses if our probability scores are close to reality. For example, if our model predicts 80 % of fraud chance, the Brier Score checks if, on average, this type of case is actually fraudulent in 8 out of 10 cases.
Imagine two models with an equivalent Gini coefficient. The first attributes very torn scores (close to 0 or 100 %), while the second concentrates the majority of its predictions around a gray area (including, for example, between 40 and 60 %). Although their overall capacity for discrimination can be comparable, the first model inspires more confidence, because it seems more reliable in its decisions.
It is precisely this notion of reliability that the Brier Score comes to measure by evaluating the difference between the probability predicted by the model (80 % risk for example) and the observed result (0 or 1). The lower this gap, the more the predictions are consistent and close to reality.
What are the advantages of the Brier Score?
At Meelo, we are convinced of the relevance of the Brier Score . Thanks to this efficient metric:
- Our decisions are more enlightened , because we do not just separate the vouchers from the bad, we assess the certainty of each prediction;
- We generate fewer “gray areas ”: optimization of the Brier score pushes our models to generate more extreme scores, reducing additional risks and verifications, synonyms of costs and friction in the customer journey;
- Our approach is even more ethical : conservative policies are avoided as much as possible thanks to more sharp decisions based on a real conviction of risk;
- Our users are more satisfied : by offering a complementary perspective to the GINI, the Brier Score allows our customers to better understand the reliability and distribution of fraud scores.
And the Brier Skill Score, Kézaco?
To facilitate the interpretation of the Brier Score , we often use the Brier Skill Score (BSS). The BSS measures if our prediction model is more efficient than a simple reference model. The higher the score, the more precise our model in its predictions. This is our way of validating that our model really brings added value to our customers.
Optimization of evaluation metrics: the results speak for themselves
During the re-training of our models, the integration of the Brier Score made it possible to significantly improve our performance. With equal data, the score has gone from 35 to just over 60 points, a gain of 30 points .
We therefore identify more fraud and more formally. This approach guarantees the reliability of our predictions and reduces the risk that the model behaves unstable in the face of atypical profiles.
At Meelo, we are convinced that the performance of a fraud detection model is not limited to its overall ability of discrimination. Going beyond standard indicators, we build solid confidence thanks to a truly lit fraud detection and we provide our customers with a solution that is both efficient and responsible.
Do you want to exchange with our anti-fraud experts and go further in analyzing your scores?
