What is an ROC Curve?


A commonly used method to evaluate the accuracy
of a classification model is a graphical representation called Receiving Operator Characteristics
curve, for short ROC curve. The ROC Curve shows the quality and robustness
of the classification model as defined by the false positive rate and true positive
rate – or in other words – sensitivity and 1-specificity. Many scoring metrics measure the performance
of a classification model for only one specific value of the threshold applied to the probability
of the positive class. However, this is just one possible working
point for the classification model. The ROC curve attempts to provide a more comprehensive view of the model performances for different classification thresholds. Here you see the ROC Curve. On the x-axis you see the false positive rate and on the y-axis the true positive rate. This blue line is the ROC curve of the model
we have trained. This black line is the ROC curve of a random classifier. So, how are all those different values of the false positive rate and the true positive rate calculated? Let’s have a look. For each input event, a classification model
produces a class probability that determines the class assignment. By default, the events are assigned to the
positive class if the probability of the positive class is 0.5 or higher, and to the negative
class otherwise. If we increase this classification threshold,
fewer data points will be assigned to the positive class. If we decrease the threshold, more events
will be assigned to the positive class. Here we see a number of events ordered by
their probabilities to belong to the positive class. By default, the value of the classification
threshold is set to 0.5. And here are the corresponding assignments
to the two classes. Next, we decrease the threshold value, for
example to 0.2. Now, we see that the class assignment changes
for these data points with a probability for the positive class falling between 0.2 and 0.5. Finally, let’s increase the classification
threshold to 0.7. Now, only those events will be assigned to
the positive class, because their probability for the positive class is greater than 0.7. As you can see here, the true positive rate
and the false positive rate also change every time we change the value of the threshold. The true positive rate is calculated as the
number of correct predictions to the positive class divided by the total number of events in the positive class. The false positive
rate is calculated as the number of incorrect predictions to the positive class divided by the total number of events in the negative class. For these threshold values, let’s plot the
false positive rates on the x-axis and the true positive rates on the y-axis. This dot represents the performance of the
model for the default threshold, this dot stands for the threshold
of 0.2, and this dot for the threshold of 0.7. If we now connect these dots, we have a sketch
of the ROC curve. In reality, the ROC curve is drawn based on
many more threshold values and therefore many more pairs of these two rates. The ROC curve of a perfect classifier would
move along the axis to reach this point for the perfect threshold, lying in this corner. Here, the true positive rate is 1 and the false positive rate is 0. The closer the curve of our model is to that
point, the better the model is performing. In the case of a random classifier for a two-class
problem, all events have the same probability for the positive class independently of their
actual class values. By varying the value of the classification
threshold the ROC curve becomes this straight line. This is the worst performance your model can
produce. If the ROC curve of your model lies below
the line of the random classifier then you probably made a mistake in building the ROC curve. The ROC curve has a few advantages as a model
evaluation technique: Firstly, it allows you to visually compare
different models in terms of performance. If we have more than one model, the model
with the highest ROC curve will be the best performing one. Secondly, if the eye comparison among model
curves becomes complicated, we can always use the Area Under the Curve, for short AuC,
measure. The AuC measures this area here. It is 1 for the perfect classifier and 0.5 for the random classifier. Thirdly, the ROC curve allows us to choose
the threshold that produces the best performance. One method to select the optimal threshold
is to minimize the Euclidean distance between the curve and the point in the top left corner
indicating perfect model performance. The point in the curve that minimizes the
distance is this tangent point shown here. In this video, we have shown how an ROC curve is drawn, given the actual class values, probabilities
of the positive class, false positive rate, true positive rate, and classification thresholds. We have also shown the robustness of the ROC
curve as a method to quantify and compare model performances.

One Reply to “What is an ROC Curve?”

Leave a Reply

Your email address will not be published. Required fields are marked *