Accuracy=(TP+TN)/All Predictions
Misclassification Rate=(FP+FN)/All Predictions or 1-Accuracy
Sensitivity=TP/(TP+FN)
- a.k.a. True Positive Rate, Recall
Specificity=TN/(TN+FP)
- a.k.a. True Negative Rate
Precision=TP/(TP+FP)
- a.k.a. Positive Predictive Valu
F1 score=2*(Precision*Recall)/(Precision+Recall)
- F1 score is the harmonic mean of precision and recall, if you care about precision and recall roughly the same amount, F1 score is the metric to use.
- The goal here is to get F1 score get close to 1.
Balanced Accuracy Score=(Sensitivity + Specificity)/2
- It’s especially useful when the classes are imbalanced.
- The goal here is to get Balanced Accuracy Score get close to 1.
ROC Curve:
- We use the area under the ROC curve (abbreviated ROC AUC or AUC ROC) to quantify the gap between our distributions.
- If you have an ROC AUC of 0.5, your positive and negative populations perfectly overlap and your model is as bad as it can get.
- If you have an ROC AUC of 1, your positive and negative populations are perfectly separated and your model is as good as it can get.
- The closer your ROC AUC is to 1, the better. (1 is the maximum score.)
- If you have an ROC AUC of below 0.5, your positive and negative distributions have flipped sides. By flipping your predicted values (i.e. flipping predicted 1s and 0s), your ROC AUC will now be above 0.5.
- ROC curve helps us to visualize our tradeoff between sensitivity and specificity and understand how well-separated our populations are.
Using which metrics are really depend on your problem, for example, if you want to predict spam email, then you want to minimize False Positives (predict spam emails when they are not), because you don’t want to miss any of non spam emails. Another example would be medical diagnosis, you would want to minimize False Negatives (predict benign tumor when the patient actually have malignant tumor), because you don’t want to miss any patient that actually have malignant tumor.