Explore essential concepts of precision, recall, and ROC analysis to evaluate models on imbalanced datasets. This quiz helps reinforce key metrics and reasoning behind their use for fair, reliable machine learning assessment.
In a dataset with 5 actual positive cases, a model predicts 8 as positive, of which 4 are actually positive. What is the precision of the model?
Explanation: Precision is calculated as true positives divided by total predicted positives. Here, 4 true positives out of 8 predicted means a precision of 4 divided by 8, which is 0.50. The other options are incorrect because 0.80 uses total positives instead of predicted, 0.67 uses different counts, and 0.75 does not fit the calculation.
If a classifier identifies 6 out of 10 actual positive cases correctly, what is the recall of the model?
Explanation: Recall measures the proportion of actual positives detected, which is 6 out of 10 in this case, resulting in 0.60. 0.40 is too low, 0.85 and 0.90 are much higher than the actual recall and do not match the data given.
Which metric is most likely to be misleading in a highly imbalanced dataset with very few positive cases?
Explanation: In imbalanced datasets, accuracy can appear high because predicting the majority class correctly dominates the metric, even if the minority class is poorly detected. Recall and precision are less influenced by the class balance, and F1 combines both precision and recall, making them more informative.
Given a confusion matrix with 3 true positives, 2 false positives, 7 true negatives, and 1 false negative, what is the precision?
Explanation: Precision is true positives divided by the sum of true and false positives: 3 divided by (3 plus 2), which is 0.60. 0.75 uses incorrect values, 0.86 is not supported by this data, and 0.33 underestimates the value.
What does the ROC curve illustrate when evaluating a binary classifier?
Explanation: ROC curves plot the true positive rate (sensitivity) against the false positive rate at various thresholds, helping visualize trade-offs. Precision vs recall forms a different curve, sensitivity vs specificity is not how ROC is plotted, and accuracy vs error rate is unrelated to ROC analysis.
Why is the F1 score often favored over accuracy when evaluating models on imbalanced data?
Explanation: The F1 score combines both precision and recall into a single measure, making it more representative when data is imbalanced. It does not specifically penalize false negatives, only specificity is not captured, and while it doesn't use true negatives directly, that's not its main reason for use.
If a model has high recall but low precision, what is it most likely doing?
Explanation: High recall means most actual positives are found, while low precision indicates many of the positive predictions are incorrect (false positives). The other options describe opposite situations or misunderstand the definitions.
What does a higher value of Area Under the ROC Curve (AUC) indicate for a model?
Explanation: A higher AUC means the model separates positive and negative classes more effectively across thresholds. It does not specifically mean higher accuracy, increased positive predictions, or just reducing false negatives.
In which scenario is maximizing precision more important than recall?
Explanation: In fraud detection, a high number of false positives (low precision) is costly, so precision is key. Medical diagnosis values higher recall to avoid missing dangerous cases, and spam filters often prefer higher recall, while counting purchases is unrelated.
Which of the following is true about evaluating models on imbalanced datasets?
Explanation: Precision and recall help understand a model's effectiveness on the minority class, which accuracy can easily obscure in imbalanced datasets. Accuracy alone often hides poor class detection, ROC curves are still very useful, and high accuracy does not guarantee high recall.