Explore your understanding of fairness metrics in machine learning model evaluation with these beginner-friendly questions. Learn key concepts such as disparate impact, demographic parity, equal opportunity, and more to assess ethical and unbiased model performance.
Which fairness metric requires that the proportion of positive predictions be equal across different demographic groups, regardless of actual outcomes?
Explanation: Demographic Parity demands that a model produces positive outcomes at the same rate for all demographic groups, promoting equal treatment. Equal Accuracy is incorrect, as it refers to equal performance rather than equal positive rates. Predictive Value Parity focuses on the probability of correctness, not outcome rates. Specificity measures true negatives, unrelated to demographic consideration.
If a hiring model selects 50% of applicants from group A and 25% from group B, which fairness concept is potentially being violated?
Explanation: Disparate Impact refers to a substantially different selection rate between groups, as in this scenario. Calibration involves aligning predicted and actual risk but doesn't focus on selection rates. Recall Parity relates to true positive rates across groups and isn't directly about selection. False Omission Rate is about incorrect negative predictions, not overall selection disparity.
What does the fairness metric 'Equal Opportunity' require for a model used to predict loan approvals?
Explanation: Equal Opportunity ensures that qualified candidates, or those who should receive a positive prediction, have equal chances regardless of group. It doesn't guarantee equal group sizes, making the second option incorrect. Equal false positive rates describe a different fairness metric (Equalized Odds), and balanced overall accuracy is about performance, not opportunity.
Which of the following best describes statistical parity in model evaluation?
Explanation: Statistical parity, synonymous with demographic parity, focuses on ensuring equal chances of positive outcomes. Equal likelihood of negative outcomes pertains to different metrics. A symmetric confusion matrix is unrelated to fairness between groups, while identical ROC curves focus on overall model performance, not parity of outcomes.
When a model’s positive predictions are equally accurate for all groups, which fairness metric does this most closely describe?
Explanation: Predictive Parity ensures that the probability of a correct positive prediction is the same across groups. Base Rate Parity is about equal probability of belonging to a group. Equalized Odds requires both true and false positive rates to be equal, while Balanced Error Rate averages errors, not prediction correctness.
What does the fairness metric 'Equalized Odds' require from a predictive model in healthcare?
Explanation: Equalized Odds aims for both true positive and false positive rates to be equal among groups, ensuring fair errors and successes. Disease prevalence is about world facts, not a model metric. Maximizing specificity doesn't address fairness between groups. Identical threshold values might not guarantee fairness if distributions differ.
In the context of fairness, what does model calibration refer to?
Explanation: Calibration in fairness involves ensuring that, for each group, predicted probabilities align well with actual observed outcomes. Assigning identical predictions would ignore individualized predictions, rendering models useless. Maximizing true negatives or balancing false positives are different concerns, not calibration.
A school admission classifier wrongly predicts more students from one group as qualified than another; which metric has likely failed?
Explanation: False Positive Rate Parity seeks to ensure groups are equally likely to be incorrectly assigned positive outcomes. Overall precision measures accuracy of positive predictions, not group fairness. True negative rate measures correct rejections, but doesn’t address false positives. Log loss evaluates probabilistic errors, not parity between groups.
Why are fairness metrics important in evaluating machine learning models for social applications?
Explanation: Fairness metrics help identify and address potential biases affecting outcomes experienced by specific groups. While speed is important, fairness metrics do not focus on computation time, making the second option wrong. They supplement—not replace—accuracy and precision. Achieving perfect fairness is often unrealistic, so the last choice is incorrect.
When should you choose Equal Opportunity over Demographic Parity for evaluating a model?
Explanation: Equal Opportunity is appropriate when focusing on fairness for those truly eligible, ensuring equal true positive rates. Lack of protected groups negates the need for group-based fairness metrics. Equal base rates refer to population statistics, not the metric's suitability. Guaranteeing identical overall outcomes is neither practical nor the goal of Equal Opportunity.