Isonomy · Bias audit report

Bias Audit: ResumeRank v2

Demo Corp (synthetic data) · Generated 2026-11-21 02:54 UTC · Selection analysis on advanced_to_interview
Adverse impact detected
14 finding(s) across 3 demographic dimension(s), 1 of them intersectional.
People assessed
12,000
Overall selection rate
30.3%
Audit period
2026-03-01 → 2026-08-31
Findings
9 high · 5 medium

Key findings

high · impact
Adverse impact: Black or African American (race_ethnicity)
Impact ratio 0.66 (95% CI 0.60–0.72), selection rate 22.6% across 1,741 people; statistically significant (Holm-adjusted p = 1.7e-20).
high · impact
Adverse impact: Hispanic or Latino (race_ethnicity)
Impact ratio 0.71 (95% CI 0.65–0.77), selection rate 24.2% across 1,996 people; statistically significant (Holm-adjusted p = 1.1e-16).
high · impact
Adverse impact: Two or More Races (race_ethnicity)
Impact ratio 0.78 (95% CI 0.66–0.91), selection rate 26.6% across 418 people; statistically significant (Holm-adjusted p = 0.0027).
high · impact
Adverse impact: Female / Black or African American (sex × race_ethnicity)
Impact ratio 0.56 (95% CI 0.49–0.65), selection rate 21.1% across 802 people; statistically significant (Holm-adjusted p = 1.6e-18).
high · impact
Adverse impact: Male / Black or African American (sex × race_ethnicity)
Impact ratio 0.64 (95% CI 0.56–0.72), selection rate 23.9% across 913 people; statistically significant (Holm-adjusted p = 4.2e-14).
high · impact
Adverse impact: Female / Hispanic or Latino (sex × race_ethnicity)
Impact ratio 0.64 (95% CI 0.57–0.72), selection rate 24.0% across 963 people; statistically significant (Holm-adjusted p = 2.6e-14).
high · impact
Adverse impact: Male / Hispanic or Latino (sex × race_ethnicity)
Impact ratio 0.66 (95% CI 0.59–0.74), selection rate 24.7% across 1,004 people; statistically significant (Holm-adjusted p = 2.2e-13).
high · drift
Fairness regression: Hispanic or Latino (race_ethnicity)
Impact ratio fell from 0.82 (2026-03) to 0.55 (2026-08). Check for a model or policy change in this window.
high · drift
Fairness regression: Female (sex)
Impact ratio fell from 0.98 (2026-03) to 0.77 (2026-08). Check for a model or policy change in this window.
medium · proxy
Proxy feature: employment_gap_months ↔ sex
'employment_gap_months' is strongly associated with sex (Cramér's V 0.31) and with the outcome (V 0.20). It is a likely channel for disparity.
medium · proxy
Proxy feature: zip_region ↔ race_ethnicity
'zip_region' is strongly associated with race_ethnicity (Cramér's V 0.40) and with the outcome (V 0.14). It is a likely channel for disparity.
medium · explanation
Different reasoning for Female (sex)
'employment_gap_months' lowers this group's scores by 1.16 points more than the population average (explanation disparity 0.02).
medium · explanation
Different reasoning for Black or African American (race_ethnicity)
'zip_region' lowers this group's scores by 2.21 points more than the population average (explanation disparity 0.07).
medium · explanation
Different reasoning for Hispanic or Latino (race_ethnicity)
'zip_region' lowers this group's scores by 1.68 points more than the population average (explanation disparity 0.04).

1. Impact ratios (NYC Local Law 144 summary of results)

Impact ratio = category selection rate ÷ highest category selection rate. Categories below 2% of the sample are excluded from the impact-ratio calculation but still reported. Significance: Fisher's exact test vs. the reference group, Holm-adjusted within each table (α = 0.05).

sex

Female0.85Male1.000.80 threshold
CategoryNumber assessedSelection rateImpact ratio95% CIAdj. pStatus
Female5,71927.6%0.850.80–0.906.2e-09Monitor
Male6,11932.6%1.00——Reference group

Individuals with unknown category (not included above): 162

race_ethnicity

Black or African American0.66Hispanic or Latino0.71Two or More Races0.78Asian0.97White1.000.80 threshold
CategoryNumber assessedSelection rateImpact ratio95% CIAdj. pStatus
Black or African American1,74122.6%0.660.60–0.721.7e-20Adverse impact
Hispanic or Latino1,99624.2%0.710.65–0.771.1e-16Adverse impact
Two or More Races41826.6%0.780.66–0.910.0027Adverse impact
Asian1,22033.1%0.970.89–1.060.49Pass
White6,31534.2%1.00——Reference group
Native American or Alaska Native14825.7%———Excluded (small group)
Native Hawaiian or Pacific Islander4436.4%———Excluded (small group)

Individuals with unknown category (not included above): 118

Intersectional: sex × race_ethnicity

Female / Black or African American0.56Male / Black or African American0.64Female / Hispanic or Latino0.64Male / Hispanic or Latino0.66Female / White0.81Female / Asian0.88Male / Asian0.89Male / White1.000.80 threshold
CategoryNumber assessedSelection rateImpact ratio95% CIAdj. pStatus
Female / Black or African American80221.1%0.560.49–0.651.6e-18Adverse impact
Male / Black or African American91323.9%0.640.56–0.724.2e-14Adverse impact
Female / Hispanic or Latino96324.0%0.640.57–0.722.6e-14Adverse impact
Male / Hispanic or Latino1,00424.7%0.660.59–0.742.2e-13Adverse impact
Female / White3,01130.4%0.810.76–0.871.2e-08Monitor
Female / Asian58632.9%0.880.78–1.000.081Pass
Male / Asian62633.2%0.890.79–1.000.081Pass
Male / White3,21637.4%1.00——Reference group
Female / Two or More Races21219.3%———Excluded (small group)
Male / Two or More Races19933.2%———Excluded (small group)
Male / Native American or Alaska Native7428.4%———Excluded (small group)
Female / Native American or Alaska Native7223.6%———Excluded (small group)
Male / Native Hawaiian or Pacific Islander2433.3%———Excluded (small group)
Female / Native Hawaiian or Pacific Islander1936.8%———Excluded (small group)

Individuals with unknown category (not included above): 279

2. Proxy and leakage analysis

How well the model inputs jointly predict each protected attribute (cross-validated AUC; 0.5 = no information, 1.0 = fully encoded):

FeatureProtected attributeAssoc. with attributeAssoc. with outcomeProxy riskLevel
employment_gap_monthssex0.310.200.25strong
zip_regionrace_ethnicity0.400.140.24strong
commute_distance_kmrace_ethnicity0.260.130.18moderate
years_experiencesex0.020.390.08weak
skills_assessmentrace_ethnicity0.010.540.07weak
years_experiencerace_ethnicity0.010.390.07weak
employment_gap_monthsrace_ethnicity0.010.200.05weak
skills_assessmentsex0.000.540.00weak
education_levelsex0.000.190.00weak
education_levelrace_ethnicity0.000.190.00weak
zip_regionsex0.000.140.00weak
commute_distance_kmsex0.000.130.00weak

Association = bias-corrected Cramér's V (numeric features binned into deciles). Proxy risk = geometric mean of the two associations.

3. Explanation fairness

Explanation disparity = Jensen-Shannon distance between a group's average feature-importance profile and the population's (0 = identical reasoning). Stability deviation = how much less (+) or more (−) consistent explanations are for the group. Score penalty = feature whose contribution is most lower for the group than average.

sex

GroupnExplanation disparityStability deviationLargest score penalty
Female5,7190.020+0.007employment_gap_months (-1.16)
Male6,1190.019-0.007skills_assessment (-0.05)

race_ethnicity

GroupnExplanation disparityStability deviationLargest score penalty
Black or African American1,7410.072-0.001zip_region (-2.21)
Native Hawaiian or Pacific Islander440.062+0.006zip_region (-0.81)
Hispanic or Latino1,9960.037-0.006zip_region (-1.68)
Native American or Alaska Native1480.033-0.006zip_region (-0.39)
Asian1,2200.029-0.004education_level (-0.15)
White6,3150.027-0.002education_level (-0.01)
Two or More Races4180.018+0.004zip_region (-0.47)

sex × race_ethnicity

GroupnExplanation disparityStability deviationLargest score penalty
Female / Black or African American8020.089+0.009zip_region (-2.19)
Male / Black or African American9130.060-0.012zip_region (-2.22)
Female / Native American or Alaska Native720.057+0.007employment_gap_months (-1.75)
Female / Hispanic or Latino9630.044+0.003zip_region (-1.65)
Male / Asian6260.041-0.013skills_assessment (-0.51)
Male / White3,2160.041-0.008education_level (-0.03)
Male / Hispanic or Latino1,0040.039-0.017zip_region (-1.69)
Female / Two or More Races2120.029-0.002employment_gap_months (-1.03)
Female / Asian5860.029+0.003employment_gap_months (-0.96)
Female / White3,0110.023+0.003employment_gap_months (-0.99)
Male / Two or More Races1990.019+0.013zip_region (-0.52)
Male / Native American or Alaska Native740.010-0.016zip_region (-0.15)

4. Fairness drift over time

sex: impact ratio by month

0.500.751.002026-032026-042026-052026-062026-072026-08MaleFemale

race_ethnicity: impact ratio by month

0.480.741.002026-032026-042026-052026-062026-072026-08WhiteHispanic or LatinoBlack or African AmericanAsianTwo or More Races

5. Less discriminatory alternatives

Each alternative was retrained on the supplied features with 3-fold cross-validation and selects the same share of people as the current tool (30.3%). Validity = precision at that selection rate, measured against ground truth: job_success. Fairness = the worst impact ratio across every single and intersectional category.

Recommended alternative
Drop employment_gap_months
Worst impact ratio 0.56 → 0.86; adverse-impact groups 7 → 0; precision 79.2% → 85.6%. Remove the proxy feature 'employment_gap_months'.
AlternativePrecisionAUCWorst impact ratioWorst groupAdverse groupsUses protected data in training
Current tool (as deployed)79.2%—0.56Female / Black or African American (sex × race_ethnicity)7no
Retrained, same features85.5%0.910.85Male / Hispanic or Latino (sex × race_ethnicity)0no
Drop employment_gap_months recommended85.6%0.910.86Male / Asian (sex × race_ethnicity)0no
Drop zip_region85.6%0.910.83Male / Hispanic or Latino (sex × race_ethnicity)0no
Drop commute_distance_km85.6%0.910.85Male / Asian (sex × race_ethnicity)0no
Drop all flagged proxies85.5%0.910.86Male / Hispanic or Latino (sex × race_ethnicity)0no
Reweighing85.5%0.910.84Male / Hispanic or Latino (sex × race_ethnicity)0yes
Concept nullspace projection84.7%0.900.87Male / Hispanic or Latino (sex × race_ethnicity)0yes
Drop proxies + nullspace85.5%0.910.86Male / Hispanic or Latino (sex × race_ethnicity)0yes

6. Recommendations

7. Method and limitations

This report is decision-support software output, not legal advice. NYC Local Law 144 requires the bias audit to be conducted by an independent auditor; this report is designed to support that auditor and the employer's own ongoing monitoring.