Bias audit. The audit, and the honest state of it.
New York City Local Law 144 requires an annual independent bias audit before an automated employment decision tool is used, and requires the summary to be published. This is that page. Right now it says what most vendors would rather not say.
No independent bias audit has been completed yet, because Live has not yet been used to screen candidates at production volume. Publishing selection rates computed from a handful of pilot sessions would be worse than publishing nothing: the cells would be far too small to mean anything, and a number that looks like evidence but is not is the thing this regulation exists to prevent.
The audit is scheduled for the quarter in which the first customer reaches a defensible sample. Until this page carries a completed audit, Live must not be used for a role covered by Local Law 144, and the product enforces that: NYC is geo-gated at the job level and an invitation cannot be sent for a covered role.
What will be published here
The format is fixed now so it cannot be quietly softened later. Illustrative figures below, clearly marked, so you can see the shape of what you will get.
| Field | Value |
|---|---|
| Auditor | Independent third party, named, with no financial interest in the outcome |
| Audit date | Within the last 12 months, stated exactly |
| Tool distribution date | When this version of the tool was first used, as the rule requires |
| Scope | The tool as configured, by rubric version. A material change to the rubric or the weights triggers a re-audit rather than inheriting the old one |
| Data source | Voluntarily self-reported demographic data only, kept out of the scoring system’s inputs, which are the transcript and the rubric |
| Unavailable data | Reported as a count, never imputed. If most candidates decline to self-report, the audit says so and the numbers are weaker |
Selection rates and impact ratios · illustrative format, not real data
| Category | Selected | Total | Selection rate | Impact ratio | Four-fifths |
|---|---|---|---|---|---|
| Group A | not yet | not yet | not yet | 1.00 | reference |
| Group B | not yet | not yet | not yet | not yet | not yet |
| Intersectional | not yet | not yet | not yet | not yet | not yet |
| Unavailable | not yet | not yet | n/a | n/a | reported as a count |
Cells below the minimum size are suppressed rather than published, because a selection rate computed over four people is noise wearing a percentage sign.
What happens when a disparity shows up
Assume it will. A fairness program that has no procedure for finding something is not a program.
It is detected before the annual audit, not by it
Adverse impact analysis is planned to run quarterly and on every rubric version, not once a year, and is not built yet. The annual audit is meant to confirm what monitoring should already have found.
The affected component is isolated
Disparity is traced to a dimension, a level, or a problem. Assistance signals are planned and not built yet; when they ship, they will be reviewed separately every quarter for flag-rate disparity, and any signal with a material gap will be disabled pending investigation rather than left running.
Customers are told
Including customers who are not in a jurisdiction that would require telling them. A finding that only reaches regulated customers is a finding being managed rather than fixed.
The fix is re-tested against a calibration set
Because the obvious way to remove a disparity is to make the rubric measure less, and that trade has to be visible rather than accidental. No calibration set exists yet; building one is planned.
The rest of the fairness program
| Activity | Frequency | Artifact |
|---|---|---|
| Adverse impact analysis | Quarterly, and per rubric version, once built | Planned, not built yet: selection rate report by group, small cells suppressed |
| Independent bias audit | Annually | The summary on this page |
| Calibration agreement | Per model or rubric change | Planned, not built yet: agreement against an expert panel, with a floor below which a change will not ship |
| Integrity signal fairness | Quarterly, once signals ship | Planned, not built yet: flag rate by group, per signal |
| Accessibility audit | Annually | WCAG 2.2 AA conformance report covering the interview room specifically. An inaccessible interview is a discriminatory interview |
| Human oversight review | Monthly, once built | Planned, not built yet: override rate and delta by unit, which will also be a customer-facing product feature |
Aggregate selection-rate data is retained independently of candidate deletion, because it is aggregate and because the audit requires it. Deleting your interview does not remove your row from a count that no longer identifies you.
Your legal team can have all of this.
The audit package, the DPA, the sub-processor register and the accessibility conformance report are available before you sign anything.