Statistical Evidence in Discrimination Class Actions: Six Questions for Any Expert Report

Two qualified experts can analyze the same personnel data and reach opposite conclusions without either making an arithmetic error. The difference is almost always in a handful of choices made before any test is run. These six questions locate those choices. They apply equally to a report that finds a pattern and to one that finds none.

This guide is written by Roman Garagulagian, Ph.D., principal of Forensic Economic Services, LLC and Vice President–West of the National Association of Forensic Economics. It describes statistical method, not legal standards. Which legal theory applies and what it requires are questions for counsel.

Why statistics carry so much weight in these cases

In a pattern-or-practice case, the plaintiffs must show that discrimination was the employer’s regular practice and not an isolated event (International Brotherhood of Teamsters v. United States, 431 U.S. 324 (1977)). No single personnel file shows that. The Supreme Court has said that gross statistical disparities alone may, in a proper case, make out a prima facie showing (Hazelwood School District v. United States, 433 U.S. 299 (1977)). It has also said that statistics “are not irrefutable” and that their usefulness depends on the surrounding facts (Teamsters). Both statements apply to every report discussed below.

1. Who is being compared with whom?

Every disparity is measured against a benchmark. For hiring, Hazelwood said the proper comparison was the qualified population in the relevant labor market, not the general population. For pay and promotion inside a workforce, the benchmark is employees who were similarly situated: eligible for the same decision, in the same period, under the same decision-makers.

Check whether the comparison group includes people who could never have received the outcome, which dilutes a disparity, or excludes people who could have, which can create one or hide one. Ask to see the result under at least one alternative definition.

2. At what level were the data pooled?

This choice moves results more than any other. Splitting a workforce into many small units can leave each one too small to show statistical significance even when the overall pattern is strong. Pooling everything can produce a large disparity that partly reflects where each group works, not how it was treated within any unit.

Courts have recognized both problems. The Ninth Circuit has said that aggregated data may be used where it is more probative than subdivided data, particularly where small samples would distort the analysis (Paige v. California, 291 F.3d 1141 (9th Cir. 2002)). The Supreme Court has said that a regional disparity cannot by itself establish a uniform, unit-by-unit one (Wal-Mart Stores, Inc. v. Dukes, 564 U.S. 338 (2011)).

The right level follows where decisions were actually made and what the legal theory claims. A sound report shows the result at more than one level and includes a stratified test that combines the within-unit evidence.

3. Which controls are in the model, and where did they come from?

A regression compares employees who are alike on the variables it includes. In Bazemore v. Friday, 478 U.S. 385 (1986), the Supreme Court said that omitting variables normally goes to the weight of a regression, not its admissibility. The argument is therefore usually about which variables matter. Two checks:

• Missing controls. Does the model account for the major legitimate factors, such as job, tenure, prior experience and location? An omitted factor matters only if it both affects the outcome and differs between the groups.
• Controls shaped by the employer. A performance rating, a discipline history or a job assignment is decided by the same managers whose decisions are challenged. If the claim is that those decisions were biased, controlling for them can absorb the very effect being measured. Whether a control is tainted in this sense is partly a factual question, so the report should show the result with and without each such variable.

How the controls are chosen, what a tainted variable does to the estimate and how the result is read are covered in regression analysis in pay discrimination class actions.

4. Is the whole outcome in the data?

Before reading the model, read the data section. Does “pay” mean the base rate or total earnings? Does the period cover the whole class period or only part of it? Were employees dropped because a field was missing, and how many? An analysis of base rates can show no disparity while total earnings differ, and the reverse. Exclusions should be counted and explained.

5. What does “significant” mean in this report?

Courts often refer to a difference of more than two or three standard deviations as making chance an unlikely explanation, following a footnote in Hazelwood. For selection rates, the federal Uniform Guidelines use a four-fifths rule of thumb (29 C.F.R. § 1607.4(D)). Neither is a bright line.

Statistical significance depends on sample size. A small disparity in a very large workforce can be significant, and a large one in a small group may not be. Look for both numbers: the standard deviations, and the practical size of the gap in dollars, percentage points or positions. A report that gives only one is giving half the answer.

6. Does the result survive when the choices change?

A robust finding holds under reasonable alternatives. A fragile one reverses when a single choice changes. Ask for the workpapers, re-run the model, and change one choice at a time: the comparison group, the level of pooling, each contested control, the pay measure, the period. The choices that move the result are the ones to examine at deposition. The same approach applies to damages reports; see five problems to look for in an opposing economic damages report.

An illustrative example

The employer and every figure in this example are invented for illustration. It does not describe any real matter.

Suppose a hospital system with two facilities promotes hourly support staff into lead positions. Over three years, the records show:

One expert pools the data. Women were promoted at 14 percent and men at 30 percent, a difference of about 5.8 standard deviations, and women received about 36 fewer promotions than their share of the eligible pool would predict.

A second expert analyzes each facility separately. The gap is five percentage points in each: 1.4 standard deviations at Facility A and 0.9 at Facility B. Neither reaches two. Both experts’ calculations are correct.

The results differ because Facility B had more lead positions to fill and most women worked at Facility A. A stratified test, which compares women and men within each facility and then combines the evidence, gives about 1.6 standard deviations and a shortfall of about eight promotions.

Which figure answers the legal question depends on facts outside the table. If employees chose their facility, the within-facility comparison is the better measure of how promotion decisions were made. If the employer assigned them, and assignment is part of the claim, then controlling for facility removes part of what is being challenged, and assignment has to be tested on its own. The statistics do not settle that. They show what turns on it: the difference between roughly eight promotions and thirty-six. The tests used on promotion and hiring decisions, and the applicant and feeder pools they are run on, are set out in analyzing promotion and hiring disparities.

What to do with the answers

If you have received a statistical report, the six questions tell you which choices to probe. An economist can replicate the analysis from the workpapers and measure how much each choice moves the result, so you know where the dispute lies before deciding whether a competing study is needed. Our pay equity and wage discrimination practice does this work for plaintiff and defense counsel, and the rebuttal review page explains how to send a report.

Frequently asked questions

Is a statistically significant disparity proof of discrimination?

No. It shows that chance is an unlikely explanation for the difference. What does explain it is decided on the whole record, including testimony and documents. Statistics are one kind of evidence.

Can statistics matter when the group is small?

They can, but a small group limits what any test can show. Exact tests designed for small samples help. An honest report says when the numbers are too thin to support a conclusion either way.

What data should counsel request?

Personnel records with hire, job, location and separation history for every employee in the relevant jobs; payroll data with every pay component; discipline and performance records; applicant and promotion-eligibility records where selection is at issue; and the policies that governed each decision. The fields requested determine what can be tested later.

When should a statistical expert be brought in?

Before the data requests are final. An expert who sees the request list early can identify the fields a later analysis will need, and an early look at the data shows which claims the numbers can support.