Analyzing Promotion and Hiring Disparities: How Economists Test Selection Decisions

Hiring and promotion decisions are analyzed differently from pay. Pay is a continuous outcome measured for everyone; a selection decision is a yes or no, made among a defined set of candidates, one position at a time. The statistics therefore turn on how the pool of candidates is defined, which test suits the size of the numbers, and whether decisions made in many units or years can be combined. This note sets out how economists test selection decisions in hiring and failure-to-promote cases, individual and class, and what the results can and cannot show. The practice is on our pay equity and wage discrimination page.

Why selection decisions are analyzed differently

A pay analysis compares what employees were paid. A selection analysis compares what happened to candidates: of the people who could have been chosen, how many from each group were, and how does that compare with what would be expected if the group made no difference. The expected number depends entirely on who is counted as a candidate, so the pool is the choice that matters most, and it is the one the Supreme Court addressed in Hazelwood School District v. United States, 433 U.S. 299 (1977), when it held that the relevant comparison for teacher hiring was the qualified labor market, not the general population. Everything else in the analysis follows from the pool: the test, the standard deviations, the shortfall and the remedy.

Defining the pool

Hiring: applicant flow or availability

Where the employer kept applicant records, the pool is the applicants, and the analysis compares selection rates among them. The Uniform Guidelines on Employee Selection Procedures require most employers to keep records of applicant flow by race, sex and ethnicity (29 C.F.R. § 1607.15), and the Census Bureau’s EEO Tabulation and the American Community Survey supply external benchmarks when they did not. An availability analysis estimates the share of qualified workers in the relevant labor market who belong to each group, by occupation and geography, and compares it with the share hired. The geographic area matters as much as the occupation: Hazelwood turned in part on whether the St. Louis city labor market belonged in the comparison. Where applicant data exist but are incomplete, the two methods are run side by side and the report says where they agree.

Promotion: eligible and feeder pools

For promotion the pool is the employees who were eligible when each position was filled: in the feeder jobs, with the required tenure or rating, at the location or within the posting area. Two designs are common. A requisition-by-requisition analysis treats each filled position as a separate event with its own pool, which is precise but demands records of who was considered for each opening. A period analysis counts promotions from each feeder job over a year and compares promotion rates by group among the incumbents of that job, which needs only personnel history. Where positions were never posted and managers selected by tap on the shoulder, the pool is defined by the eligibility rules, and the absence of applicant records is itself part of the record.

Terminations and reductions in force

Terminations and reductions in force are selections in reverse: who was chosen to leave, from the pool of those at risk, and the same tests apply with the groups reversed. They raise their own issues of pool definition, the decisional unit in a layoff, the treatment of voluntary departures, and are addressed on our employment litigation page.

The tests, and when each applies

Four tests cover nearly every selection case, and the choice among them depends on the size of the numbers and the structure of the decisions.

The bridge from any of these tests to the remedy is the shortfall: the difference between the number of group members selected and the number expected under neutral selection. It is reported with every test, because the magnitude of a disparity is a separate question from whether it is significant, and both have to be stated.

Aggregation and the unit question

A disparity can be present in the aggregate and absent in every unit, when one group is concentrated in units that select few people of either group, and it can be present in every unit and invisible in the aggregate. Wal-Mart Stores, Inc. v. Dukes, 564 U.S. 338 (2011), held that regional and national statistics could not by themselves establish a common pattern where store managers held the discretion. The statistical answer is to test at both levels. The stratified test combines the within-unit results without assuming the units are alike, and a separate heterogeneity test asks whether the disparity differs across units by more than chance. Combining across years is a related question, and the Ninth Circuit approved it in Paige v. California, 291 F.3d 1141 (9th Cir. 2002), where promotion data from several examination cycles were aggregated. The six questions to put to any statistical report include a worked example of a pooled promotion disparity that disappears when the facilities are separated; the illustration below shows the opposite case.

Subjective systems and the record

Promotion systems that rely on managerial judgment are not beyond statistical analysis. Watson v. Fort Worth Bank & Trust, 487 U.S. 977 (1988), held that subjective or discretionary practices can be challenged under the disparate-impact framework, and the Civil Rights Act of 1991 provides that where the elements of a decision-making process cannot be separated for analysis, the process may be analyzed as one employment practice (42 U.S.C. § 2000e-2(k)(1)(B)(i)). The difficulty is evidentiary rather than statistical: without postings or candidate slates, the eligible pool has to be reconstructed from the eligibility rules and the personnel history, and the report has to say how. The employer’s own documents usually define the rules, and the analysis is only as defensible as its fidelity to them. The pattern-or-practice framework of International Brotherhood of Teamsters v. United States, 431 U.S. 324 (1977), pairs the statistics with the anecdotal evidence; the statistics show the pattern, and the testimony shows the mechanism.

From disparity to remedy

Once liability is found, the shortfall becomes the measure of relief. The remedies are instatement or promotion to the position that would have been obtained, retroactive seniority (Franks v. Bowman Transportation Co., 424 U.S. 747 (1976)), and back pay equal to the difference between what the class member earned and what she would have earned in the position. In the second stage of a Teamsters proceeding each member is presumed a victim and the employer bears the burden of showing that she would not have been selected anyway. Where there were fewer openings than victims, courts have divided class-wide back pay among the shortfall group rather than attempting to identify which individuals would have been chosen (see, for example, Pettway v. American Cast Iron Pipe Co., 494 F.2d 211 (5th Cir. 1974)).

The economist’s contribution at this stage is the expected-earnings path. For each class member the model computes the probability that she would have been promoted in each year under neutral selection, from the pools and rates established at the liability stage, and builds an expected but-for pay path from the pay of the positions she would probabilistically have held. Summed across members, the paths give class-wide back pay; read individually, they give each member’s share. Where a member was later promoted or left the employer, the path adjusts at that date. The method connects to the class-wide pay model described in calculating class-wide damages in equal pay litigation, and the components that move with pay, overtime, bonus and retirement contributions, are handled the same way.

Reviewing an opposing selection analysis

When retained to review a selection analysis, ours or the other side’s, we ask seven questions. How was the pool defined, and what happens under the next most reasonable definition? Were candidates counted as eligible at the time of each decision, or as of a later snapshot? Were requisitions, units and years pooled, stratified or both, and does the conclusion change? How were withdrawn candidates, internal transfers and declined offers treated? Was the test one-tailed or two-tailed, and was that stated? Which legitimate factors were controlled for, and are any of them products of the practice being challenged? And can the analysis be rerun from the records produced? A report that cannot answer the first and last questions is not yet a report.

A worked illustration

The figures are invented and describe no employer. A company received 1,000 applications for 120 openings in one job over two years: 400 from women and 600 from men. It hired 30 women and 90 men. Women’s selection rate was 7.5 percent and men’s 15.0 percent, a ratio of 0.50, well below the four-fifths threshold. Under neutral selection the expected number of women hired is 48; the shortfall is 18, and the standard deviation of that shortfall is about 5.0, so the pooled disparity is 3.6 standard deviations.

The company hires through three divisions, and the pooled result conceals where the disparity lives.

Combining the three divisions with a stratified test gives a shortfall of 19.3 women and 3.9 standard deviations, slightly stronger than the pooled result because women applied disproportionately to Division A, which hired at the highest rate. The division results show that the pattern is concentrated in Division B, where women were selected at one quarter of the men’s rate and the disparity alone is 4.0 standard deviations; Division A is within the range of chance and Division C, with twenty hires, is too small to decide on its own. Fisher’s exact test gives the same verdicts for the small divisions. For plaintiffs’ counsel the result identifies the practice to examine; for defense counsel it identifies the divisions where the statistics do not support the claim. Neither reading is available from the pooled figure alone.

Where to start

If you are evaluating a hiring or failure-to-promote matter, start with a conflict check using party names only; the data the analysis needs, applicant flow, requisition records, eligibility rules and personnel history, are listed in the hub’s what we request from counsel. If the other side has served an adverse-impact analysis, send it to us for review. The certification questions these analyses have to answer are summarized in our note on class certification.

Frequently asked questions

What is an applicant flow analysis?

A comparison of selection rates among the people who actually applied for a position or a set of positions, by race, sex or other protected group. It is the preferred hiring analysis where applicant records are complete, because the pool is the employer’s own.

What is an availability analysis?

An estimate of the share of qualified workers in the relevant labor market who belong to each group, built from Census Bureau occupation and geography data, used as the benchmark when applicant records do not exist or are unreliable. Its accuracy depends on defining the occupation and the recruiting area the way the employer actually drew its workforce.

Is the four-fifths rule a legal standard?

It is an enforcement guideline in the Uniform Guidelines, 29 C.F.R. § 1607.4(D), that agencies use as a screen for adverse impact. Courts have treated it as evidence rather than a rule of decision, and the Guidelines themselves say that smaller differences can constitute adverse impact if they are significant in statistical and practical terms. A careful report gives the selection-rate ratio and the standard deviations together.

Which test is right for small numbers?

Fisher’s exact test, or the hypergeometric distribution it is based on, which computes the exact probability of the observed shortfall without relying on an approximation. Its limitation is power: with a pool of a dozen candidates, even a large disparity may not reach conventional significance, which is a reason to combine small units with a stratified test rather than to test each alone.

Can promotion discrimination be shown without applicant data?

Yes, where the eligible pool can be reconstructed from the employer’s eligibility rules and personnel history, which is the usual situation when positions were not posted. The analysis compares promotion rates among eligible incumbents of the feeder jobs, and the report states how eligibility was determined and what the result is under alternative definitions.