I think this partially buries the lede: "As a single hiring vendor comes to dominate screening for an industry, it may be more likely that candidates are shut out."

If we move to using just a small number of AI models to help do things like hiring, we will amplify biases and possibly completely lock out portions of the population. We need to be very careful when using AI systems to evaluate people in general -- not because they might be biased (which they might be), but because even a small bias, if used by virtually everyone, can be damning.

Anyone who’s done hiring wouldn’t be shocked by this:

We find applicants are more likely to be rejected from every position they apply to than would be predicted by the baseline of each position making statistically independent decisions.

Obviously a rejected resume is more likely to be rejected by every other employer and an accepted resume is more likely to be accepted by every other employer. Like online dating, most employers are looking for some baseline indicators that you are going to be successful and stable.

• zeroonetwothree

  today at 10:09 PM

  Yes I don’t understand why this is surprising or problematic at all?

  Actually the fact that they found this result didn’t hold in a different dataset is especially weird.

Did I miss the part of the article where they break down how they determined race? Is the algorithm blind to race? It looks like they specifically looked at 83k people applying to ~100 companies which notably were Fortune 500 companies. Could there simply be candidate discrepancies here? Hard for me to follow the full methodology but it doesn't necessarily seem either malicious or that well structured. Don't you need to have a control group of applicants who are similar on paper? To allege DISCRIMINATION is quite bold.

Definitely open to opposing or critical views

• rayiner

  today at 10:02 PM

  That’s not the data set used for this paper: https://algorithmichiring.github.io/

  If you click through, the paper says the race is self-reported.

  “Our data tracks 4,197,168 applications. It includes applicant gameplay features and for each application, the application date, the position name and employer, metadata about the position and employer, and the numerical score and final recommendation each applicant received for each completed application. 40.2% of applicants self-report race with a breakdown of 16.8% Asian, 14.2% White, 3.6% Black, 3.0% Hispanic, and all other racial categories below 2% (i.e. fewer than 100,000 applicants).”

• zerocrates

  today at 9:09 PM

  The 83,000 applications to Fortune 500 companies, that was a different previous study they compared their results to. This paper's takeaway is that unlike that Fortune 500 data, the applications here that went through an ML vendor's screening process showed evidence of "systemic rejection," where some applicants got rejected across the board at higher rates than you'd expect if they were facing independent would-be employers.

• gacgacgac

  today at 8:47 PM

  Yes. You missed it. They are using a test dataset of 83k resumes generated in 2022 for this paper and comparing it as a baseline against their observational data: https://www.nber.org/papers/w29053

  The dataset is constructed, deliberately, to hold candidate performance constant and vary the names of candidates to appear to be associated with a specific race.

  • xp84

    today at 10:08 PM

    Wow. So, all the 'people' and 'resumes' involved are fake, but they submitted them to real jobs?

    Cool.

    In any event, I'd happily support a ban on all parts of the ATS that could be involved in automated approval, rejection, or scoring being able to see candidate names. But I sense the author of this has a bigger agenda.

  • today at 10:00 PM

  • AStrangeMorrow

    today at 9:31 PM

    From looking at how that was done, it seems they (the paper you linked) used an older paper which looked at which names are frequent enough and more biased toward a certain demographic (90% of that name occurrence falls within that demographic).

    But they picked 9 family names per group. Which sounds quite low. And combined that with first names to reach 500 first+last names per group.

    I wonder how much of the bias we see has to do with the names actually picked versus it being racially motivated (absolutely not denying that this probably is a factor, but might not be the only one).

    For example, in France there is the national BAC end of high school exam. If you you at the names X grade distribution, and look at the higher “very good” bracket: some names are heavily under-represented (less than 5% of say “Jordan” get that grade) while some are over-represented (35% of “Josephine” get such a grade). The exam is for the most part anonymous, but some names are definitely heavily correlated with lower/higher income groups. So nothing surprising: Josephines tend to come from richer families, thus in average get better education/support, thus better grades. Same thing is true with family names to a smaller extent.

    So I wonder how much of the bias we see, be it from real persons or the AI has more to do with a class thing than a racial thing. Again those are not neatly separate things, but still

Misleading title the paper [0] does not mention any CV screening that might suggest racial or gender bias. It is purely about assessment tool. No AI or LLMs.

I'm not saying AI is not biased, but this study does not prove that.

[0] https://arxiv.org/pdf/2605.27371

From the paper:

> Fig. 1. The pymetrics process. > Stage 1: Applicants apply to positions. > Stage 2: Applicants are directed to the pymetrics platform to play assessment games. > Stage 3: pymetrics algorithms use applicant gameplay features to recommend 58.2% of applicants per position on average. > Stage 4: Employers decide which applicants to interview or hire, typically rejecting applicants that were not recommended by pymetrics.

The European Union passed The Artificial Intelligence Act, which classifies:

High-risk – AI applications that are expected to pose significant threats to health, safety, or the fundamental rights of persons. Notably, AI systems used in health, education, recruitment, critical infrastructure management, law enforcement or justice. They are subject to quality, transparency, human oversight and safety obligations

That's a pretty common sense legislation to me.

• anon373839

  today at 8:22 PM

  The AI “safety” industry is lobbying for federal preemption so that states won’t have the power to enact these types of sensible regulations.

• 72027372920

  today at 8:27 PM

  [dead]

> To measure adverse impact, we apply the EEOC’s “four-fifths rule,” which flags a position when one group is recommended at less than 80% of the rate of the most-recommended group

That seems like a nonsensical way to measure racial discrimination. What could justify it?

• gacgacgac

  today at 8:38 PM

  Have you googled this? The EEOC is a federal agency, and they've published on this topic quite extensively. The four fifths rule is used to define if there is a "substantially different selection rate". It does not measure racial discrimination. It measures selection rate.

  It indicates there may be adverse impact to one group. It specifically is not used to resolve racial discrimination.

  It's purely a signal for "we should consider asking more questions, because this appears unusual". That's what your quote says too, it "flags" a low recommendation -- it's indicating further study and investigation is likely warranted.

  • rayiner

    today at 9:11 PM

    Your summary of the EEOC guidance is correct. The problem is that the study here is using the four-fifths rule as a measurement of discrimination, instead of as a flag that triggers further investigation. It's in section 3.1 of the paper: https://arxiv.org/pdf/2605.27371.

    "Adverse impact occurs when there is (i) practically and (ii) statistically significant disparities in the selection rate for the group of interest when compared against the selection rate ′ of the most selected group ′ . Practical significance requires the impact ratio ... to be less than 0.8, which is why the EEOC guidance is colloquially referred to as the 'four-fifths' rule."

    The headline numbers reflect the positions for which the 4/5 rule was triggered, not the result of some further investigation: “We discovered that 26% of Black applicants and 15% of Asian applicants applied to positions where the AI system discriminated against their racial group.” Based on the methodology, I think that means that 26% of black applicants applied to positions that were flagged under the 4/5ths rule.

• nemomarx

  today at 8:32 PM

  I guess it measures if there's more than one std deviation gap between highest and lowest? Assuming that's twenty percent here

  it sounds like how you'd get that kind of metric at least

• paisawalla

  today at 8:38 PM

  This is an application of the disparate impact doctrine. Even facially neutral policies are considered suspect if they produce results that correlate against protected groups, irrespective of intent.

  This doctrine is the basis for much of employment law. It is a significant reason why employers don't administer IQ tests (or equivalents) to screen candidates since ~the 90s.

  A common objection to the doctrine is that it leads to unfalsifiable discrimination claims, which is why it seems nonsensical to you.

  • 59percentmore

    today at 9:31 PM

    And a common rebuttal to the objection is that systemic racism is often difficult to untangle in a way that produces a neat chain of cause and effect (not least of which because discrimination can happen unconsciously or secretly); because the impact exists whether intent can be shown or not, the desire remains to ameliorate that impact.

    If the issue happens upstream of the defendant to a claim - generally an organization being sued by an individual with fewer resources - it incentivizes such entities to push for changes upstream, so that they don't get stuck with the bill.

    • paisawalla

      today at 9:46 PM

      What evidence would disprove the claim that systemic racism is the cause of a persistent disparity?

  • gacgacgac

    today at 8:40 PM

    Importantly, the rule is not used to resolve racial discrimination claims. It's purely meant as the first test to evaluate whether a deeper dive is warranted. Fast, first pass data analysis tools are very useful for spotting unintended consequences.

    • Manuel_D

      today at 9:49 PM

      To the contrary, companies have been found liable for discrimination solely based on having the wrong percentages outcomes in its objective hiring assessments: https://en.wikipedia.org/wiki/Griggs_v._Duke_Power_Co.

    • paisawalla

      today at 8:55 PM

      You are selectively adhering to the letter of the law, when the practical effects are already well known and studied. One is not obligated to ignore literature, nor abstain from doing a simple extrapolation from the incentives placed on the table.

      There is a large body of literature concerning the question "does disparate-impact enforcement cause employers to alter hiring behavior in ways unrelated to actual productivity or discrimination?" and the answer is largely "yes". As you suggested elsewhere in this discussion, Google may be useful.

      • SiempreViernes

        today at 9:20 PM

        That's not particularly surprising nor objectionable, of course legislation that reminds employers they shouldn't discriminate based on race changes practice even for companies that aren't actually caught doing it.

        To act like it's bad that people of colour have a more fair chance of getting employed because of some piece of legislation is simply insidious. It's just been over a month since black people lost the right to a fair vote.

        • rayiner

          today at 9:42 PM

          > It's just been over a month since black people lost the right to a fair vote.

          Literally the opposite happened. The Supreme Court ruled that there was VRA §2 liability when there was evidence of racially-motivated gerrymandering: "In short, §2 imposes liability only when the evidence supports a strong inference that the State intentionally drew its districts to afford minority voters less opportunity because of their race." (Louisiana v. Callais, p. 26)

      • runako

        today at 9:15 PM

        > selectively adhering to the letter of the law

        Are you suggesting that companies should violate the law here? What do you recommend?

        Edit: charitably, "adhering to the letter of the law" is sometimes shortened to "law-abiding" and is generally what we want.

• logicchains

  today at 8:35 PM

  >What could justify it?

  The assumption that applicants from all races are on average equally qualified for every position. Whole subfields of modern academia are based on that assumption.

  • aenis

    today at 9:58 PM

    I am wondering - if in those circles, questions such as 'is NBA intentionally discriminating against asians - or is the fact that long distance running is dominated by, say, Ethiopians an example of discrimination' are ever discussed - or declared taboo and racist? I don't doubt that the assumption is just plain, demonstrably wrong - we all evolved under different types of environmental pressures - I am just wondering if the proponents of the all-the-races-are-same-on-average are ever discussing those obvious facts, and what answers do they come up with to explain the, say, unfair underrepresentation of Japanese in the NBA.

  • 59percentmore

    today at 9:35 PM

    The assumption is that no one has the authority to decide that all races aren't equally qualified for every position.

    • xp84

      today at 10:04 PM

      "Races" aren't qualified for anything. Neither are star signs or favorite Hogwarts houses.

      Individuals are qualified or unqualified. If a company happens to end up with less than 1/4 Ravenclaws or not very many Virgos, it doesn't mean hate is a reason. It could be that the Ravenclaws that applied were a bit less qualified than those from the other houses.

      I guess my point is, doing the statistical analysis for race and gender and drawing conclusions, while being completely blind to the one single factor any sane hiring manager should be focusing on -- actual qualifications for the role -- doesn't make any sense.

  • sdellis

    today at 9:29 PM

    Unless you believe that Black people are racially inferior, I think this is simply evidence of racial discrimination at a systemic level, from education through employment. AI merely reenforces the systems built to favor white people.

• moate

  today at 8:33 PM

  It's a starting point to flag.

  Here's some analysis of what it is and why it's useful as a canary in the coal mine: https://www.prevuehr.com/resources/insights/adverse-impact-a...

  • dash2

    today at 8:40 PM

    Thanks. I read the article:

    > Since the 80% test does not involve probability distributions to determine whether the disparity is a “beyond chance” occurrence, it is usually not regarded as a definitive test for adverse impact. Instead, other statistically significance tests, such as the standard deviation analysis, may be used for this purpose.

    But then my question recurs: isn’t this a ridiculous way to measure discrimination? It’s assuming that the only thing that differs between the different ethnic applicant pools is their ethnicity, which is essentially never going to be true.

    • gacgacgac

      today at 8:44 PM

      It's not used to measure discrimination. It's used to identify outcomes that appear to be potentially discriminatory. You have to do the legwork afterwards.

      Like. If I am evaluating a developer on lines of code written, I am a bad manager. But if an engineer has 40% fewer lines of code than the team median, it's absolutely ok for me to go, "Interesting. What's the story there? Are they slower or is there some other factor?"

      Same idea -- this is purely a fast, first pass metric that can quickly assess if something warrants a deeper evaluation.

      • blharr

        today at 9:45 PM

        You are correct, but especially in current day that analogy is quite bad.

        I expect Median LoC might be very high with the average developer using AI these days... but the dev who is making atomic changes that are fixing the AI output is probably tiny LoC but way more important

    • moate

      today at 8:46 PM

      How would you like me to define "starting point" in a way that you believe you'll be able to understand?

      If you are trying to say "more data needed, headline misleading" you should say that instead of misrepresenting the 4/5ths rule. Also the word "can" implies uncertainty of conclusion. This isn't ridiculous, the authors point out that this is the first large scale study of this topic. Nothing has been "proven" here, it's showing that this warrants further investigation and attention.

      Do you read many academic papers, because you seem to be having a rough go here.

      • kolbe

        today at 9:25 PM

        You could be an Iranian sponsored bot. I'm not saying you are. You could be so don't get mad at me for publishing that statement. Because if I say "can," then I don't need to be accountable for any misinformation.

• poplarsol

  today at 9:09 PM

  The desire to subsidize employment for Democratic constituencies by threatening legal action if they aren't given enough jobs.

  • throwaway62844

    today at 9:18 PM

    [flagged]

Genuine curiosity: Is there any speculation as to what these tools are keying on to reject those particular applicants? It seems like it just being the applicant's name is too easy an answer, but I could be overthinking it.

Interesting timing as Workday is facing Discrimination Claims in California doing the same thing.

https://www.yahoo.com/news/us/articles/california-judge-upho...

Ayres, I., Banaji, M. and Jolls, C. (2015), Race effects on eBay. The RAND Journal of Economics, 46: 891-917. https://doi.org/10.1111/1756-2171.12115

"Cards held by African-American sellers sold for approximately 20% ($0.90) less than cards held by Caucasian sellers, and the race effect was more pronounced in sales of minority player cards."

I truly don't doubt it's possible for the AI to be 'racist'.

>If the AI had recommended Black and Asian candidates at the same rate as it recommended the most-favored group (typically white applicants), 40,000 more of their applications would have advanced to the next stage of hiring.

I don't think this is the right benchmark here, or at least, it would be very interesting if the actual outcome, offer or rejected, was considered at the end.

• gacgacgac

  today at 8:53 PM

  You are misreading this sentence. This sentence is saying: "Using a constructed dataset of resumes, whose only difference was a name change, we would anticipate a system evaluating on qualifications to produce an equal distribution of candidates across names. Our observed result was highly unequal, and that warrants further investigation."

  • _0ffh

    today at 9:33 PM

    To me it appears as if the study using the constructed dataset was a completely different one than the one that was concerned with AI.

    For the AI study real data from "3.4 million people who submit 4 million job applications to 1,700 job postings across 150 employers and 11 industry sectors" was used.

There is no isolation of variables. This is not science. This is propaganda.

The paper is here: https://arxiv.org/pdf/2605.27371

They find "disparate impact" of pymetrics across racial groups, but it doesn't seem like they controlled for anything.

• efavdb

  today at 9:39 PM

  They also say that if they do the analysis globally the effect goes away. Curious, does that not imply that if one domain is biased against some group there would be another where the bias was in its favor?

Some job application websites I've seen actually have a yes or no option to consent to AI review that they claim is to simply assist HR and not actually screen you. I always select no. There is no way that selecting yes would ever be in my interest. I'm sorry, I'm going to force a real human to look at my stuff if I still can.

• bluefirebrand

  today at 8:15 PM

  My fear is that pressing "no" on stuff like that is going to become an auto-rejection in the vast majority of cases

  • simpaticoder

    today at 8:23 PM

    It won't be rejected. Your resume will be meticulously placed into a human review queue pending the allocation of someone to look at the contents. Meanwhile the position will be filled, and so serving no purpose the review queue will be emptied.

    • bluefirebrand

      today at 8:26 PM

      Oddly enough, being rejected by process versus being rejected by a person doesn't actually make me feel any better about the coming future

      :)

  • jcims

    today at 8:30 PM

    It's probably not going to be an auto-rejection, it's just going to sit in a queue that looks like this

        Screened Applications [13]
        Unscreened Applications [39148]

• booleandilemma

  today at 8:33 PM

  [dead]

I think the discrimination aspect is downstream from this fact:

> We follow 3.4 million people who submit 4 million job applications to 1,700 job postings across 150 employers and 11 industry sectors. Each job application was assessed by an AI hiring tool built by a single third-party vendor.

3.4 million people applying to just 150 employers... Who are all using just 1 platform. WTF. This is where the discrimination is happening. Why the f do 3.4 million people feel forced to apply to just 150 employers and why the f do all these 150 employers feel forced to use just one platform. WTF.

2 days ago there was another interesting article on the effects of AI in hiring[1]

I guess this one just compounds.

[1] https://news.ycombinator.com/item?id=48620142

I don’t think AI screening is effective. But this study is just disparate impact.

I am not surprised.

AI works by learning patterns. So it will become bias by just learning from factors like education history, schools attended, employment history, ZIP codes, or geographic location. Those 3 factors alone are an easy proxy for race.

And if you add names into the equation (if the AI was trained without removing applicant names), the model can become even more bias.

Would be very interested to see how this affects post-50 workers. That's a protected class and I would imagine an ambulance chasing lawyer would be excited for a class action lawsuit.

I expected more information from the article and 'the paper' -

I see nothing that shows any system was making a decision on race. How is the race being presented to the AI?

All this is showing from what I can see, is that certain groups of people were more often denied a next step in the process - but why?

Was the AI going by spelling and grammar? Were there names that were different but the rest of the resume was exactly the same? Were there pictures?

There were mentions that the rate of each group may be more prominent in the data when you split apart different types of jobs instead of all jobs in aggregate.. One could read that like it's inferred; that more warehouse jobs are offered to a race and less admin jobs.. but that same would happen if AI was more focused on perfect grammar for one job and it was not as much of a factor for a warehouse job.

Also if the people applying for the various jobs were self selecting, acceptance percentages this would skew things based upon which ones were applied / not applied to right?

There are so many ways you could draw conclusions like this from data, however correlation is not causation, yet this seems to say it is.

I feel this is an important thing to watch, but Stanford may not be the place to trust with 'Policy Recommendations' as it's very unclear there is any proof that 'AI Hiring Tools Yield Racial Bias and Systemic Rejection' from this study and paper.

PS - now that I see the HN title did not have the word "can" in it, and the title of the article is actually "Tools Can Yield" - maybe that is less accusing and more noting.

A racially disparate outcome is not evidence of racial bias.

• TheMagicHorsey

  today at 9:39 PM

  Imagine if they applied this same logic to the NBA draft.

The Pymetrics game is rigged by design:

Only 40% self report gender/race

no resume data, no education information, degrees, schools, GPA, major, work experience, skills/certifications

Zero job qualifications

• zerocrates

  today at 9:35 PM

  Well, they're only looking at whether the pymetrics gameplay algorithm ML thing recommends the candidate, not any of that other stuff. The outcome they're looking at here isn't whether the people actually got hired, or got passed by other screening layers or anything.

This study only looks at one specific vendor algorithmn (a job assesment given by a company called pymetrics)

• all2

  today at 8:29 PM

  LLMs are trained on the Internet, which isn't exactly known for it's race agnostic opinions.

> Using our large dataset of real hiring AI recommendations, we test our hypothesis. We find that people who submit multiple applications to positions screened by the same algorithmic hiring vendor are more likely to be rejected from every position to which they apply than would be true if the companies made decisions statistically independently from one another.

I would be surprised if the results were different.

I’m sure (really sure) there are real problems with AI and bias, but this is a weird study that isn’t looking at resumes or anything, it’s looking at how candidates did in some weird psychometric tests.

• gacgacgac

  today at 9:03 PM

  Double check the link. The study clearly looked at resumes.

  • petesergeant

    today at 9:21 PM

    I’ve rechecked it, and I still think I’m right. What am I missing? This is the paper under discussion: https://arxiv.org/pdf/2605.27371

I'm struggling to figure out what they're trying to say here in the linked (and very anemic) paper:

> 30% of Black applicants apply to at least one position that demonstrates adverse impact against Black applicants.

The whole thing reads like a tautology.

• gacgacgac

  today at 8:59 PM

  You are reading a paper without understanding the language of the paper. Adverse Impact has a specific meaning, and in this case it's specifically meaning that Black candidates were selected only four fifths as often as white candidates when their qualifications were identical. The study is only suggesting that further investigation is warranted.

> To put this in perspective: If the AI had recommended Black and Asian candidates at the same rate as it recommended the most-favored group (typically white applicants)

Some people just can't help but put their biases on display at every opportunity, even when it comes to the most minute details.

• gacgacgac

  today at 9:02 PM

  Nothing in this has any bias in it? Which words are you suggesting are biased? This study measured constructed resumes where only names were changed, and observed the rate each group was favored (the percentage of resumes that passed). One group must be "most favored" because thats how math works. It's the group whose percentage was the highest. The resumes were fictional and equivalent across race, only the names were changed.

  • bakugo

    today at 9:33 PM

    Look closer at the capitalization of the words in the quoted sentence.

• moate

  today at 8:41 PM

  Where do you think this sentence shows bias?

  The phrase "most-favored" means, "most recommended by the AI relative to the field".

  What did you think this sentence meant?

  • redsocksfan45

    today at 8:48 PM

    [dead]

You don’t need a complicated study to find out, do it yourself for science. Get a resume, make few different versions but keep the context the same, change the layout (one time education on top other on bottom etc etc), and use different names to signal different backgrounds, and you can extend it to schools too and gender, and send it to the same employers, you will see wonders!!

I tried it before, and discrimination is there, I would get one resume rejected quickly and few days later the same company would invite another resume for a screening call. I tried this before and after AI hype, results weren’t that different btw, and that was tested in US and Canada employers only.

Could the AI actually see the race of the applicants? Or was it just discriminating on the basis of some factor it found that was correlated with race, like SAT scores?

• runako

  today at 9:19 PM

  > discriminating on the basis of some factor it found that was correlated with race, like SAT scores

  Hypothetical SAT score: 1060

  How does that help you predict the race of an individual applicant? It's been a while since I took the SAT, but I didn't realize one's score provided so much information.

• foolserrandboy

  today at 8:56 PM

  It rejected Asians more because of their higher SAT scores? If it’s not directly based on applicants disclosing their ethnicity then probably something more obvious like names.

• gacgacgac

  today at 8:57 PM

  Name. Other factors were controlled.

  • today at 9:58 PM

  • moate

    today at 9:01 PM

    Where was this listed in the study? I can't find this anywhere in either the linked page or the Github https://algorithmichiring.github.io/

• moate

  today at 8:58 PM

  I'm going to assume that people aren't allowed to put "don't send me black applicants" into their process even if they do see race in the application as that's entirely illegal.

  The paper's conclusion, that we need to study this more, is showing the authors likely believe this to be a byproduct of inherent/invisible bias.

Its fucking crazy that people are using these systems for important tasks like hiring. They have zero understanding about how these systems work. And LLMs are absolutely not designed to do those sorts of jobs, they're designed to be chatbots and to fool a human conversing them that they are responding intelligently. Of course they're gonna be useless at other tasks.

(I assume they're just using a big LLM for this, it doesnt say, it just says "AI" when they say "AI like that they usually mean LLM".. A custom trained hiring ML system would be better)

• engineer_22

  today at 8:26 PM

  Isn't HR basically just an LLM with ears and teeth?

[dead]

[dead]

[dead]

[dead]

[flagged]

• techblueberry

  today at 9:08 PM

  > These results are consistent with AI hiring tools being completely racially unbiased, and real-world hiring managers feeling social pressure to hire underqualified black people

  And so managers are feeling social pressure to hire under qualified Asians as well? I must not be up to date on the latest culture war talking points, because I thought Asians were underrepresented.

  • JuniperMesos

    today at 9:22 PM

    Yeah, if they themselves are asian. One of the most prevalent complaints about Indian hiring managers in the silicon valley tech industry is that they preferentially hire Indians and push out non-Indians to a tremendous degree, and are often helping Indian hires commit pretty blatant credential fraud.

• today at 9:06 PM

• dzonga

  today at 9:08 PM

  does your anecdote comprise of the various instances when CVs were discriminated against cz people's names sounded black ?

  but you want to spew nonsense. every racial group includes its own under-qualified people ! there's no social pressure i.e DEI excuse you wanna give - but just economic agents acting for their own interests

[flagged]

Many people seem to think racism begins and ends with using a slur. You can usually get a measure of this by seeing someone's reaction to the statement:

> There is no such thing as anti-white racism.

If you find yourself wanting to disagree with that then, I'm sorry but you simply don't know what racism is. Racism is pervasive, insidious and systemic.

A good example in the hiring space is what's called the "second syllable name problem". Traditionally Afrcian names often stress the second syllable (eg Jamal, Lakisha, Malik, Lashonda). Studies have shown that such names have higher rejection rates in job applications [1]. So if you're wondering about the four-fifths rule, it's because it exposes this kind of bias. It's not proof of bias. It simply means further investigation is required.

The problem with AI hiring tools is the logic is opaque. You have no idea why an AI system is rejecting or selecting candidates and you may find it's doing something illegal. Some companies want to hide behind this opaqueness, arguing that if no explicit decision was made then there is no bias. But that's not how system racism works.

There are many such signals that correlate with race that if they affect selection rate, it could be a problem. Did you go to an HBCU? Was your high school in a minority-majority area? What about your previous employers?

This kind of bias doesn't have to be intentional.

[1]: https://www.npr.org/2024/04/11/1243713272/resume-bias-study-...

• kbelder

  today at 9:54 PM

  > > There is no such thing as anti-white racism.

  > If you find yourself wanting to disagree with that then, I'm sorry but you simply don't know what racism is.

  You are saying that if you think anti-white racism can exist, you don't know what racism is. That's obviously ludicrous.

• peyton

  today at 9:21 PM

  I’m sorry, this is catechism. Everyone deserves a fair shot, but you can’t expect the world to follow this liturgical logic.

We can't take blanket percentages as a reason for racial bias. Were they all equally qualified?

Too many of these studies only focus on percentages and the end result is unqualified candidates getting hired from minority groups at the expense of qualified ones.

• gacgacgac

  today at 8:56 PM

  Please read the study or at least the comments here before jumping to the conclusion. Yes, they used constructed resumes, so the qualifications were exactly the same. And no, literally no one is suggesting this proves racial discrimination. It's applying the four fifths rule, a fast, coarse evaluation that is used to identify if maybe theres worth investigating more for a conclusive evidence of racial discrimination.

  The authors are saying it's worth doing more research, because in a controlled data set the results appear unbalanced.

  • Oras

    today at 9:05 PM

    > Please read the study or at least the comments here before jumping to the conclusion. Yes, they used constructed resumes

    Looks like you didn't read the paper. There are no resumes involved. It is about assessment games.

This is something I've been working on exposing to AI labs through my startup LatentEvals[1], and found similar results in other industries from lending to insurance claims.

Happy to share some sample reports if anyone is interested!

1. https://www.latentevals.com/

• etchalon

  today at 8:04 PM

  Don't have much to add beyond being grateful for everyone working to call this out, with a hope some lawsuits drop and our SCOTUS doesn't decide racial bias in AI is fine because we can't prove the AI is racist in its heart.