How Bias Gets Built Into AI Systems
Bias is made through a chain of choices
When an AI system produces an answer that feels unfair, it is tempting to look for one bad line of code. Usually the story is longer. Somebody chose the problem to solve, the records to collect, the examples to label, the result to reward, and the place where the tool would be used. Every choice can shape what the system learns and whose experience it recognizes.
This page goes deeper into the machinery behind the video. It is about the routes by which bias can enter an AI system, even when the team building it says it wants a fair result. Understanding those routes helps us ask better questions. It also makes “the algorithm decided” a much less satisfying excuse.
Bias does not mean every AI output is wrong or that no one benefits. These systems can help with drafting, discovery, translation and accessibility. The point is to check whether the promised benefit arrives for the people affected, under the conditions in which the tool is actually used.
First, someone defines success
Imagine a company wants an AI system to identify good job applicants. What is a “good” applicant? If the team uses past hiring decisions as its answer, the model may learn the preferences and blind spots of earlier decision makers. If it uses retention, it may reward the circumstances that helped some people stay and punish those who lacked support. The target can be flawed before a model is trained.
A search system has the same problem in another form. If success means clicks, it can favour attention-grabbing results over careful answers. If it means a fast response, it may neglect the time someone needs to understand a complicated issue. The number chosen for optimization is a proxy for a human goal, not the goal itself.
Write down the intended benefit and the possible harms before building. Who gets a better result? Who could lose an opportunity, receive an inaccurate label, or be unable to use the interface? NIST's AI Risk Management Framework calls this kind of contextual work part of mapping risk. That step belongs at the start, not after a complaint.
Data reflects the world it came from
Training records are not a neutral snapshot of everyone. They reflect who was measured, who had access to the service, what questions were asked, and what was recorded. A group that was excluded from a process may be missing from its data. A person who used an alternative route may look like an outlier because the standard route was the only one counted.
Historical records can contain real inequity. Copying the records faithfully can reproduce the inequity. A data set may also be incomplete in quieter ways: one region, language, age group or communication style dominates. Adding more examples from that same source does not automatically solve the gap.
Ask where the data came from, what period it covers, what consent and privacy conditions apply, and whose experience is absent. Documentation should describe limitations in words a product team can use. A large number of rows is not evidence that the right people and situations are represented.
Labels carry human judgment
Many systems learn from examples marked with labels such as “helpful,” “professional,” “high quality” or “safe.” Those words can hide disagreement. One reviewer may interpret a short, direct message as efficient; another may call it rude. If their judgment becomes a training label, the model can learn a social preference as if it were a fact.
This matters when the label affects a person’s prospects. Someone whose communication is literal or unconventional should not be marked less capable because a reviewer expected a particular tone. The label may also come from a past outcome that had nothing to do with ability. A person who did not complete an inaccessible form did not demonstrate a lack of interest.
Document how labels were defined, who applied them, how disagreements were resolved, and whether the definition still fits the intended use. Test the system on examples where a reasonable person might disagree. An uncertain case should not be disguised as an objective score.
Features and proxies can reintroduce a pattern
Removing an explicit field does not remove everything associated with it. A model may use other features that correlate with the information a team intended to exclude. Location, past access to a service, gaps in a record, or a particular writing style can become proxies in some settings. The relationship is contextual; it cannot be assumed from the name of a column alone.
This is why a simple rule such as “we did not give the model disability data” does not prove that disability-related patterns cannot affect an outcome. At the same time, collecting sensitive data without a clear purpose raises privacy concerns. Teams need a lawful, careful way to assess impact and protect the people whose data is involved.
NIST's publication on identifying and managing AI bias describes sources of bias across the AI lifecycle. The practical lesson is to investigate the system and its context, not search for one forbidden variable and declare victory.
Evaluation can miss the people who matter
An overall accuracy figure can look impressive while performance varies in the situations that carry the most risk. If the test set resembles the training set, both may leave out the same people. If the team tests a model alone, it may miss the form, workflow or explanation that prevents a person from using the result.
Measure the actual task. Could people submit information? Did the system return an appropriate answer? Could they tell when it was uncertain? Could they correct an error? How much work did a failure create? Averages can conceal a small group facing repeated problems. Examine relevant subgroups where it is appropriate and privacy-respecting, and look at qualitative feedback as well as numbers.
A model may also change over time, or the organization may change how it uses the output. Testing once before launch is not enough. NIST's framework includes measuring and managing risk through the system's life. Record incidents, review new use cases, and know who can pause the feature.
Deployment changes the meaning of an answer
The same model output can be a low-stakes suggestion in one setting and a high-stakes gatekeeper in another. A writing assistant that offers several possible phrasings is different from a screening tool that silently rejects an application. The interface can make an uncertain estimate look like an instruction. A score beside a person's name may carry more authority than its evidence deserves.
Ask who sees the output and what they do with it. Does a human reviewer have the time and authority to disagree? Is there a clear path to challenge a decision? Can the person affected see the information that needs correcting? Human oversight is meaningful only when it changes what happens after an error.
Organizations can also create pressure to trust the machine because it is quick. A rushed reviewer may accept a recommendation that should have been investigated. Training and accountability must match the real workflow, not the ideal one described in a policy document.
Generative AI has its own traps
A generative model produces plausible language, which can make a stereotype sound polished. It can omit a caveat in a summary, invent a citation, or describe a person through a narrow assumption. A user may never see the material the answer was based on. The risk depends on the prompt, model, safeguards and setting; do not assume every output will have the same flaw.
Test varied prompts and contexts, including direct and unconventional language. Look at whether refusals, safety messages or rewriting suggestions treat people differently. Compare an output with its source. When a claim matters, link to the original material and let a person verify it. A confident paragraph is not evidence.
For neurodivergent users, an assistant can reduce friction and provide a useful first draft. It can also repeatedly push a writer toward a supposedly normal voice. Give people control over edits and keep the original. Help should support the person's intent, not erase it.
What better practice looks like
Begin with a specific use case and its stakes. Decide what the tool should do, where it should defer, and what a person can do when it fails. Write down foreseeable harms and the evidence you will need to test them. Bring affected users into design and testing, pay them, and make it safe to report a problem.
Document the data and labeling decisions. Evaluate the full journey as well as the model. Check accessibility with people using different devices, input methods and communication styles. W3C's cognitive accessibility guidance is useful when the interface asks people to understand choices, remember steps or recover from errors.
After launch, monitor the outcomes that matter, not only the ones easiest to count. Give people a route to correction and an accountable human contact. If the system causes harm, change or stop its use. NIST describes the ongoing functions as govern, map, measure and manage. They are verbs because responsible work does not end when a model is released.
One useful internal check is to ask someone outside the product team to explain the result and how to appeal it. If that person cannot find the evidence or the correction route, the affected user is unlikely to manage it under pressure. The explanation and challenge process deserve the same testing as the model itself.
Questions people ask
Can more training data remove bias?
Sometimes better, more representative data helps. More of the same records can reinforce the same gaps. The goal, labels, evaluation and deployment also need scrutiny.
Is an AI system unbiased if it does not collect disability status?
No. Other features and historical patterns can still affect outcomes. Assess the actual impact while protecting privacy and avoiding unnecessary collection of sensitive information.
What does a fairness score tell me?
It tells you something about a particular measure on a particular test set. Ask which people, errors and consequences it leaves out, and whether the result holds in the real workflow.
Who is responsible when an AI recommendation is wrong?
The organization using the tool must provide accountability, review and correction. A model's output does not remove the need for a person to examine a consequential decision.
Keep exploring
For the effects on people, read How AI Bias Can Exclude Neurodivergent People. For the interface people must use, watch Designing Digital Spaces for Neurodivergent Individuals. And for the human side of relying on these tools, see How AI Can Affect Mental Health and Emotional Well-Being.
The useful question is not whether a system can appear objective. It is whether we can trace the decisions behind it, see where it fails, and change course when people pay the price.
Go deeper
Read Feeling Seen by a Machine online, or keep the complete edition on Kindle or in paperback.
Read the book