There is a kind of mirror at old carnivals that does not show you as you are. It stretches you, narrows you, gives you a stranger’s proportions. You laugh, because you can see that it is wrong, and you can step away.
Your datafolk stands in front of a different mirror. This one does not show the present at all. It shows a prediction, what you are likely to do next, who you are likely to become, whether you are likely to repay, reoffend, recover, resign. And here is the part that stops being funny: when an institution trusts the mirror more than it trusts you, you cannot step away. The reflection makes the decisions.
From Describing You to Betting on You
In the last chapter, the bazaar sorted your datafolk into labels so it could be sold. This chapter is about what happens when someone stops merely describing your datafolk and starts wagering on it.
The tool is a predictive model A system trained on the past to make guesses about the future. It does not understand why anything happens; it only knows what tended to follow what. . You have met its outputs many times. The bank that approved your card in nine seconds did not read your life story; it ran your application through a model. The lending app that gave your cousin a loan and refused your neighbour used one too. Your credit score A single number meant to summarise whether you repay what you borrow. In India, your CIBIL score. is the famous example: a single number, produced by a model, that decides the interest rate on your home, the limit on your card, and sometimes whether a landlord will rent to you.
A predictive model is not magic and it is not malicious. It is a very patient student of the past. You show it a few million old loans, who repaid and who defaulted, and it learns which patterns tended to end in repayment. Then you show it you, and it bets. The bet is often good. Models like these have brought credit to millions of Indians who had no formal financial history at all, which is a genuine good and worth saying plainly.
But notice what the model is actually doing. It does not see you. It sees people who looked like you, and assumes you will do what they did. Your datafolk is being judged not on its own actions but on the company it statistically keeps. Most of the time you never feel this. You feel it sharply on the day the model is wrong about you, and there is no human in the room to argue with.
The Mirror Inherits Our Worst Habits
Where does a model’s opinion of “people who look like you” come from? From its training data The pile of past examples a model learns from. If the pile is biased, the model is biased — faithfully and at scale. , which is to say, from us. From our history. And our history is not a neutral record.
This is the second hard lesson of the mirror: bias is not a bug the model invents. It is a pattern the model learns from the world we already built. Cathy O’Neil, whose book sits on this book’s own shelf of inspirations, called these systems “weapons of math destruction”, models that are opaque, large-scale, and quietly punishing, precisely because they launder old prejudice through new mathematics and hand it back wearing the clean clothes of objectivity.1
The starkest demonstration came from the criminal justice system in the United States, where a tool used to predict whether a defendant would reoffend was found to flag Black defendants as high-risk far more often than white defendants who went on to commit the same crimes, or fewer.2 Nobody had typed “be unfair” into the model. The unfairness was in the data the model was raised on. It looked in the mirror of the past and faithfully reproduced its distortions, then dressed them up as a risk score a judge could cite.
And you cannot escape this by simply refusing to answer the sensitive questions. A model forbidden from asking your religion or your caste can often reconstruct them from a proxy variable A stand-in. A model that can't ask your caste can often guess it from your name, your pin code, or your contacts. Remove the question and the proxy answers it anyway. , your surname, your locality, the names in your phone. “We don’t collect that” is comforting and frequently beside the point. The mirror finds another angle.
When the Reflection Rewrites the Room
The most unsettling property of the predicting mirror is that it does not always just observe the future. Sometimes it causes it.
Consider predictive policing, increasingly trialled in Indian cities through CCTV networks, facial recognition, and “crime mapping”. A model predicts that a certain neighbourhood is high-risk. More police are sent there. More policing produces more recorded stops and arrests in that neighbourhood, not necessarily because more crime happens there, but because that is where the watching happens. The new arrest data flows back into the model, which now reports, with even greater confidence, that the neighbourhood is high-risk. Send more police. Repeat.
This is a feedback loop When a prediction changes the world in a way that makes the prediction look correct. , and it is the mirror at its most dangerous: a prediction that changes reality until reality agrees with the prediction. The reflection reaches out and redraws the room. A person born in the “high-risk” pin code inherits a datafolk that was flagged before they did anything at all. The same shape appears in hiring tools that learn to favour the kind of candidate a company already hired, and in lending models that thin out credit for whole localities, each one quietly making its own forecast come true.
You Have a Right to Not Be Only a Number
There is an Indian counterweight to all of this, and it is worth knowing about.
In 2017, the Supreme Court of India ruled, unanimously, that privacy is a fundamental right under the Constitution.3 The judgment is sprawling, but one thread runs straight to this chapter: the Court recognised that the harm is not only being watched. It is being sorted, profiled, and predicted, having decisions made about you by systems that reduce you to a pattern. The right to privacy, in that reading, includes a right not to be permanently defined by your datafolk’s reflection.
That principle is slowly working its way into law, including the data-protection regime we will reach in Chapter 6. It will not switch off the mirrors. But it establishes something the models do not grant on their own: that you are a person before you are a prediction, and that a distorted reflection is not allowed to be the last word about a real life.
Why This Matters
The bazaar wanted to sell your attention. The mirror wants to decide your access, to credit, to a job, to bail, to a fair price, and it wants to decide it in advance, from a reflection assembled out of other people’s pasts. When the mirror is right, you never notice; the loan simply arrives. When it is wrong, the cost lands entirely on you, and the system offers no one to appeal to, because “the model said so” is treated as if it were a fact rather than a bet.
So the literacy this chapter asks for is small and stubborn: when a system tells you what you are, remember that it is telling you what it predicted, from data about other people, through a mirror that inherited our history. That does not make every prediction wrong. It makes every prediction arguable. And knowing a thing can be argued with is the difference between a verdict and a guess.
In the next chapter, the trouble changes shape again. So far your datafolk has been sorted and predicted by systems that were, at least, supposed to hold it. Next, it gets loose, leaked, copied, stolen, and forged, and discovers the one rule that governs all of them: data does not forget.
Next: When Datafolk Are Copied, where your datafolk escapes its boundaries, turns up in places it should not, and occasionally pretends to be you.