Machines Judging Humans — Frank Pasquale

About this talk

Speaker: Frank Pasquale
Event: HCIL UMD, 37th Human-Computer Interaction Lab Symposium, workshop “Human-Centered AI: Reliable, Safe & Trustworthy”
Date: 2020-05-28 (workshop date from the video description; uploaded 2020-07-06)
Length: 18:54
Channel: HCIL UMD
Watch: Frank Pasquale - Machines Judging Humans: The Promise and Perils of Formalizing Evaluative Criteria

How to read this page. It has two kinds of content, always marked:

• 🎙 In the talk. What Pasquale says, with [MM:SS] timestamps. Only these parts are his views.
• 📚 Deeper theory. Background from the traditions he draws on, including ones he does not name. It explains why he thinks what he thinks and who disagrees. It is not his claim unless a timestamp says so.

Suggested route. Summary → Map → Key terms → Part 2: The intellectual map (read once, slowly) → Deep dives in order → “Putting it together”.

Note on dates in Part 2. Dates, titles and attributions in Part 2 and in the 📚 blocks are from memory unless a link is given. They are standard references, but they were not individually checked.

---

Speakers

Label	Who	Role
Pasquale	Frank Pasquale	Law professor. Says he is at the University of Maryland and “moving to Brooklyn Law School as of July 1st” [00:00]. Only speaker; no Q&A on the recording. Confidence: high (named in the video details; he gives his own Twitter handle and book)

Caption warnings. Auto-captions, so names and terms are often wrong. Corrections used on this page:

• “Kara Bridges” [13:09] = Khiara Bridges (author of The Poverty of Privacy Rights; from memory).
• “Good Heart’s law” [07:26], [09:02] = Goodhart’s law.
• “Royal Melbourne Institute of Technology” is RMIT. “Primarily separated Australia” [00:00] = “primarily based in Australia”. The team is almost certainly the ARC Centre of Excellence for Automated Decision-Making and Society (from memory; not checked).
• “paper the news I’m she’s running” [00:00] = The Markup, the news site Julia Angwin runs.
• “Christian Johnson and Jennifer Smith” [14:46] = most likely Kristin Johnson, Frank Pasquale and Jennifer Chapman, “Artificial Intelligence, Machine Learning, and Bias in Finance: Toward Responsible Innovation” (Fordham Law Review, 2019). From memory; not checked.
• “Eric Corbett” [13:09], the Parkinson’s-from-mouse-movement paper: the name looks wrong. I did not identify the paper; treat the attribution as unknown.
• “Viljoen and Green” [16:23] = Salomé Viljoen and Ben Green, “Algorithmic Realism” (FAT* 2020) (listing). Caption also writes “green and Viljoen”.
• “Manheim and Garrabrant” [07:26] is correct: arXiv 1803.04585 (paper).
• “Accurateism” and “Accuratism” are both in the captions. He says Accuratism [13:09].
• Garbled spots: “historicist or formalist” [01:39] is probably “positivist or formalist” (my reading; not sure). “Normals” [03:14] = “norms”. “Luper about having human centered AI” [03:14] is unreadable. “Thin tech firm” [11:34] = “thin-file” or “fintech firm”. “go for the former and suppose the latter” [13:09] may be “supplant” (my guess). The chapter times in the video description (e.g. 02:46, 03:24) differ slightly from the transcript timestamps used here.

Factual slips by the speaker.

1. “The Goodhart line” [09:02]: “when a measure becomes a target it ceases to be a good measure” is the standard slogan, but it is Marilyn Strathern’s 1997 rewording, not Charles Goodhart’s own 1975 sentence. Goodhart’s original is about monetary-policy statistical regularities collapsing once used for control (from memory; not checked). Common shorthand, not a real error.
2. “The Markup … today” [00:00]: the tenant-screening exposé is real and dated 28 May 2020, the workshop day. But it was a joint project of The Markup and the New York Times, reported by Lauren Kirchner and Matthew Goldstein, not by Angwin herself (The Markup, “Locked Out”). He only says it was published by The Markup, which is fair.
3. “Heartsaver … about seven years ago” [05:51]: ProPublica’s game was from about 2013 (from memory; not checked). Consistent.
4. The law-school joke [05:51] (“second best in the country because it has the second most students”) is told without a name. I could not check it.
5. “Indian Twitter” example [11:34]: he credits Privacy International with the first two examples (political tweets in India; typing style). I could not find the India-political-tweets claim in the Privacy International sources searched. They document Lenddo’s use of social-network and phone data (PI on Lenddo). Treat that specific example as unverified.
6. He says “four things” for the legal-process conception [03:14]. The list that follows in the captions has four elements, so no slip, but the captions garble them. See Deep dive 1.

---

Summary

Central question. When a machine scores or judges a person (for a hospital ranking, a loan, a flat), what makes that judgment fair? And is the usual answer, “make the rules formal and explicit”, the right answer?

Main thesis (his words, condensed). There have been waves of algorithmic accountability.

1. Wave one says: make decision systems fairer by using better rules, more indicators, more data. This has a long pedigree in law and economics: make decision-makers “algorithmic” so they can be held to account.
2. Wave two says: the scoring system itself needs to be held to account, and sometimes the right move is to question scoring altogether, not refine it.

The two waves complement each other and also pull against each other.

He builds the argument on two cases:

• Medicine. Hospital ratings. Different raters share “few common scores”. Ratings can accelerate the closure of hospitals in poorer boroughs. Quantitative targets get gamed (Campbell, Goodhart), and risk adjustment breeds consultants who game the adjustment.
• Finance. Fintech lending. A 35% APR loan that ruins a household still counts as a success for a model that only sees repayment. More data does not fix it. His label for the bad outcomes: predatory, creepy and subordinating inclusion.

Close. Formal criteria “cannot often accommodate” rival narratives about what a score is (fair allocation by risk, or machine-made stigma). He cites Viljoen and Green’s algorithmic realism: AI should go through what law went through with legal realism in the early 20th century and admit it needs social science, economics, philosophy and discretion. He frames narrow AI as intelligence augmentation, not replacement.

Stance. Sympathetic to formalisation as one tool, not the only one. Pro-human-in-the-loop, but a human with social-science expertise. Pro-regulation of inputs (ban some data uses, cap interest, limit data). He explicitly refuses to say which narrative about credit scoring is true; he wants both held in mind [16:23].

---

Map — talking points

Times are the transcript’s segment stamps (the video description’s chapter stamps differ by up to ~1 minute).

#	Point	Time
1	Why this topic: his book New Laws of Robotics; the RMIT automated-decision-making programme; The Markup’s tenant-screening story. Usual legal remedies: explanation, human in the loop, review	[00:00]
2	Thesis: waves of algorithmic accountability, and their tension. AI as intelligence augmentation	[01:39]
3	Do formal rules lead to fairness? Depends on your conception of the rule of law	[01:39]–[03:14]
4	The third conception: legal process (procedural fairness, reasonableness, link to authority, judicial review)	[03:14]
5	Legal process is expensive in money and in power. Hence the turn to scoring	[04:15]
6	Example 1: hospital ratings replace inspectors. Raters disagree. Kernighan’s college-ranking joke	[04:15]–[05:51]
7	Second wave: question scoring in general. ProPublica’s Heartsaver game	[05:51]–[07:26]
8	Hospital closures, stratification, pandemic surge. Campbell’s law, Goodhart’s law	[07:26]–[09:02]
9	Gaming and risk adjustment; the Raters and the Rated chart; surveillance and manipulation	[09:02]
10	Example 2: fintech lending. The 35% APR “success”	[10:37]
11	More data? Thin-file credit, phone data resold, trade secrecy, “a new deal for data”	[11:34]
12	Three odd data sources: political tweets, form-filling style, Parkinson’s mouse tremors. Explanation vs plausible story	[11:34]–[13:09]
13	Accuratism. Bridges and Eubanks on data and the poor	[13:09]–[14:46]
14	Predatory, creepy, subordinating inclusion. Remedies: usury law, data limits, rules on credit. Impact on “inexplicable AI”	[14:46]
15	Rival narratives of credit scoring	[16:23]
16	Algorithmic realism (Viljoen and Green), from legal realism	[16:23]–[17:57]
17	Close: rethink metrics, data collection, discretion; substitution vs complementarity of AI; the book	[17:57]

---

Part 1 — Before you read

Key terms

• Algorithmic accountability. Holding automated decision systems, or the people who run them, answerable. Pasquale uses it in two directions: (a) making human decision-makers accountable by forcing them to follow explicit, rule-like procedures; (b) making the algorithms and their owners accountable. The word “algorithm” changes between the two. In (a) it means “a clear rule”; in (b) it means “a software system”.
• Rule of law. Watch out: it means different things in different traditions.

  Conception	Core demand	Typical source
  Formal (“thin”)	Rules are clear, public, prior and applied evenly	Dicey; Raz; Weber’s “formal rationality”
  Substantive (“thick”)	The rules must also be just, protect rights	Dworkin; Bingham
  Legal process	Fair procedures, reasoned decisions, independent review	Hart & Sacks; Fuller (partly)
  Pasquale calls the formal view “a rule conceived of as a clear prescription that exists prior to its application” [01:39] and prefers the third. See Part 2.1.

• Legal process. A mid-20th-century American school of legal thought (Henry Hart and Albert Sacks) that asks which institution should decide what, and by which procedure. Here it is not “any process”; it is a theory of institutional competence and reasoned decision.
• Scoring. Reducing a person or institution to one number or rank, made by a model from data. Hospital ratings, credit scores, tenant-screen reports and college rankings are all scores.
• Campbell’s law. Donald Campbell (1976, as Pasquale says: “the more any quantitative social indicator issues for social decision making are more subject to corruption pressures”). The more an indicator is used to decide, the more it is gamed, and the more it distorts what it measures.
• Goodhart’s law. Charles Goodhart (1975) on monetary policy, popularised as “when a measure becomes a target it ceases to be a good measure”. Manheim and Garrabrant (2018) split it into four mechanisms: regressional, extremal, causal and adversarial (arXiv).
• Risk adjustment. Correcting outcome statistics for how sick the patients are, so a hospital that treats harder cases is not punished. Needed for fairness; also a lever for gaming.
• Excess mortality. How many more patients die than the risk-adjusted average predicts. A standard outcome score for hospitals.
• Predatory / creepy / subordinating inclusion. His three labels for bad forms of “financial inclusion” [14:46]. Predatory: you get credit on terms that damage you. Creepy: you get credit by being watched. Subordinating: you get credit by staying quiet or staying in your place.
• Accuratism. His coinage for the ideology that the answer to bias is always more data [13:09].
• Intelligence augmentation (IA). Computing as a partner that extends human ability, the line from Licklider and Engelbart. Opposite pole: AI as substitute for humans.
• Algorithmic realism / legal realism. Legal realism (1920s–30s US) held that judges do not just apply rules; law is shaped by social facts, so law must use social science. Algorithmic realism is Viljoen and Green’s proposal that algorithm design make the same turn.
• Lead generation. Online advertising that finds more people who look like past customers the model counted as successes. In his loan example, it spreads the model’s idea of success [10:37].

People and works named (a reading list)

Person / work	Why they come up	Start with
Frank Pasquale	The speaker	New Laws of Robotics (Harvard UP, 2020); The Black Box Society (2015)
Julia Angwin / The Markup	Tenant-screening exposé published that day	Locked Out
RMIT automated decision-making programme	Pasquale is on the research team	Programme site (not looked up)
Brian Kernighan	Wrote a satirical piece on college rankings in the Daily Princetonian	Piece not located
ProPublica, Heartsaver	Game showing heart-attack response gaps	Not located
Donald Campbell	Campbell’s law	”Assessing the impact of planned social change” (1976)
Charles Goodhart	Goodhart’s law	Goodhart (1975)
Manheim and Garrabrant	Variants of Goodhart’s law	arXiv 1803.04585
Khiara Bridges	Poor people and data/privacy	The Poverty of Privacy Rights (2017)
Virginia Eubanks	Automated systems that punish the poor	Automating Inequality (2018)
Privacy International	Reported on odd credit data sources	Fintech report (2017)
Salomé Viljoen and Ben Green	”Algorithmic realism”	FAT* 2020 paper
Licklider	The workshop’s IA lineage	”Man-Computer Symbiosis” (1960)
Johnson, Pasquale, Chapman	Co-authored article on AI in finance (if captions are read correctly)	Fordham Law Review 2019

Pasquale’s own frame

Pasquale’s work sits in the “second wave” he names. His earlier books are The Black Box Society (2015), which he cites here for trade secrecy [11:34], and his 2014 Scored Society article with Danielle Citron on credit scoring (from memory; not checked). This talk is a short bridge from those to New Laws of Robotics, which he says is “coming out this fall” [00:00]. The book’s argument, as I recall it, is that AI should complement and not substitute for professionals, and it sets out four new “laws” (from memory; not checked). He is a legal scholar addressing computer scientists, and says so up front: “my talk may be a bit of a change of pace” [00:00]. That is why he starts from law’s own history.

The current debate (2026)

This talk is from May 2020. These are the live threads it connects to today. Search results are cited; anything from my own memory is marked.

• Tenant screening and housing. The Markup’s Locked Out series (28 May 2020) found automated screening reports often pull in wrong or mismatched records, and landlords deny on that basis (The Markup; Digital Government Hub summary). A later Senate Banking chair asked the CFPB to review the industry, citing the series (The Markup follow-up). Pasquale names this as the human-in-the-loop problem in the wild.
• Europe now gives scores a legal hook. In the SCHUFA case (C-634/21, 7 Dec 2023) the CJEU held that a credit agency’s score can itself be an “automated decision” under GDPR Article 22, even if the bank makes the final call; the case went back to the German court on how far explanation and human intervention go against trade secrecy (MediaLaws; Mondaq). This is the ruling closest to the structural point he makes about the scorer and the decider. The EU AI Act classes credit-scoring systems as high-risk (Annex III) (Il Sole 24 Ore; CEPS, 2025). One practitioner source gives 2 Dec 2027 for the high-risk obligations (DEV Community); I recall that the timeline may have been pushed back again, from memory; not checked. A right to explanation of individual decisions under the Act (Article 86) is reported in secondary sources, but I could not confirm the adopted wording, so check the Official Journal text.
• US: explanations for AI credit denials. The CFPB’s 2022 and 2023 circulars said a creditor using complex algorithms still owes the applicant specific reasons, and that checklists and boilerplate do not satisfy ECOA (American Banker; National Law Review). From memory; not checked: in 2025 the CFPB withdrew a large batch of guidance, and I believe these circulars were in it. Do not rely on the circulars as current law without checking.
• Goodhart goes mainstream in AI. Manheim and Garrabrant’s taxonomy (arXiv) is now used in AI-safety work on reward hacking and over-optimisation. A recent arXiv paper argues for principled limits on optimising general-purpose AI against proxies (arXiv 2510.02840; title seen in search results, not read). This extends his point that gaming is not only a social-science problem.
• Algorithmic realism lives on. Ben Green’s later work carries the “algorithmic realism” frame and contrasts it with “algorithmic formalism” (author page; second-hand description). From memory; not checked: Green also published a critique of laws that require human oversight of algorithms, arguing that people are poor at the oversight role. That bears directly on Pasquale’s “human in the loop”.
• Not searched. Pasquale’s own post-2020 position, and any critique of his two-wave framing.

---

Part 2 — The intellectual map

Pasquale spoke for under 19 minutes to computer scientists and drew on a few fields very quickly. This section lays out the traditions he is standing in, including thinkers he does not name. Nine families of ideas meet in the talk (the last, rating societies, sits outside the diagram):

 THEORIES OF RULE OF LAW ─────┐                    ┌──── SOCIOLOGY OF METRICS
 Dicey · Weber · Fuller       │                    │     Campbell · Goodhart · Strathern
 Hart & Sacks · Raz · Dworkin │                    │     Porter · Espeland & Sauder · Muller
                              ▼                    ▼
 LEGAL REALISM ─────►  ┌─────────────────────────────────┐  ◄───── HEALTH-CARE QUALITY
 Holmes · Pound ·      │  PASQUALE: when machines judge  │         Donabedian · risk adjustment
 Llewellyn · Cohen     │  humans, formal criteria are    │         hospital ratings · ACOs
                       │  not enough: they hide rival    │
 CREDIT, FINANCE ────► │  narratives and social context  │  ◄───── DATA, POVERTY, SURVEILLANCE
 ECOA/FCRA · FICO ·    └─────────────────────────────────┘         Gandy · Bridges · Eubanks
 predatory inclusion          ▲                   ▲
                              │                   │
              DUE PROCESS & EXPLANATION      AUGMENTATION vs AUTOMATION
              Goldberg · Mathews · Citron    Wiener · Licklider · Engelbart
              GDPR Art. 22 · Rudin · Green   Brynjolfsson · Autor

2.1 Theories of the rule of law — formal, substantive, procedural

The problem it addresses. “Rule of law” is praised by everyone and means different things. Does it mean rules (clear, public, general), good rules (just rules), or good process (hearing, reasons, review)? Whether you believe automated scoring is “lawful” depends on which.

The chain of thinkers

1. A. V. Dicey, Introduction to the Study of the Law of the Constitution (1885). The classic British statement: no one is punished except for a distinct breach of law established in the ordinary courts; everyone is equal before the law. A formal, rule-centred view. Key point for Pasquale: this is the root of the “clear prescription prior to application” ideal he describes at [01:39].
2. Max Weber, Economy and Society (1922). “Formal rationality” in law: decisions follow general rules applied to facts, like a machine (“a slot machine” is his image, as I recall it; from memory). Key point: Weber saw the pull of calculable law for capitalism and the tension with substantive justice. Automated scoring is the end of that line: calculable in the extreme.
3. Lon Fuller, The Morality of Law (1964). Eight principles of legality (generality, publicity, non-retroactivity, clarity, consistency, possibility of compliance, constancy, congruence of official action with the rule). Fuller thought these are also an “inner morality” of law, so form carries some substance. Key point: Fuller’s list is a very good checklist for an automated system. Many scoring systems fail “publicity” (trade secrets) and “congruence” (the rule on paper is not the model in use).
4. Henry Hart and Albert Sacks, The Legal Process (mimeographed 1958, published 1994). Law as a system of institutions with different competences (courts, legislatures, agencies). Decisions should be reasoned, and each decision should be made by the institution best placed to make it. This is the “legal process” school Pasquale prefers. Key point: this tradition values reasoned elaboration and review, not only clear rules. It is the school that treats procedure as a good in itself.
5. Joseph Raz, “The Rule of Law and Its Virtue” (1977), and Ronald Dworkin, A Matter of Principle (1985). The formal-vs-substantive debate. Raz: rule of law is formal and not a measure of justice (a knife can be sharp and used for murder). Dworkin: rights-based, substantive.
6. Due-process cases. Goldberg v. Kelly (1970): welfare benefits cannot be cut off without a prior hearing. Mathews v. Eldridge (1976): the balancing test weighing the private interest, the risk of error and the government’s cost. Key point: Mathews explicitly makes process a matter of cost. Pasquale’s observation that “legal process is expensive” [04:15] is a description of the same trade-off the Court wrote into doctrine.

The thread that ties them together. Formalists say fairness is predictability and equal treatment; process theorists say fairness is a hearing and a reason. Automation delivers the first cheaply and removes the second. Pasquale’s claim is that cost pressure drives the shift from the second to the first.

2.2 Legal realism — from rules to social facts

The problem it addresses. If law is a set of rules applied to facts, why do judges decide similar cases differently, and why do legal results track social interests? Realism says the rules do not decide the case; judgement and social context do.

The chain of thinkers

1. Christopher Langdell (Harvard, 1870s). Law as a science: derive outcomes from a small set of principles by logic. The target of the later realists. Key point: this is the “autonomy” fantasy Pasquale describes at [16:23] when he says law once claimed “we have rules … we don’t need social science”.
2. Oliver Wendell Holmes Jr., “The Path of the Law” (1897). The “bad man” theory: law is what courts will do in fact. “The life of the law has not been logic: it has been experience” (The Common Law, 1881). Key point: law is a prediction of official behaviour. Close to what a scoring model does.
3. Roscoe Pound, “Law in Books and Law in Action” (1910). Gap between the written rule and the working rule. Key point: the same gap appears in algorithms: the policy document vs the deployed model.
4. Karl Llewellyn, Jerome Frank, Felix Cohen (1920s–30s). Llewellyn’s “paper rules” vs “real rules”; Frank’s Law and the Modern Mind (1930), on judges’ hunches; Cohen’s “Transcendental Nonsense and the Functional Approach” (1935), attacking legal concepts that are used as if they explained outcomes. Their programme: bring in economics, sociology, psychology. Key point: Pasquale says “law needs economics, law needs a lot of social science” [16:23], which is the realist slogan.
5. Louis Brandeis, the “Brandeis brief” (Muller v. Oregon, 1908). A legal argument built on empirical social data. An early instance of social science entering law.
6. Viljoen and Green, “Algorithmic Realism” (FAT* 2020). The talk’s source. They argue that algorithm design tends to “formalism” (the problem is stated mathematically, the maths is solved, the social context is somebody else’s job) and should adopt the realist posture. Details are from the talk and the paper listing only; I did not read the full text.

The thread. Realism does not abandon rules; it declines to treat them as self-applying. Pasquale borrows the historical analogy: law tried autonomy, failed, and learned to live with other disciplines; AI will probably repeat this.

A caution. The analogy cuts both ways. Realism also fed “law and economics”, which in turn pushed legal decisions toward cost-benefit formulae, the very trend Pasquale worries about in wave one. Realism is a call for more kinds of expertise, not for a single better one.

2.3 Sociology and economics of metrics — Campbell, Goodhart and the reactivity of measures

The problem it addresses. A measure is made to represent something. Once it is used to reward or punish, people change what they do to improve the number, and the number stops representing what it was for.

The chain of thinkers

1. Charles Goodhart (1975), on monetary policy: statistical regularities collapse when used for control. Robert Lucas (1976) made a close point for macroeconomics: relationships estimated under one policy regime break when the policy changes. Key point: the shared idea is that a model of a system fails when the system reacts to the model.
2. Donald Campbell (1976). The social-science form, which Pasquale quotes [07:26]: the more an indicator is used for social decisions, the more it invites “corruption pressures”. He used teacher-testing as his case.
3. Marilyn Strathern (1997), “‘Improving ratings’: audit in the British university system”: the slogan “when a measure becomes a target, it ceases to be a good measure”. This is the phrasing Pasquale uses.
4. Theodore Porter, Trust in Numbers (1995). Quantification wins in public life because it replaces trust in persons with a rule that looks impersonal. Numbers are a technology of distance. Key point: this explains why officials prefer scores to inspectors [04:15].
5. Wendy Espeland and Michael Sauder, “Rankings and Reactivity” (2007). Law-school rankings changed how law schools behave. People react to being measured. Key point: Pasquale’s college-ranking jokes [05:51] are the lighter side of this finding.
6. Jerry Muller, The Tyranny of Metrics (2018). A popular synthesis: “metric fixation”, the belief that all performance can be measured and that rewards should follow numbers.
7. Ian Hacking, “looping effects of human kinds” (1990s). Classifying people changes them, which changes the classification. Key point: a credit score is a classification that people respond to; this loop is built into the subject.
8. Manheim and Garrabrant, “Categorizing Variants of Goodhart’s Law” (2018). Four mechanisms: regressional (selecting on a noisy proxy also selects noise), extremal (push the metric to an extreme where the link to the goal fails), causal (intervening on the proxy does not move the goal), adversarial (others exploit the proxy) (arXiv). Their paper treats Campbell’s law as a related formulation.

The thread. Metrics fail in different ways, and “fix the metric” works against only some. Regressional failures can be handled with better statistics. Adversarial ones start an arms race, which is what he describes with risk adjustment and consultants [09:02]. This is also why the “wave one” fix, add more indicators, can feed wave two’s problem.

2.4 Health-care quality measurement — from inspection to outcome scores

The problem it addresses. How do you know if a hospital is good? Early methods were visits and accreditation. Modern ones are outcome and cost statistics.

The chain

1. Ernest Codman’s “End Result System” (c. 1910s). A Boston surgeon who tracked patients’ outcomes by hospital and published them. He was driven out of his position at Massachusetts General. An early outcome-measurement pioneer (from memory; not checked).
2. Avedis Donabedian, “Evaluating the Quality of Medical Care” (1966). Structure, process, outcome: three things you can measure. Key point: each has a different gaming profile. Outcomes are what patients care about but are noisy and confounded by case mix.
3. Risk adjustment (1980s onward). Hospitals treating sicker patients have worse raw outcomes, so statistics adjust. Medicare’s hospital mortality lists in the 1980s were an early public example (from memory). Key point (his): risk adjustment is needed for fairness and also creates a target for gaming [09:02].
4. Public ranking and reporting. Commercial lists such as the one he mentions (“best hospitals NS news” [05:51], i.e. U.S. News) and public star ratings. A Health Affairs paper, cited by him, found national hospital rating systems “share few common scores”.
5. Payment reform and accountable care organisations (ACOs). US reforms pay providers for population-level cost and quality. He says he will give the ACO example in questions [09:02]; it does not appear in the recording. Key point: ACOs show the same loop: pay linked to a number, then risk selection (“cream-skimming”), upcoding and measurement games (from memory).
6. Place and closure. Hospital closures concentrate in low-income and minority areas, and rural areas. His chosen example is New York, where Queens and the Bronx lost hospitals [07:26] (the facts of the closures are his claim; not checked).

The thread. Medicine is the cleanest case of the wave-one/wave-two split. Wave one asks: are the ratings valid, risk-adjusted, comparable? Wave two asks: who does the rating serve, and what does it do to access? A rating that closes a hospital produces an effect (a longer ambulance ride) the rating never measured.

2.5 Credit, scoring and “inclusion” — law, history, critique

The problem it addresses. Credit decides who can buy a home, start a business or get through an emergency. Lenders always needed to judge risk. The question is how, and with what data, and on what terms.

The chain

1. From character to score. Credit assessment moved from local reputation and “character” to bureaus (1900s), then to statistical scores. FICO’s general-purpose score dates from 1989 (from memory). Marion Fourcade and Kieran Healy, “Classification Situations” (2013), show how scores sort people into “lendable” and “unlendable” groups and make that difference feel natural.
2. Statutes. Fair Credit Reporting Act (1970; access and correction rights). Equal Credit Opportunity Act (1974; bans discrimination by race, sex and other protected classes, and requires reasons for denial). Fair Housing Act (1968). Key point: the ECOA reasons requirement is the oldest “explanation right” in US law, and it is what the CFPB circulars (see current debate) apply to machine-learning models.
3. Predatory inclusion. Louise Seamster and Raphaël Charron-Chénier (2017) coined predatory inclusion: groups once excluded from credit are now included on worse terms. Keeanga-Yamahtta Taylor’s Race for Profit (2019) applies it to housing. Key point: this is the source of the category Pasquale calls “predatory inclusion” [14:46], and the reason “more inclusion” is not automatically good. The Taylor and Seamster attributions are from memory; not checked.
4. Financial inclusion as policy. Development banks and fintech argue that people with “thin files” should be brought into credit via alternative data (phone use, social graph, bill payments). Mehrsa Baradaran’s How the Other Half Banks (2015) is the critical history of a two-tier banking system.
5. Alternative data in practice. Companies such as Lenddo used social-network and phone data in scoring (PI on Lenddo). Privacy International’s 2017 fintech report covers the sector’s data practices, using Kenya as a main case (PI report). Key point: this is the evidence behind his “creepy” category.
6. Legal critique. Citron and Pasquale, “The Scored Society” (2014); Pasquale, The Black Box Society (2015); Barocas and Selbst, “Big Data’s Disparate Impact” (2016); Cathy O’Neil, Weapons of Math Destruction (2016); Hurley and Adebayo, “Credit Scoring in the Era of Big Data” (2016). All argue that scoring can launder bias and hide behind secrecy (from memory; not checked).
7. Anti-classification vs anti-subordination. Jack Balkin and Reva Siegel (2003) contrast equality law as banning classification by protected traits with equality law as preventing group subordination. Key point: Pasquale’s complaint that using political activity as a signal is “politically subordinating” [11:34] is the anti-subordination argument, not a bias-by-proxy argument. A model could be accurate and still be subordinating. This is a different test from “is it biased?“.

The thread. In the lender’s loss function, a 35% loan that is repaid in full is a win. Social cost is outside the objective. The remedies he lists (usury law, data limits, rules on credit) change the objective or the inputs from outside, which no tuning inside the model can do.

2.6 Data, poverty and surveillance — the case against “more data”

The problem it addresses. Statistical fairness work often says: collect more data and the model will be less biased. Critics note that data collection is not neutral for the people collected on.

The chain

1. Oscar Gandy, The Panoptic Sort (1993). Early account of how personal data is used to sort people into categories of opportunity and harm.
2. Foucault on discipline and James Scott on legibility (Seeing Like a State, 1998). States and firms make people “legible” in order to govern. Legibility has benefits (services) and costs (control). Key point: the tension Pasquale draws between data that “could be taken as very useful” and data used for “surveillance and control” [09:02].
3. Khiara Bridges, The Poverty of Privacy Rights (2017). Poor mothers in the US (especially those on Medicaid) face intrusive questioning and tracking that wealthier women avoid; privacy law protects those who can afford its protection. Key point: supports his reading of data collection about the poor as punishment.
4. Virginia Eubanks, Automating Inequality (2018). Case studies of automated welfare eligibility, homeless-services matching and child-welfare risk scoring. Her thesis: automated tools “profile, police and punish” the poor (from memory; not checked).
5. Helen Nissenbaum, contextual integrity (2004, 2010). Data flows have norms tied to their original context. Taking phone data given for one purpose and reselling it for another breaks the context. Key point: this is the usual theory behind his objection to resale in the “thin-file” bargain [11:34].
6. Pasquale’s “Accuratism”. His name for the belief that bias is a data-shortage problem [13:09]. The critique: when people are being sorted for exploitation, a more accurate sort is not better.

The thread. Wave one treats data as an input to be improved. This tradition treats data collection as an act with a target, and asks who benefits.

2.7 Due process, explanation and the “human in the loop”

The problem it addresses. When an automated system affects you, what do you get: an explanation, a person, an appeal?

The chain

1. Danielle Citron, “Technological Due Process” (2008). Automated systems that decide benefits need notice, hearing and reasoned decisions built in. Key point: ties the process tradition of 2.1 to software.
2. GDPR Article 22 (2016/2018) and the “right to explanation” debate. Wachter, Mittelstadt and Floridi (2017) argued the GDPR does not give a strong right to a specific explanation; Selbst and Powles (2017) argued for a “meaningful information” reading. The CJEU’s SCHUFA ruling (2023) strengthened the Article 22 reach to scores (MediaLaws).
3. Kroll et al., “Accountable Algorithms” (2017). Technical tools (logging, cryptographic commitments, verifiability) for proving that a decision procedure was applied as claimed.
4. Cynthia Rudin, “Stop explaining black box machine learning models for high stakes decisions” (2019). Post-hoc explanations are often unfaithful; use inherently interpretable models. Key point: this is the technical side of Pasquale’s question whether an explanation is of “the system itself” or merely “plausible” [13:09].
5. Human oversight and automation bias. Research on people over-trusting machine output (Skitka and colleagues, 1999, from memory) and, more recently, work arguing that laws requiring humans in the loop assume a capacity that people often lack (Ben Green; Crootof, Kaminski and Price, Humans in the Loop, 2023; both from memory; not checked). Key point: why Pasquale’s phrase “persons who have some sociological, social scientific, other forms of expertise” matters: he asks not just for a human, but for a particular kind of one.
6. Trade secrecy. Companies defend scoring models as proprietary. The talk’s remark that data practices are hidden “through the trade secrecy” [11:34] ties to The Black Box Society. The SCHUFA remand weighs explanation against trade secrets (see the current debate).

The thread. Wave one’s remedy, “give a reason, add a person”, is real but thin. His point at [00:00]: those remedies are hard to deliver when the owner of the algorithm is legally protected from scrutiny.

2.8 Augmentation vs automation

The problem it addresses. Should computing replace human work, or extend it? The answer shapes design.

The chain

1. Norbert Wiener, The Human Use of Human Beings (1950). Warned about machines used to displace human judgement, and about who controls them.
2. J. C. R. Licklider, “Man-Computer Symbiosis” (1960). Partnership between human and computer, each doing what it does best. Pasquale names Licklider [01:39], having heard him cited in earlier talks.
3. Douglas Engelbart, “Augmenting Human Intellect” (1962). Computing as a toolkit for human thinking. The origin of much of HCI, and of the host lab’s tradition.
4. Michael Jordan, “Artificial Intelligence: The Revolution Hasn’t Happened Yet” (2018) and the “IA” label. Argues for treating most ML as engineering that augments humans (from memory; not checked).
5. Labour economics. Erik Brynjolfsson, “The Turing Trap” (2022), warns that building AI to match humans pushes toward substitution and concentrates power. David Autor’s work on tasks and “new work” argues for complementary design (both from memory; not checked).

The thread. Pasquale’s last slide shows works on the left that look at AI as a substitute and on the right those that look at working together [17:57]. His point is that IA is also a political position about who keeps authority, the same power question he raised about legal process at [04:15].

2.9 Rating societies — reputation, platforms and the reciprocal gaze

The problem it addresses. His “Raters and the Rated” chart [09:02] shows patients, doctors, hospitals and insurers all rating one another. Rating has spread from a few experts (bond-rating agencies, inspectors) to everyone, through platforms. Who governs the ratings, and what do they do to the rated?

The chain

1. Credit-rating agencies and the “rating” as an institution. Moody’s (1909) and similar firms sold judgement about creditworthiness to investors. Thomas Sinclair, The New Masters of Capital (2005), argues the agencies became private governors of public and corporate finance. Key point: ratings are a form of power even when no law says so (from memory; not checked).
2. Auditing and the “audit society”. Michael Power, The Audit Society (1997): the spread of audit as a ritual of verification, which produces auditable surfaces more than good practice. Strathern’s Goodhart line (2.3) comes from the same debate.
3. Reputation platforms. Online ratings for drivers, hosts, doctors and landlords turn reputation into a number that travels. Studies of gig platforms show ratings used to discipline workers by the platform and by customers, with little recourse (from memory). Pasquale and Citron’s “reputation society” worries in The Scored Society are in this line.
4. Manipulation and fakery. Review fraud, inflated view counts and bought followers. His YouTube example [09:02] is typical. Key point: when a rating drives money, an industry grows to produce ratings, which is the adversarial Goodhart pattern in market form.
5. Surveillance by rating. Ratings give a firm or state a cheap way to monitor many people without watching them. China’s social-credit schemes are the stock example, but US tenant screening and employment background checks are the nearer case; the Markup’s 2020 story on tenant screening is one documented instance (The Markup).
6. Two-sided effects. Reciprocal rating can also check the powerful: patients rating hospitals, tenants rating landlords. Key point: this is why he says the same data “could be taken as very useful” and also be used for “surveillance and control” [09:02]. The direction of the gaze, who can rate whom, and who can appeal, decides which it is.

The thread. Ratings are not only measurements; they are relationships of power. Wave one treats a rating as an instrument to be calibrated. Wave two asks who holds the instrument, who can contest it, and who gets rated back.

How the traditions meet (one picture)

 Cost pressure on process  ──►  Scores replace inspectors/judges       (2.1, 2.4)
          │                              │
          ▼                              ▼
 Wave one: fix the score        Metrics react: Campbell/Goodhart       (2.3)
 (more indicators, data)  ───►  gaming, risk adjustment arms race
          │                              │
          ▼                              ▼
 Data hunger ("Accuratism")  ──► predatory / creepy / subordinating      (2.5, 2.6)
          │
          ▼
 Wave two: question scoring itself, bring in social science             (2.2)
          │
          ├──► remedies from outside the model: usury law, data limits, bans
          └──► human with expertise in the loop; augmentation, not substitution (2.7, 2.8)

---

Part 3 — Deep dives

1. Three conceptions of the rule of law, and why process is expensive

🎙 In the talk

• He opens with the remedies law usually offers: “an explanation, a human in the loop, or some forms of review” [00:00]. He says these are difficult “given other legal protections of entities that are in charge over and controlling the relevant algorithms” (in his words, trade secrecy and similar protection).
• He frames wave one: for twenty years there has been a movement “to make legal systems more algorithmic” as a way to make decision-makers more accountable if they have to describe what they do as based on rules [01:39].
• Question: “do formal rules lead to fairness? it may depend on your conception of the rule of law” [01:39]. The formal view treats law as “a rule conceived of as a clear prescription that exists prior to its application” [01:39].
• The third conception, legal process, needs “procedural fairness”, reasonableness, a link to prior sources of authority, and “judicial review as a guarantor of procedural fairness” [03:14]. He calls it “a much more complicated and I think richer concept” [03:14].
• The cost problem: “legal process is expensive” [04:15], both in money (independent decision makers, appeals) and in power. A single controlling entity can “really force multiply the nature of their authority by coding it” [04:15]. Distributed expertise is “a sacrifice of some power unless you have them completely under control” [04:15].
• Result: “a turn to simpler algorithms, complex data processing and scoring” [04:15].

📚 Deeper theory

• The formal conception is Dicey and Weber; the process conception is Hart and Sacks (Part 2.1). The talk’s tidy contrast hides the Raz–Dworkin dispute (formal vs substantive), which is a different axis from rule-vs-process.
• Mathews v. Eldridge already prices process: it asks how much error an extra procedure would remove and what it would cost the state. His point is a structural reading of the same trade-off, applied to AI: automation changes the price list.
• The “power” argument is a principal-agent argument. A hierarchy gets its will through rules it can encode, and loses control when it delegates to people with judgement. This is the old Weber–bureaucracy point, now with software. Software is a delegate with no discretion, which is why owners like it.
• Fuller’s eight principles (2.1) are a usable audit: publicity and congruence fail most often in scoring systems.

Where others push back

• Formalists: explicit rules are the protection against arbitrary power. A human with “reasonableness” can be biased too, and bias at least can be measured in a rule. Algorithms make discretion visible.
• Economists of law: process has to be costed. If process is too expensive, fewer decisions get reviewed at all, and cheap automation could extend decisions to people who were never served.
• Technologists: systems can be built with appeal channels, logs and contestation designs; automation is not necessarily the end of process.

What they leave out

• He does not say what share of automated decisions would go to legal process if it were affordable. Much of the harm is in decisions that never reach a court.
• He does not discuss the cost side of the affected person: appeals fall on those least able to bring them.
• The “force multiply authority by coding it” claim is an assertion about motive. The cost-saving motive and the control motive are not separated.

Check yourself

1. Which of the three conceptions treats a hearing as essential, and why does automation tend to remove it?
2. What does “force multiply the nature of their authority by coding it” mean, and what has to be true of the organisation for this to be a gain?
3. How does Mathews v. Eldridge turn the “legal process is expensive” point into law?

2. Scoring replaces inspection, and the ratings disagree

🎙 In the talk

• Hospital quality used to be judged by inspectors visiting hospitals, “doing all sorts of qualitative and sometimes quantitative measurements” [04:15]. Pressure moved this to “just numerical outcome measures based on the data produced by the hospital”, for example “excess mortality rate”, risk-adjusted [04:15].
• Scoring “on one level seem to be more fair” than diverse inspectors with varying ideas of quality [05:51]. But “the rating systems themselves had varying conceptions” [05:51]. He cites a Health Affairs article: “national hospital rating systems share few common scores and may generate confusion” [05:51].
• The humorous cases: Brian Kernighan’s editorial that college rankings could be built to make almost any college number one; a law school that is bad on many metrics but “second best in the country because it has the second most students” [05:51].
• Wave one is “trying to make sure that these systems are fair and … that they incorporate more and better indicators” [05:51].

📚 Deeper theory

• Porter’s Trust in Numbers (2.3) explains the preference: numbers look impersonal and travel across distance, so they replace trust in inspectors’ judgement.
• Disagreement between raters is not noise only. Each rating encodes a different weighting of what “good” means: mortality, readmissions, patient experience, cost, volume. A composite is a political choice. Rating-design choices are close to what Espeland and Sauder study with law-school rankings.
• The “second most students” joke is a clean case of the extremal and adversarial Goodhart patterns: a metric chosen because it is easy to count (size) is promoted as quality (Manheim and Garrabrant).
• Donabedian’s structure/process/outcome (2.4) tells you why inspectors measured structure and process, and why moving to outcomes brings case-mix problems.

Where others push back

• Measurement advocates: inspectors are also inconsistent and capturable, and public outcome data has pushed some hospitals to improve. Imperfect scores can still beat none.
• Statisticians: disagreement between rating systems can be diagnostic. It shows the measures capture different dimensions, so publish the dimensions.
• Hospital-sector critics: the real problem is that ratings are used to pay and to market, not that they exist.

What they leave out

• No evidence is given that ratings changed patient outcomes in either direction; the argument rests on one Health Affairs piece and an anecdote.
• He does not discuss what inspectors’ judgement misses (for example, favouritism or poor reproducibility). The pro-inspector side is stated more kindly than the pro-score side.

Check yourself

1. Why is “scoring seems more fair than inspectors” true on one level and incomplete on another?
2. If two hospital rating systems share few common scores, what does that tell you about the construct they claim to measure?
3. Which Goodhart variant does the “second most students” ranking illustrate, and why?

3. The second wave: question scoring itself, and Goodhart at work

🎙 In the talk

• “there’s a second wave of algorithmic accountability … that questions the role of scoring the systems in general” [05:51].
• ProPublica’s Heartsaver game: get a heart-attack patient to a hospital in time. It was easier in wealthier parts of Manhattan than in parts of Queens and the Bronx “where a lot of hospitals been closed” [07:26].
• Ratings and closures: hospitals close partly because “they get a bad reputation” [07:26]. A market view says informed customers do better and bad hospitals should close. His other reading: “this seems to be reinforcing dynamics of stratification and exclusion” [07:26], and he links it to pandemic preparedness and the value of “distributed hospital systems” in a surge [07:26].
• The turn: “perhaps we need to go beyond trying to make ranking and rating systems fairer to thinking about what is the social context that they play into, what are the existing advantages and disadvantages that they either reinforce or challenge” [07:26].
• Campbell’s and Goodhart’s laws: “when a measure becomes a target it ceases to be a good measure” [09:02]. Risk adjustment fights gaming, but “the more you do risk adjustment the more you also spark the development of consultancies” [09:02] that find ways around it.
• His chart of “Raters and the Rated” [09:02]: patients, doctors, hospitals and insurers all rate each other. This data can help quality understanding; it “can be used for surveillance and control or it can also be manipulated” [09:02], YouTube view counts being the classic example.

📚 Deeper theory

• The wave-two move is a change of level. Wave one asks about the validity of the measure (an inside-the-model question). Wave two asks what the measure does in its setting (an outside question). This matches the realist move in 2.2.
• The market-informed-consumer story assumes that a closed hospital’s patients can go elsewhere. In emergency care geography binds (the Heartsaver point). Emergency care is the textbook case where “consumer choice” fails.
• Why the risk-adjustment arms race is hard to escape: it is adversarial Goodhart (arXiv). Every adjustment creates a new variable to optimise; whoever has the most resources to read the formula benefits.
• Hacking’s looping effect (2.3): rating a hospital changes the hospital and the patients who choose it, so the ratings drift.
• “Raters and the Rated” matches the move from one-way inspection to reciprocal rating systems (platforms). Reputation data is a new form of governance with little oversight.

Where others push back

• Market defenders: a closed hospital may have been unsafe, and keeping it open for access does not make it good. Underlying access gaps have other causes (payer mix, staffing).
• Epidemiologists: the causal link from ratings to closures is not shown in the talk. Closures have many drivers.
• Quality advocates: the answer to gaming is better audit, not dropping measurement.

What they leave out

• No data on how many closures follow from ratings. The Heartsaver game shows a distribution; it does not show cause.
• Little on what wave two does in practice. “Rethink metrics” is the answer given at the end. Wave two so far reads as critique more than as procedure.
• The ACO example he promised [09:02] is not on the recording.

Check yourself

1. What is the difference between asking “is this rating valid?” and asking “what does this rating do?”
2. Why is emergency care a weak fit for the “informed consumer” argument?
3. Give a mechanism, in your own words, by which risk adjustment can increase gaming.

4. Fintech lending: the 35% “success”, and the three kinds of bad inclusion

🎙 In the talk

• The question: are lending algorithms “capable of effectively compassing all of the social problems that might be sparked by certain forms of algorithm funding?” [10:37].
• The case: a 35 percent APR loan; the borrower may be “desperate to repay it, not buy food for their children, fall behind, but the lender is paid back in full” [10:37]. His sense: “the narrow AI driving a lot of FinTech would say this is a success” [10:37], and lead generation would then “find more people like that” [10:37].
• “More data” as the fix: the bargain of a thin-file borrower who lets a firm “download and analyze all information on their mobile phone and resell the data”, without knowing what is done with it, while “trade secrecy” blocks finding out [11:34]. He wants “a new deal for data and a better deal for data” [11:34].
• Three data uses (the first two he credits to Privacy International) [11:34]: lenders who see political activity on Twitter in India and judge repayment harder; a firm that looks at how you fill in a form; a Parkinson’s-from-mouse-tremor paper that would be “very troubling” if used in finance [13:09].
• His judgement: the political one is “politically subordinating and it should be outlawed” [11:34]. On the form-filling one: “I am a bit up in the air about that” [13:09].
• Explanation: is it “an explanation of the system itself of what it’s doing or are we looking for explanations of that are just plausible?” and “we actually have to go for the former and suppose the latter” [13:09] (see caption note).
• Accuratism: “an ideology that I’ve called Accuratism frames the answer to bias is always looking for more data” [13:09]. His response: “sometimes more data is not the answer” [13:09]. He cites Bridges and Eubanks [13:09].
• Summary: such inclusion is “predatory, creepy and subordinating” [14:46]. Examples of remedies: usury law (predatory), data limits (creepy), rules about credit (subordinating) [14:46].
• A consequence: “the more legal standards are there for the nature of financial evaluation, the more we effectively are hemming in the role of not explicable AI” [14:46]. For recognising these harms he wants “persons who have some sociological, social scientific, other forms of expertise in the loop” [14:46].

📚 Deeper theory

• Why the model calls it a success. The label is “repaid”. A lender’s loss function counts defaults and revenue, not household harm. High-interest loans repaid in full are among the most profitable. The mismatch is a case of Goodhart’s causal failure: the proxy (repayment) is not the goal (a borrower better off) (arXiv).
• Feedback. Lead generation trained on “successes” finds more people in the same trap: a self-reinforcing loop, as Hacking’s looping effects (2.3) would predict.
• Predatory inclusion has a history: Seamster and Charron-Chénier, and Taylor’s Race for Profit (2.5). It is the formal name for the pattern he describes.
• Anti-subordination. Banning a signal because it punishes political activity is an anti-subordination argument, not a bias-by-proxy one (Balkin and Siegel, 2.5). Note this is a principled limit on correlation: an accurate predictor is excluded on moral grounds.
• Contextual integrity (Nissenbaum, 2.6) is the theory behind the objection to resale of phone data.
• Explanation. The “plausible vs faithful” distinction is Rudin’s critique of post-hoc explanation (2.7). A plausible story (“people who type in all caps are excitable”) is the kind of thing post-hoc methods can generate without the model using it.
• Why legal standards limit “inexplicable AI”. If the law bans using a trait or its proxies, the lender must show the model does not use them, and a model it cannot explain cannot show that. This links to GDPR Article 22 and SCHUFA (MediaLaws).
• Current note: the Lenddo and Privacy International sources document social-graph and phone-data scoring (PI on Lenddo). I could not find the India political-tweets example in those sources.

Where others push back

• Fintech and inclusion advocates: people with thin files are excluded by old scoring; alternative data does let some get credit at lower rates. Evidence from Kenya and India on mobile lending is mixed (from memory; not checked). Cheap short-term credit has real demand.
• Economists: high APRs can reflect high default risk and high servicing costs; a usury cap can cut off the very borrowers it means to protect.
• Data-science view: the example is a choice of objective, not an AI problem. Change the label (for example, “repaid without missing other bills”) and the model changes. Nothing here needs “inexplicable AI” to be wrong.
• Privacy-minded liberals: a ban on some correlations is paternalistic if the borrower consents. His reply would be that consent is hollow when the data bargain is hidden [11:34].

What they leave out

• No evidence is given for how common the 35% case is, or whether any named lender uses the success label he describes. “My sense is” [10:37] is his own phrase for it.
• He does not say whether form-filling or typing-style signals are accurate, or how a regulator would test for it.
• Remedies are listed in one slide; trade-offs between them (a data limit may reduce credit access) are not discussed.
• Outside the US (India is named) he does not discuss local law.

Check yourself

1. Why would a model trained on repayment count the 35% loan as a success, and what would you change to fix it?
2. What is the difference between “creepy” and “subordinating”, and which remedy does each call for?
3. Why does he think legal limits on correlations would constrain “inexplicable AI”?

5. Rival narratives and algorithmic realism

🎙 In the talk

• The two narratives of credit scoring [16:23]. One: “a fair allocation of credit according to risk”, with a chance to redeem yourself and rebuild your score. The other: the system “builds upon subordination of the disadvantage rationalized by an algorithm and a machine for generating stigmatic harms that enable profit such as higher interest rates” [16:23].
• His stance: “I’m not saying one of these is true and one is false but … we’ve got to hold both narratives in mind as we deal with the policy for machines judging humans” [16:23].
• The problem: “formal evaluative criteria cannot often accommodate these rival narratives”, being “driven toward maximizing your satisfying metric and rule-based” [16:23].
• Algorithmic realism, from Viljoen and Green: “AI may need algorithmic realism the way law needed legal realism” [16:23]. Law in the early 20th century tried to show autonomy: “we have rules … we can order reality” and need no input from others. Since the 1920s legal scholarship has said “law needs economics, law needs a lot of social science” [16:23].
• If we see this history, “we should see a similar type of intellectual history within AI and algorithms that would bring in more of these points of view” [17:57]. That means “rethinking metrics, rethinking data collection, discretion” [17:57].
• Close: his works group into AI as substitute (left) and as working with humans (right), and he brings them together in the book [17:57].

📚 Deeper theory

• Rival narratives are what social scientists call contested constructs. A credit score “measures creditworthiness”, but what that means is disputed, and fixing a metric fixes the dispute in code. Compare the contested-concept literature (Gallie, 1956; from memory).
• This is why the realism analogy matters: realists did not choose a winner among doctrines; they widened the evidence admitted. Pasquale likewise wants multiple stories held at once, which an optimiser cannot do. A single objective function is by construction one narrative.
• Where the analogy is shaky. Legal realism changed legal education and doctrine over decades and still did not stop formalism returning (law-and-economics, cost-benefit analysis). Algorithm design has a faster feedback loop and stronger commercial incentives for closed formalism. I would not read the analogy as a prediction.
• Discretion. “Discretion to make her change rules” (caption garble, [17:57]) reads as a call to keep rule-changing discretion in human hands. Compare Davis’s Discretionary Justice (1969; from memory) on structuring, not eliminating, discretion.
• Link to Ben Green’s later work (see current debate): the realist stance and the doubt about human oversight sit together. Realism asks for the right kind of human expertise, not human presence.

Where others push back

• Engineers: we cannot optimise two narratives. A system needs one objective; you can run multiple models and compare, but the decision rule is one rule. Pluralism has to enter at the level of governance, not code.
• Neutral-metric defenders: credit scoring is a risk-rating exercise. The second narrative confuses the effects of poverty with the effects of scoring.
• Critics of realism: realism can slide into scepticism about any rule, and in law it was used to rationalise judicial policy-making.

What they leave out

• How to operate with two narratives: who decides, with what evidence, and what happens in the meantime to the person being scored.
• Any case where a score was right and the harm remained, versus a case where a different score would have fixed it. The two narratives imply different tests, but he does not give them.
• The talk ends at a call to dialogue between computer science and law, not at a design.

Check yourself

1. State each credit-scoring narrative in one sentence. What evidence would move you toward one?
2. What did legal realism change in law, and which parts of that story does Pasquale carry over to AI?
3. Why can’t a single objective function hold two narratives, and where could pluralism enter instead?

---

Putting it together

Mental model. Fairness in judging people has two supports: a rule that treats like cases alike, and a process that hears the person and gives reasons. Process costs money and gives up control, so organisations replace it with scores, which are cheap, comparable and obedient. Scores then change the behaviour they measure (Campbell, Goodhart), spread the lender’s or rater’s definition of “success” (lead generation, rankings), and sort access in ways that reinforce existing stratification (closed hospitals, predatory credit). Wave one answers by improving the score. Wave two asks what the score is for and who pays for it. The remedy he points to is not one better metric but a widening of who gets to define the metric: social scientists, affected people, regulators who can ban inputs (political activity, resold phone data), and a human in the loop who has the expertise to see what the score cannot. In his view, AI should augment people doing that work and not replace it.

Open tensions

• Wave one vs wave two. Formal rules do protect against arbitrary power. Pasquale says the two waves are complementary but gives no rule for when to switch from one to the other.
• More data vs less data. He criticises “Accuratism” but wants “a new deal for data and a better deal for data” [11:34]. That is more regulated data, not less data. It is unclear who enforces it.
• Accuracy vs principle. Banning political-activity signals gives up some accuracy for anti-subordination. He accepts this; lenders will price it.
• Explanation vs capability. If a legal standard needs the model to be explainable, the most accurate opaque models are ruled out. That is a cost he welcomes [14:46].
• Human in the loop. He wants experts in the loop, while evidence suggests human oversight often fails (see 2.7, from memory). What makes an expert-in-the-loop work is open.
• The analogy. Legal realism may not be a good forecast for AI (Deep dive 5).

Reading path

• Start: the talk (19 minutes), then Citron and Pasquale, “The Scored Society” (2014), then the Manheim and Garrabrant paper on Goodhart (arXiv).
• Then: Pasquale, New Laws of Robotics (2020); Green and Viljoen, “Algorithmic Realism” (listing); Porter, Trust in Numbers; Eubanks, Automating Inequality; the Markup’s Locked Out series (link).
• Deeper: Hart and Sacks, The Legal Process; Fuller, The Morality of Law; Espeland and Sauder, “Rankings and Reactivity”; Seamster and Charron-Chénier on predatory inclusion; Rudin (2019); the CJEU SCHUFA judgment and commentary (MediaLaws).
• Sibling pages (same speaker, other talks): pasquale-new-laws-of-robotics · pasquale-future-of-ai-in-law · pasquale-ai-changing-medical-practice.