New Laws of Robotics — Frank Pasquale in conversation

About this talk

Speaker: Frank Pasquale
Event: FUTURES Podcast #36 (interview about the book New Laws of Robotics)
Date: 2021-02-15 (upload date)
Length: 52:22

How to read this page. It has two kinds of content, always marked:

• 🎙 In the talk. What Pasquale or the host say, with [MM:SS] timestamps. Words in quotation marks are verbatim from the transcript. Only these parts are Pasquale’s views.
• 📚 Deeper theory. Background from the traditions behind his argument. It explains why he thinks what he thinks, and who disagrees. It is not his claim unless a timestamp says so. Unless a link is given, dates and attributions in 📚 blocks are from memory; not checked.

Suggested route. Summary, Map, Key terms, Part 2 (read once, slowly), then the deep dives in order, then “Putting it together”.

Sibling pages on Pasquale (written from other talks): pasquale-machines-judging-humans · pasquale-future-of-ai-in-law · pasquale-ai-changing-medical-practice. This page stands alone.

---

Speakers

Label	Who	Role
Host	Luke Robert Mason	Host of the FUTURES Podcast. Names himself in the intro and Pasquale says “Luke” at the end. He asks the questions and adds framing (data double, “inmates running the asylum”, corporations as artificial persons)
Pasquale	Frank Pasquale	Professor of Law, Brooklyn Law School (per the video description, 2021). Author of The Black Box Society (2015) and New Laws of Robotics: Defending Human Expertise in the Age of AI (Harvard UP, 2020). Main speaker

Confidence: high. Two speakers, both named in the audio and the video details. The transcript has no speaker labels of its own, but the turns are marked by the questions (“Mason:” and “Pasquale:” are in the text).

Caption warnings. This is an English auto-caption with an editorial pass, and a few errors remain. Corrections used on this page:

• “Becker, who is an economic sociologist” [13:02] is almost certainly Jens Beckert (Imagined Futures, 2016), the economic sociologist of “fictional expectations”. Same sentence: Deirdre McCloskey and Robert Shiller (Narrative Economics, 2019) are correct.
• “National Institute of Computer Assisted Journalism” [20:18] is the National Institute for Computer-Assisted Reporting (NICAR), run by Investigative Reporters and Editors (from memory; not checked).
• “cut the cue” [06:03] is cut the queue. “pass people’s faces” [35:55] is parse. “an infinite rigorous problem” [19:13] is an infinite regress. “decepticon of an attribution” [38:11] is likely deception of an attribution (my reading of the sentence).
• “work I’m doing now for a company called the Oxford Handbook of Expertise” [19:13]: it is a book project, not a company.
• “something called ‘Debt in the Silicon Savannah in Kenya’” [26:57]: probably the Boston Review essay “Perpetual Debt in the Silicon Savannah” (Donovan and Park, 2019; from memory; not checked).
• “TAILS” [40:45] is the host’s pun on the four laws plus “tails” as markers on an AI. It is not a term from the book.
• The intro says “healthcare educational warfare” [00:06]: read it as “healthcare, education and warfare”.
• The host says “data double”; Pasquale says “digital double”. They mean the same idea.

Factual slips and overstatements by the speaker (checked where a link is given, otherwise from memory; not checked):

1. “Unregulated bot speech” [38:57]. US robocalls are in fact regulated (the Telephone Consumer Protection Act of 1991 and later call-authentication rules). The real problem is weak enforcement and a loophole-ridden definition. His point (spam teaches people to ignore real calls) stands; the word “unregulated” is too strong.
2. The judges-and-lunch study [20:18]. He says it “showed or purported to show”, which is the right hedge. The 2011 Danziger et al. finding has been criticised as an artefact of how cases were ordered, so do not treat it as settled fact.
3. Volkswagen [44:54]. He says key actors were identified and “punish[ed]“. Some individuals were prosecuted, but the most senior ones largely avoided punishment for years. “Some” is fair; “accountability worked” would not be.
4. Tay [45:40]. Microsoft’s bot is framed as “machine learning online” that went racist through interaction. Tay was also exploited by a “repeat after me” feature. Minor, but it matters for attribution: the people who attacked it are a different category from the people who built it.
5. Ed Felten and the red recording light [40:45]. I could not find the source; take it as his recollection.
6. The China classroom system [09:54]. Described from a news story; I did not verify the companies or their claims.

---

Summary

Central question. Asimov’s laws say a robot must not hurt us. Pasquale asks a different question: how do we keep human control over technology that is deployed by firms and states in markets? His answer is four new laws, aimed at institutions as much as at machines.

Main thesis. AI and robotics should complement human expertise, not replace it, and the law should push technology in that direction. The four laws, as he states them:

1. AI and robotics should complement professionals, not replace them [02:47].
2. They should not counterfeit humanity [04:34].
3. They should not contribute to unproductive arms races [05:15].
4. Any robotic system should be attributable to a person or group of persons [06:03].

How the argument runs.

• The fear that robots make us obsolete is partly a story, spread by people who saw their own industry (journalism) collapse, and by managers who want to cut labour costs [11:57]–[13:02].
• The “digital double” model (record what a worker does, then replace them) fails where work needs improvisation, judgment and consultation [16:27]–[18:20].
• “Meta-experts” who claim to judge all other experts hit an infinite regress [19:13]. Better: experts and technologists work together, as with the judge’s-lunch example and data journalism [20:18]–[21:55].
• Cyber-libertarian “more data, freer contracts” is alienating. Microlending is the example [25:03]–[26:57].
• “Intelligence augmentation” is a better frame than “AI”, and law can push software toward the explainable, augmenting kind [27:37]–[29:35].
• Warfare is the hardest area, and needs a global frame. Drone asymmetry turns war into policing, which brings policing rules [30:27]–[35:14].
• Counterfeiting humanity is both a deception problem and a devaluation problem [35:55]–[40:04].
• Practical tools: visceral notice (a light, an icon), and corporate-law style record keeping for attribution [40:45]–[45:40].
• What to do: scenario analysis by sector, 10–15 years out, and take stories seriously [46:24]–[50:14].

Close. He ends on culture: films like Ex Machina show “concrete” ethical limits, and experts and narrative together can inform policy [50:14].

Stance to note. He is neither a techno-optimist nor a Luddite. He concedes he cannot predict 2100 [22:46]. He wants to bridge the short-term AI-ethics camp and the long-term existential-risk camp [23:35]. His main tool is law as a lever on the direction of technology, not a ban.

---

Map — talking points

#	Point	Time
1	Intro: Asimov’s Runaround (1942); why update the laws	[00:06]–[02:06]
2	Why Asimov is not enough: non-harm vs “durable human control”; political economy, law	[02:06]
3	Law 1: complement professionals. Why domain experts must sit between patient/client and technology. Snake oil	[02:47]
4	Law 2: do not counterfeit humanity. Label GPT-3 text; no GAN faces posing as people	[04:34]
5	Law 3: stop unproductive arms races. Killer robots, and zero-sum use in law and finance	[05:15]
6	Law 4: attribution. Pedestrian (drone registry) and ambitious (cuts autonomy that worries x-risk theorists) versions	[06:03]
7	Capitalising on human strengths: KPIs are hard; pain reports; patient-reported outcomes	[07:13]–[09:14]
8	Capitalising on human weakness: China classroom face analysis; the “supercharged student record”	[09:54]
9	Why “robots will take our jobs” became a meme: journalism’s collapse, narrative economics, labour-vs-capital cost pressure, “infinite work”	[11:15]–[14:00]
10	Robotic lawyers and doctors: the robot anaesthesiologist; labour made more valuable	[14:10]–[15:36]
11	The digital double: the online will, “version 2.1”, improvisation and routinisation, growth in care work	[16:27]–[18:20]
12	Meta-experts and the infinite regress; complementarity; judges and lunch; data journalism (The Markup)	[19:13]–[21:55]
13	”Do we need experts?” Concession on 2100; short-term vs long-term communities; electronic monitoring	[22:46]–[24:27]
14	Cyber-libertarians and ever-more data; microlending, repayment as the only metric	[25:03]–[27:37]
15	Intelligence augmentation vs AI; Engelbart; FDA explainable vs unexplainable software and liability	[27:37]–[29:35]
16	Governing AI via warfare; the need for a global frame; facial recognition talks with a Chinese delegation; non-proliferation	[29:48]–[32:27]
17	Lawfare, drones as policing, Moyn on “humane” war; two laws of war	[32:27]–[35:14]
18	Counterfeiting humanity: emotion attribution, vulnerability marketing, devaluation of the authentic, robocalls	[35:23]–[40:04]
19	Markers for AI: red/green lights, visceral notice (Calo), the bot emoji, ad attribution, “Asterisk Revisited”	[40:44]–[43:06]
20	The corporate analogy: liability and record keeping; Volkswagen; Tay	[43:38]–[46:24]
21	What can we do: anticipatory social research; scenario analysis; caregiving example	[46:24]–[48:47]
22	Storytelling and culture; Ex Machina; close	[48:47]–[51:10]

---

Part 1 — Before you read

Key terms

• Complementarity vs substitution. A technology is a complement if it makes human work more valuable (the human does more, or better, with it). It is a substitute if it takes the human’s place. Economists keep the two apart because they predict opposite effects on wages. The book’s first law is a policy preference for complements.
• Intelligence augmentation (IA) vs artificial intelligence (AI). Two rival goals from the 1950s–60s. IA: build tools that amplify human thought. AI: build machines that match or exceed it. The same money and talent can go either way [27:37].
• Digital double / data double. The idea that a system can record what a worker does and learn to do it. Pasquale says it was “the inspiration for the whole book” [16:27].
• Meta-expert. Someone, usually a coder or quantitative analyst, who is treated as able to judge the value of every other expert’s work [19:13].
• Counterfeiting humanity. Pasquale’s metaphor from currency: a counterfeit is worthless and it cheapens the real thing [38:11]. Not the same as “humanising” design.
• Affective computing. Systems that read, simulate or respond to human emotion (Rosalind Picard coined the field’s name, 1997; from memory). Pasquale prefers “emotion attribution” to “emotion recognition” [35:55], because the system assigns a label and does not detect a fact.
• Attribution. Tracing an action by a machine back to a responsible person or group [06:03].
• Lawfare. Warning: the word has two senses. In the common sense (Charles Dunlap, 2001; from memory) it means using law as a weapon of war. In Pasquale’s use here [32:27]–[35:14] it means the way international law and norms restrain how far a military can push an advantage. The host calls it a “neologism” and Pasquale then explains it by example.
• Visceral notice. Ryan Calo’s term for a warning you feel and cannot miss (a light, a sound), as against a text notice [41:51].
• Existential-risk (x-risk) theorists. People who worry that advanced AI could end or permanently damage humanity (Bostrom, Ord; from memory). Contrast with the “short-term” AI-ethics community that works on bias, privacy and labour [23:35].
• Explainable vs unexplainable software. A distinction in FDA thinking about software as a medical device [28:16]. Explainable software shows its reasons to the clinician; unexplainable software only emits an output.
• Political economy. The study of how law, markets and power shape each other. Pasquale uses it to say that AI governance is not only ethics [02:06].

People named (a reading list)

Person	Why they come up	Start with
Isaac Asimov	The old laws; the foil	”Runaround” (1942)
Doug Engelbart	Intelligence augmentation	”Augmenting Human Intellect” (1962; from memory)
John Markoff	History of IA vs AI	Machines of Loving Grace (2015)
Jens Beckert, Deirdre McCloskey, Robert Shiller	Stories shape economies	Beckert, Imagined Futures (2016); Shiller, Narrative Economics (2019)
Leslie Willcocks	”Infinite work” thesis	His writing on automation and outsourcing
Julia Angwin (The Markup)	Data science in journalism	The Markup’s investigations
Barbara Evans	Co-author on FDA and AI software liability	Her work on medical-device software
Sam Moyn	War made “longer lasting, more pervasive, and more humane”	Humane (2021; from memory)
Ryan Calo	Robot law; visceral notice	His essays on robotics and law
Ed Felten	Recording lights	Talks on technology policy
Sal Khan	Automated education vision	Khan Academy

Pasquale’s own frame

His earlier book The Black Box Society (2015) argued that finance, search and reputation systems run on secret rules, and that this secrecy is a political problem. New Laws of Robotics (2020) moves from exposing black boxes to asking what the rules should be. The video description gives his post as Professor of Law at Brooklyn Law School; sources found in 2026 list other affiliations too, so check before you cite a current post. The 2021 Brooklyn Law School page for this book tour is at brooklaw.edu.

The current debate (searched 30 Oct 2026)

The talk is from February 2021, before ChatGPT. Each law now has a live policy thread.

• Law 2 (no counterfeiting) became law in the EU. Article 50 of the AI Act, binding from 2 August 2026, sets transparency duties. The Commission published the final Code of Practice on marking and labelling AI-generated content in June 2026, with providers marking and detecting, and deployers labelling deepfakes and AI text. The Code is voluntary but the Article 50 duties are legal obligations (Jones Day, Tech Policy Press). The search did not confirm the chatbot-disclosure paragraph (Art. 50(1)); I believe it requires telling people they are talking to an AI (from memory; not checked).
• Law 2 in US states. California SB 243 (companion chatbots, in force 1 Jan 2026) requires a notice that the bot is not human when a reasonable person could be misled, plus break reminders for known minors, and creates a private right of action (Skadden). New York’s AI companion law took effect 5 Nov 2025, requires a not-human notice at the start and every three hours for all users, and is enforced by the Attorney General (Morrison Foerster). This is close to what Pasquale argued at [04:34], but narrower: it targets companion bots, not all simulation.
• Law 3 (arms races), at the weapons end. Secretary-General Guterres and the ICRC called for a binding treaty on autonomous weapons by 2026. The UN General Assembly held its first meeting on the topic in May 2025, and a December 2024 resolution passed 166 to 3 (Human Rights Watch, Dawn). I could not confirm the status of negotiations in late 2026, so check before you rely on it. The Geneva talks under the Convention on Conventional Weapons have run since 2014 without a treaty.
• Law 1 (complement, not replace), the labour data. The Stanford “Canaries in the Coal Mine” paper (Brynjolfsson, Chandar, Chen) finds that young workers (22–25) in AI-exposed jobs have seen employment fall relative to less-exposed peers. The August 2026 revision, using data to June 2026, reports a gap of about 19%, working mainly through reduced hiring, concentrated where AI automates and flat or rising where it augments (Stanford Digital Economy Lab, PDF). The authors call the results early and descriptive, not causal. This supports Pasquale’s split between automating and augmenting, and also challenges his “labour made more valuable” line for the entry-level rung.
• Law 1 in medicine: deskilling. A 2025 Lancet Gastroenterology & Hepatology study found that endoscopists who had routinely used AI detected fewer precancerous growths when they went back to working without it. It is observational, covered about three months, and authors and outside experts flagged other possible causes (STAT). It shows a risk Pasquale only touches: complementarity can erode the very skill that makes the human a check on the machine.
• Pasquale’s own recent work. A search found no 2025–26 book. The 2020 book remains the latest I found. I could not verify what he has said about generative AI since.

---

Part 2 — The intellectual map

Pasquale spoke for about 45 minutes to a general audience, and the book draws on far more than the conversation shows. Eight families of ideas meet here. This is the depth layer. Everything below is background, not his words, and is from memory unless a link is given.

 LAW & POLITICAL ECONOMY ───────┐                ┌──── ECONOMICS OF AUTOMATION
 Black-box society · licensing  │                │     Complement vs substitute
 liability · corporate law      ▼                ▼     Acemoglu · Autor · Susskind
                         ┌─────────────────────────────┐
 SOCIOLOGY OF PROFESSIONS│  PASQUALE: law should steer │◄─── IA vs AI HISTORY
 Abbott · Freidson       │  technology to complement   │     Licklider · Engelbart
 Polanyi · Dreyfus       │  expertise, not counterfeit │     McCarthy · Weizenbaum
                         │  it or race to replace it   │
                         └─────────────────────────────┘
        ▲                       ▲                      ▲
 NARRATIVE & FORESIGHT   AFFECTIVE COMPUTING      WAR, ROBOTICS & LAW
 Beckert · Shiller       Picard · Turkle          Calo · Moyn · Chamayou
 Jasanoff · Collingridge Weizenbaum · Barrett     CCW · Stop Killer Robots

2.1 The economics of automation — complement, substitute, and who decides

The problem it addresses. Will machines destroy jobs or create them? The talk assumes a more specific question: which kind of machine gets built, and who chooses?

The chain of thinkers

1. Keynes, “Economic Possibilities for our Grandchildren” (1930). Named “technological unemployment”. He expected a 15-hour week by 2030. The standard reference point for the “end of work” story.
2. Wassily Leontief (1983) compared humans to horses made redundant by tractors. The “human as horse” worry is the old form of the “robots will take our jobs” meme the host raises at [11:15].
3. The “task” model: Autor, Levy and Murnane (2003); Autor (2015, 2014). Jobs are bundles of tasks. Routine tasks are automated; non-routine ones are not (yet). Polanyi’s paradox (Autor’s label): “we know more than we can tell”, so tasks whose rules cannot be written down resist automation. This matches Pasquale’s classroom advice, “if you think your daily work routine could be totally routinised and predicted, watch out” [17:29].
4. Frey and Osborne (2013) estimated that about 47% of US jobs were at high risk of computerisation. It set off the “panic” he dates to the early 2010s [11:57]. Brynjolfsson and McAfee’s Race Against the Machine (2011) and The Second Machine Age (2014), and Martin Ford’s Rise of the Robots (2015), widened it. Critics (Arntz, Gregory and Zierahn, 2016, OECD) pointed out that tasks within a job matter more than the job label, and cut the estimate sharply.
5. Acemoglu and Restrepo (2018–2019): automation displaces labour but also creates new tasks; the balance depends on the direction of technical change. “So-so automation” replaces workers without much productivity gain. Acemoglu and Johnson, Power and Progress (2023) argue that the direction is a choice shaped by institutions, taxes and who holds power. This is the closest economic cousin to Pasquale’s claim that law can steer.
6. Brynjolfsson, “The Turing Trap” (2022). Building AI to imitate humans pushes toward substitution and concentrates wealth. Building it to augment shares gains more widely. A direct echo of IA vs AI.
7. Baumol’s cost disease (1966). Services such as care and education resist productivity growth, so their costs rise as manufacturing gets cheaper. Pasquale leans on this idea in the book’s care-work argument (from memory); in the talk it is the background to “the largest growing area of work now is in healthcare” [18:20].
8. Susskind and Susskind, The Future of the Professions (2015). The main target of Pasquale’s first law. They argue that professions are a pre-digital way of sharing expertise, and will be dismantled into tasks, many done by machines or non-experts. Pasquale answers that expertise includes judgment, relationship and accountability that tasks do not capture.

The thread. Automation is not one force. Economists now agree the mix of replacing and complementing technologies is a variable, and the question moves from “how many jobs” to “who steers the mix”. Pasquale’s laws are an answer to that second question.

How it feeds the talk. The “digital double” [16:27], the “infinite work” thesis [13:02] and the robot anaesthesiologist [14:10] all assume the steering question.

2.2 The sociology of professions — why experts exist at all

The problem it addresses. Why do societies give doctors, lawyers and teachers licences, self-governance and high pay? Is it public interest or guild protection? Pasquale’s first law is a defence of professions, so you need both sides.

1. Émile Durkheim, Professional Ethics and Civic Morals (lectures c. 1900). Professions as moral communities that bind self-interest to service.
2. T. H. Marshall (1939) and Talcott Parsons (1939, 1951). The “functionalist” view: professions solve problems of trust where the client cannot judge the quality of the work.
3. Kenneth Arrow, “Uncertainty and the Welfare Economics of Medical Care” (1963). Medicine has information asymmetry (the patient cannot judge), so trust institutions replace markets. Pasquale’s “snake oil” point [03:33] is a version of this.
4. Eliot Freidson, Professionalism: The Third Logic (2001). Three ways to organise work: markets, bureaucracy, and professionalism. His “third logic” is what Pasquale wants to protect against the first two.
5. Andrew Abbott, The System of Professions (1988). The critical-realist view: professions compete over jurisdiction, the right to control a kind of work. New technology redraws jurisdictions. Abbott helps read the fight over who owns “diagnosis” or “legal advice”.
6. Critics: Ivan Illich, Limits to Medicine (1975); Milton Friedman, licensing as a cartel; Magali Larson, The Rise of Professionalism (1977). The sceptical view: professions limit entry to raise their own pay.
7. Michael Polanyi and Hubert Dreyfus. Tacit knowledge and expert skill (see 2.6). Used by Pasquale’s side to say expertise cannot be fully rule-coded.
8. Meta-expertise and the “problem of expertise”. Harry Collins and Robert Evans, Rethinking Expertise (2007), separate contributory expertise (can do it) from interactional expertise (can talk with those who do) and “meta” expertise (judging experts). Pasquale’s “who are the meta meta experts” [19:13] is a regress argument against giving the last word to coders. Collins and Evans’s scheme gives the fuller vocabulary.
9. Social epistemology. Alvin Goldman, “Experts: Which Ones Should You Trust?” (2001): non-experts need indirect ways to pick experts (track record, agreement, incentives). This is the same problem Pasquale poses for judging AI.

The thread. Professions are best defended as a way of solving trust and accountability under uncertainty, not as a way of protecting incomes. That is the version Pasquale uses: experts “intermediate” between people and technology [03:33].

2.3 Intelligence augmentation vs artificial intelligence — a 60-year fork

The problem it addresses. Should computing amplify humans or imitate them? Pasquale says it is a fork in the road and that law can push one way [27:37]–[29:35].

1. Vannevar Bush, “As We May Think” (1945). The Memex: a tool for the researcher to blaze trails through knowledge.
2. J. C. R. Licklider, “Man-Computer Symbiosis” (1960). Humans and computers as partners, each doing what it does best. Licklider later funded much of the early internet.
3. Doug Engelbart, “Augmenting Human Intellect” (1962). The goal is to raise human capability. He built the mouse, hypertext and shared screens, shown in the 1968 “Mother of All Demos”. Pasquale names him [28:16].
4. John McCarthy and Marvin Minsky, the Dartmouth workshop (1956). AI as the project of making machines that think. Markoff’s Machines of Loving Grace (2015) tells the story as a rivalry between these two camps. Pasquale cites it [27:37].
5. Joseph Weizenbaum, Computer Power and Human Reason (1976). The creator of ELIZA turned critic. He argued some tasks (judging, caring) should not be given to machines even if they could do them, because the question is what kind of beings we want to be. Pasquale’s “counterfeit” law carries this.
6. Today. Brynjolfsson’s Turing Trap (2.1) and the “copilot” framing in products revive IA language. Critics say a copilot is a stepping stone to replacement, so the label tells you little. The test is what the system is optimised for (more human skill, or fewer humans).

How it feeds the talk. The explainable/unexplainable FDA split [28:16] is an institutional version: if a tool must explain itself to the doctor, the doctor stays the decision-maker, and it is a form of augmentation.

2.4 Affective computing, simulation and the ethics of seeming

The problem it addresses. Machines can now produce faces, voices and apparent feeling. Is it harmful for a machine to seem to care? Pasquale’s second law says yes, in two ways: deception, and devaluation [38:11].

1. Alan Turing, “Computing Machinery and Intelligence” (1950). The imitation game defines success as indistinguishability. The “counterfeit” law reverses the incentive: success should not be fooling people.
2. Joseph Weizenbaum’s ELIZA (1966). A crude pattern-matcher that users still confided in. Weizenbaum was shaken: people attribute understanding readily. This is the “ELIZA effect”, and it is why Pasquale’s “emotion attribution” wording fits.
3. Rosalind Picard, Affective Computing (1997). Founded the field of systems that sense and respond to emotion. Many applications are benign (tutoring, accessibility).
4. Sherry Turkle, Alone Together (2011). Social robots offer “the illusion of companionship without the demands of friendship”. Her claim that simulated care thins real relationships is the philosophical form of Pasquale’s devaluation worry.
5. Lisa Feldman Barrett and colleagues (2019 review). Scientific evidence that facial movements do not reliably map to emotions across people and cultures. This undercuts “emotion recognition” claims, and supports Pasquale’s “attribution”. Kate Crawford’s Atlas of AI (2021) makes the same argument for a general audience.
6. Mori’s uncanny valley (1970), Reeves and Nass, The Media Equation (1996). People treat computers socially by default. This is why design that exploits it (vulnerability marketing [37:20]) works.
7. Gresham’s law for authenticity. Pasquale’s currency metaphor can be read as Gresham’s law: bad money drives out good. If simulated feeling is cheap and plentiful, people discount all expressed feeling [38:57]. I am glossing here; he does not cite Gresham.
8. Daniel Dennett, “The Problem with Counterfeit People” (The Atlantic, 2023). A later, well-known essay making a very similar case (that making counterfeit people should be treated like counterfeiting money). From memory; not checked. It post-dates this talk, so Pasquale’s priority is real.
9. Opposing view: the humanising-design school. Designers argue that a warm voice or avatar reduces friction, especially for older users and in care. The strongest version is that people know they are talking to a machine and enjoy it anyway (as with fiction). Pasquale’s reply is about scale, manipulation and the erosion of trust, not about fiction.

2.5 Law of attribution — personhood, liability and records

The problem it addresses. If an autonomous system causes harm, who answers? Pasquale’s fourth law makes attribution a precondition of deployment [06:03]. The host reaches for corporate law [43:38].

1. Corporate personhood. In the common law, companies are legal “persons” able to own property, sign contracts and be sued. The US phrase goes back to 1880s case law (the Santa Clara headnote, 1886; from memory). Later fights (the Citizens United decision, 2010) are about how many rights follow.
2. Limited liability and its critics. Shareholders are shielded from the firm’s debts, which encourages investment and risk. The cost is that harm can fall on third parties. Pasquale reads the corporate form as both a model (we can trace and sue) and a warning (it can be gamed to hide ownership) [44:16].
3. Beneficial-ownership transparency. Shell-company rules (the EU anti-money-laundering directives; the US Corporate Transparency Act of 2021, whose enforcement has been contested; from memory; not checked). This is what he means by legislation that makes owners “fess up” [44:16].
4. Records and individual accountability. Sarbanes-Oxley (2002) puts personal responsibility on certified officers. The Volkswagen diesel case shows records enabling prosecution, though only partly (see slip 3).
5. Robot law. Ryan Calo, “Robotics and the Lessons of Cyberlaw” (2015): robots differ from the internet in embodiment, emergence and social valence, and the law will need to treat them as a new category. Ugo Pagallo and Gunther Teubner debated “electronic personhood” for machines. The European Parliament’s 2017 resolution floated it and drew an open letter of objection from AI and robotics experts (from memory; not checked). Pasquale’s law is the opposite view: do not make machines legal persons; keep a human behind each one.
6. The responsibility gap. Andreas Matthias (2004) argued that learning systems create a gap: no one can foresee or control their behaviour, so no one is fairly blameworthy. Pasquale’s answer is that the gap is a design and regulatory choice. Require attribution, and the gap shrinks.
7. Strict liability and tort as a design lever. Guido Calabresi’s “cheapest cost avoider” idea: put liability on the party best placed to prevent harm. Pasquale and Barbara Evans use it for medical software: liability pressure should favour explainable tools [28:16] (I recall the paper as a chapter on product-liability suits for AI-based software; from memory; not checked).
8. Technical attribution. Watermarks and provenance standards (C2PA, in the EU code of practice above) are the machine-readable version of his bot emoji and ad attribution [41:51]–[43:06].

The thread. Attribution does the work that “alignment” tries to do, but through institutions: someone must be findable, and records must exist.

2.6 Craft, tacit knowledge and what experts actually do

The problem it addresses. The digital-double story assumes you can learn a job by watching it. Pasquale says some work needs “improvisation, judgment, consultation” [17:29]. Philosophy of skill explains why.

1. Michael Polanyi, The Tacit Dimension (1966). “We can know more than we can tell.” Autor named the economic consequence “Polanyi’s paradox”.
2. Hubert Dreyfus, What Computers Can’t Do (1972). Expertise is not rule-following. The five stages from novice to expert end with someone who sees the situation and responds without rules. Expert systems of the 1970s–80s, which tried to code rules, stalled.
3. The counter-history: machine learning learns without rules. Since the 2010s, systems learn from examples, so Dreyfus’s argument against rules does not directly block them. The weaker version still stands: data captures what experts did, not why, and not what to do in the case nobody recorded. The “version 2.1” reply that every critique is folded into the next release [16:27] is the strongest form of the counter-argument.
4. Donald Schön, The Reflective Practitioner (1983). Professionals “reflect in action”, handling unique cases that textbook rules do not cover. Close to Pasquale’s “guide someone through” in the will example [17:29].
5. Atul Gawande and the checklist literature. Routine parts of expertise can be standardised, and should be (The Checklist Manifesto, 2009). Pasquale’s view is not anti-standardisation; it is that the line between routine and judgment is moving and should be drawn by the profession, not the vendor.
6. Automation bias and complacency (Parasuraman and Riley, 1997; Skitka et al., 1999). Humans paired with automation tend to over-trust it and monitor poorly. This is the major empirical threat to “the human in the loop” as safety. The anaesthesiologist monitoring several machines [14:10] is exactly the setup in which vigilance fails (see Lisanne Bainbridge, “Ironies of Automation”, 1983: the more automated the system, the harder the human’s leftover job).
7. Deskilling. Harry Braverman’s Labor and Monopoly Capital (1974) is the classic Marxist account of how automation and management remove skill from work. The 2025 colonoscopy study (Part 1) is a modern empirical case.
8. The robot anaesthesiologist. The device he alludes to was, I believe, J&J’s Sedasys (approved by the FDA in 2013, withdrawn in 2016) for sedation in routine procedures (from memory; not checked). He does not name it.

2.7 War, robots and law — asymmetry, policing and the limits of rules

The problem it addresses. The hardest chapter for him [30:27]. Weapons raise the question of global norms where national regulation does not reach.

1. International humanitarian law (IHL). Distinction (combatant vs civilian), proportionality, necessity, and the Martens clause (the “dictates of public conscience”). The ICRC and others argue that machines cannot make proportionality judgments, which are contextual.
2. The campaign. The Campaign to Stop Killer Robots (launched 2013); UN special rapporteur Christof Heyns’s 2013 report on “lethal autonomous robotics”; the Convention on Conventional Weapons discussions since 2014 (confirmed by the sources in Part 1).
3. Distinction between “human in the loop”, “on the loop” and “out of the loop”. The standard vocabulary for degrees of autonomy. US Department of Defense Directive 3000.09 (2012, updated 2023; from memory) calls for “appropriate levels of human judgment”.
4. Drones as policing. The reasoning at [33:32] resembles Grégoire Chamayou’s Drone Theory (2013), which argues that drone war removes the reciprocity of risk and so looks more like a hunt than a war. B. J. Strawser’s “moral hazard” argument (2010) and the debate on whether it is obligatory to use drones to protect your own soldiers are the other side.
5. Reciprocal risk as the moral basis of war. Michael Walzer’s Just and Unjust Wars (1977) and Paul Kahn, “The Paradox of Riskless Warfare” (2002): the right to kill in war rests on the soldier’s own exposure. Remove the exposure and the justification shifts toward policing, where force is bound by necessity and proportionality. Pasquale’s text at [33:32] is a compact version of this.
6. Sam Moyn, Humane (2021). Moyn argues that the humanitarian turn in law made war more humane and more endless. Pasquale quotes the phrase “longer lasting, more pervasive, and more humane” [34:20] and treats the restraint as evidence that societies sense a need to cap asymmetry. Moyn himself is more critical than that reading suggests: the “humane” limits may license permanent war.
7. Lawfare, two senses. The common sense is law as a weapon; Pasquale’s sense is law as a brake (see Key terms).
8. Arms-race logic. The security dilemma (John Herz, 1950): one side’s defensive build-up threatens the other. Military AI has the same structure, and there is no verification regime like the one for nuclear material. His nuclear non-proliferation analogy [31:44] is imperfect for that reason: AI software copies freely and has dual-use civilian uses (from my reading).
9. Zero-sum arms races outside war. The host and Pasquale give law and finance as examples [06:03]. Behind this is Fred Hirsch’s Social Limits to Growth (1976) and Robert Frank on positional goods: spending whose value is relative to others (from memory). High-frequency trading and e-discovery are standard examples.

2.8 Narrative, foresight and the politics of the future

The problem it addresses. The last third of the talk [46:24]–[51:10] asks how societies decide what they want from technology. Pasquale’s answer is stories plus structured scenario work.

1. Jens Beckert, Imagined Futures (2016). Economic action runs on “fictional expectations”: shared stories about what will happen. Pasquale’s “stories we tell about the economy” line [13:02] sits here.
2. Robert Shiller, Narrative Economics (2019). Popular narratives spread like epidemics and move markets. The “robots will take our jobs” meme is a case.
3. Sheila Jasanoff and Sang-Hyun Kim, “sociotechnical imaginaries” (2015). Shared visions of desirable futures that shape policy and technology. National “AI strategies” are imaginaries.
4. David Collingridge, The Social Control of Technology (1980). The dilemma: early on, a technology is easy to change but its effects are unclear; later, the effects are clear but it is hard to change. Pasquale’s “anticipatory social research” [46:24] and “scenario analysis” [47:12] are answers to this.
5. Anticipatory governance (David Guston) and responsible innovation (Stilgoe, Owen, Macnaghten, 2013). The formal methods behind his call.
6. Narratives of AI. The Oxford University Press book he mentions [49:26] is, I think, AI Narratives (Cave, Dihal and Dillon, 2020). He does not give the title, so this is my inference.
7. Hopes and fears are lopsided. Stephen Cave and Kanta Dihal’s work (from memory) argues that Western AI stories skew toward rebellion and replacement, which shapes policy attention.
8. Short-term vs long-term communities [23:35]. The AI-ethics world (Gebru, Bender, Crawford, Benjamin) and the x-risk world (Bostrom, Ord, the effective-altruism milieu) have spent years in open disagreement. Pasquale is trying to bridge. The bridge is visible in his fourth law: attribution cuts runaway autonomy (x-risk) and gives accountability (ethics) [07:13].
9. “Fully automated luxury communism” [23:35] is Aaron Bastani’s phrase (2019; from memory), a left-accelerationist hope that automation can free us from work. Pasquale lists it beside “singularitarian” visions as ones that would supercharge today’s problems.

---

Part 3 — Deep dives

1. Why Asimov is not enough — law as control, not as a harm filter

🎙 In the talk. The host introduces Asimov’s three laws from Runaround (1942) and says the way intelligent agents appear in modern life is “vastly different” from what Asimov predicted [00:44]. Pasquale says Asimov’s laws focus on making sure robots “don’t hurt us”, and that they could do some good if implemented, but he wanted “sustainable, durable human control over technology” [02:06]. That, he says, takes more than a non-harm principle. It “takes us into some principles that involve political economy, sociology, law” [02:06]. The new laws are about the working relationship between humans and robots, in the host’s phrase [01:23].

📚 Deeper theory

• What Asimov’s laws were for. They were a plot device. The stories show the laws failing in edge cases. They also assume one robot with one owner and a clear chain of command. Pasquale’s world is of platforms, firms and states, where the questions are who owns it, who benefits, and who answers.
• Why harm-avoidance is not enough. A system can avoid injuring anyone and still concentrate power, destroy a profession or deceive. These are structural effects (Part 2.1, 2.4).
• The “durable control” idea echoes Collingridge’s dilemma (2.8): control must be designed in while the technology is still malleable.
• Law shapes technology. He gives the FDA explainability example himself [29:35]. Legal regimes set what is cheap to build. Lawrence Lessig’s “code is law” (1999) said the reverse (code regulates); Pasquale’s version is that law also regulates what code gets written.
• Why four laws and not forty. The set is a small tool kit. Each law targets a different failure: replacement, deception, arms races, unaccountability. Compare the OECD AI principles (2019) and the EU AI Act, which take a risk-tier approach instead.

Where others push back.

• From engineers: broad principles are unenforceable; you need testable specifications.
• From industry: innovation-friendly regulation would use sector rules and standards, not “laws of robotics”.
• From the ethics side: four principles can be too thin. Critics of principles-lists say they lack enforcement and are easily absorbed by corporate “ethics washing”.

What they leave out. The book’s own account of how the laws would be enforced, and by whom. The talk names tools (notice, records, FDA liability) but not an enforcer.

Check yourself

• Why does Pasquale think a non-harm rule is insufficient?
• What does it mean to say law can steer which technologies get built? Give his FDA example.
• Which of the four laws deals with structure (economy) and which with the machine’s behaviour?

---

2. Complement, don’t replace — the first law and the professions

🎙 In the talk. Law 1: “robotics and AI should complement professionals rather than replacing them” [02:47]. He expects pushback on two levels: many people want professions automated (a Star Trek tricorder vision), and others think expertise is obsolete [02:47]. His case is that in medicine and other fields you need “an ongoing dialogue and cooperation” between domain experts and technologists [03:33]. Professionals who are close to the patient or client can check whether tools work and protect people from “snake oil”, as with early 20th-century pharmaceuticals [03:33].

Then the human strengths [07:13]–[09:14]:

• KPIs are hard. It is difficult to tell how well a surgical intervention has worked compared with a pharmaceutical or exercise one [07:13]. Education has no agreed measure: degree premium, civic awareness, or basic science and maths [08:32].
• Pain is nuanced. Patient-reported outcomes need “an empathetic, sympathetic interlocutor who’s actually experienced pain” [09:14].
• Human weakness is also useful. In the China classroom case, cameras analyse every student’s face every second for attention. His point: nobody can go back to a third-grade teacher and ask whether you were a menace. Automation turns that fading memory into a “supercharged permanent student record” [09:54], with unfairness akin to the right-to-be-forgotten problems in Europe.

The “robot anaesthesiologist” example [14:10]: sellers first promised replacement, but approvals for safety and efficacy were hard, so the usable product became an assistant: “anaesthesiologists monitoring several of these machines”, or extra failsafes built from the machine’s subroutines [14:10]. Result: higher productivity and more valuable labour [15:36]. The aim of the book is to bring workers in less professionalised fields, including unionised ones, into the same position [15:36].

📚 Deeper theory

• Weakness as a feature is Pasquale’s most original move here, and it needs unpacking. Systems that remember everything change social life. Forgetting works like a social lubricant (Viktor Mayer-Schönberger, Delete, 2009; from memory) and the “right to be forgotten” (Google Spain, 2014) tries to restore it. Human inefficiency leaves room for second chances.
• The measurement problem. Goodhart’s law: when a measure becomes a target, it ceases to be a good measure. Campbell’s law (1976) applies it to social indicators. Pasquale’s point that outcomes are hard to specify is a warning that any AI trained against a proxy will optimise the proxy. See also Jerry Muller, The Tyranny of Metrics (2018).
• Qualitative judgment as evidence. In health, patient-reported outcomes are now standard. But the choice of instrument (what questions, what scale) is itself a judgment, which he says needs people who know pain [09:14].
• A regulatory reason that complements win. The anaesthesia story is about approval: a regulator asked for evidence of safety and efficacy, and full replacement could not meet it. So regulation selected the complement. This is a concrete case of “law steers technology” (Deep dive 1).
• Care work and Baumol (2.1). Care is high-touch and communicative, the biggest growth area in US work [18:20], which is the zone where the replace-by-recording model works worst.
• The deskilling risk (2.6; Part 1 colonoscopy link). Complementarity can wear down the expert’s own ability, and automation bias degrades monitoring. “The human in the loop” is a design problem, not a guarantee.
• Occupational licensing literature. Studies of licensing (Kleiner, Krueger) find it raises prices and restricts entry, with mixed evidence on quality. A fair reader asks whether “defending expertise” is also defending incumbents.

Where others push back.

• The Susskinds (2.1): professions are an outdated way to share knowledge; tasks should be unbundled, and many can go to machines or paraprofessionals.
• Access arguments: a billion people lack access to a doctor or lawyer. Even a poor automated service may beat none. This is the strongest challenge to Law 1 from a global-development angle. Pasquale’s reply (from other talks and the book) is that the access gap calls for funding people and tools, not for lower-quality substitutes; I did not find that argument in this talk.
• Labour data (Part 1, Canaries): hiring of young workers in exposed occupations is falling. “More valuable labour” may be true for those already in the profession and false for entrants.
• Economic pressure: employers choose substitution when it is cheaper, regardless of what a professor recommends.

What he leaves out. In this talk: who pays for the human in the loop, and how to stop a cheaper automated tier from emerging. Also, how the junior rung of a profession survives if routine tasks go to machines and training depended on them.

Check yourself

• Why did robot anaesthesiologists become assistants? What role did regulation play?
• What does Pasquale mean by “human weakness” as a thing to capitalise on?
• What is the strongest argument that Law 1 mainly protects incumbents?

---

3. Where “robots will take our jobs” came from — a story, a business model and a mistake

🎙 In the talk. The host asks why the obsolescence idea became so dominant [11:15]. Pasquale dates the “current panic” to the late 2000s or early 2010s [11:57]. His account: journalists watched their own profession collapse as newspapers went from 400 reporters to 30, because money moved to automated platforms that matched audiences to advertisers. They “extrapolated from their own experience” that core competencies were easy to automate [11:57]. He adds the role of narrative economics (Beckert, McCloskey, Shiller) [13:02], and “an ongoing pressure to cut labour costs in order to enhance the relative value of capital” [13:02]. He does not want to blame journalists: “There are a lot of very sophisticated commentators” [13:02]. Leslie Willcocks predicts the real problem will be “infinite work”, because technology produces data that needs human judgment [13:02]–[14:00].

📚 Deeper theory

• A rival explanation. Journalism’s collapse came mostly from advertising moving online (classified ads, search ads), not from machines writing stories. So the journalists’ extrapolation mistook a platform shock for an automation shock. Pasquale’s point is that the mistake was understandable and then spread. This is a testable claim about media-history, not something the talk proves.
• Cycles of technological panic. The 1960s “automation” scare (the 1964 US Triple Revolution memo; the National Commission on Technology, Automation and Economic Progress, 1966), Keynes in 1930, the Luddites in 1811–16. Each predicted mass joblessness; none got it at the predicted scale, though the transitions hurt.
• Why managers like the story. Replacement is a bargaining threat. Whether or not it happens, it disciplines wages. Marx’s “reserve army of labour” is the old form (from memory).
• “Infinite work”. Related to the “ghost work” literature (Gray and Suri, Ghost Work, 2019) and data-labelling work: AI creates human tasks (labelling, moderation, cleaning, checking). Whether that work is good work is the question.
• Narrative contagion (Shiller). Stories spread when they are simple, emotionally charged and tied to an identity. “The robot takes my job” is all three.
• Evidence now. The Stanford canaries paper (Part 1) shows measurable effects on entry-level hiring in exposed jobs by 2026, which supports the idea that the story was early but the effect has started. This partly vindicates the panic, and partly confirms Pasquale’s point that the effects depend on automation vs augmentation.

Where others push back.

• The “story” explanation can sound dismissive; the capability jump since 2022 (large language models) is real, and labour economists who were sceptical in 2018 are less so now (from memory).
• If the panic is merely a meme, who profits from it, and how do you show that? The talk gives plausible causes, not proof.

What he leaves out. Evidence from outside rich countries, and the argument that if “infinite work” is largely low-paid cleaning of machine output, it is not the better future it sounds like.

Check yourself

• What two causes of the robot-obsolescence meme does Pasquale give?
• Why might journalism be a misleading case to extrapolate from?
• What is “infinite work”, and why is it not automatically good news?

---

4. The digital double, meta-experts and cooperation

🎙 In the talk. The host describes the data double: machines record what workers do, imitate it, and replace the worker [15:36]. Pasquale says this model “was actually the inspiration for the whole book” [16:27]. His story: when he told legal-tech people that their online will-writing program missed a case (a separate Designation of Beneficiary form decides who gets an asset, not the will), they thanked him, because it gave them “version 2.1, or version 2.2” [16:27]. “Any critique that was offered would eventually be wrapped into the automation and the AI itself” [16:27]. He has two responses. One is to join them. The other is to say: “Wait, there’s some things where it’s not quite clear what the answer is”, and then the lawyer has to guide someone through [17:29].

His advice to students: “If you think your daily work routine could be totally routinised and predicted, watch out, because it will probably be routinised and predicted by somebody” [17:29]. Work needing improvisation, judgment and consultation is growing [17:29], particularly in US healthcare and home health [18:20].

On meta-experts [19:13]: those who are identified as experts in code and quantitative analysis and thought able to judge other experts’ value. This faces a regress (“who are the meta meta experts”) and the difficulty of evaluation. The positive view is complementarity [20:18]. Examples:

• Judges. Studies purport that judges give lighter sentences after lunch and harsher juvenile sentences after their football team loses [20:18]. The right response is not a language model that outputs a sentence length, but system-level statistical feedback to the courts: “we’ll give the judge a snack before lunch, or at least warn them that these patterns are happening” [20:18].
• Journalism. NICAR-style computer-assisted reporting and Julia Angwin’s The Markup bring rigour: analysis of “10 thousand comments on the agency website” instead of five interviews [21:55].

📚 Deeper theory

• The “version 2.1” reply is the strongest objection to the book, and he states it himself. If every critique becomes training data, then no limit is final. The counter is that some questions are open (the law is unclear), so a system needs a person who can say the answer is contested. Law has “open texture” (H. L. A. Hart, The Concept of Law, 1961), meaning rules run out and judgment takes over. A system can be updated for known cases; the unknown ones define the profession.
• Why meta-experts regress. Evaluating an expert needs expertise in the field, or a track record. Collins and Evans (2.2) separate these. A coder can evaluate performance on a benchmark, but benchmark choice is itself a domain judgment.
• The sentencing example, in more detail. Algorithmic risk tools in criminal justice (COMPAS, ProPublica 2016) show the replace-the-judge route and its disputes about bias and fairness measures. Pasquale’s “feedback to judges” model keeps the human decision and uses statistics to audit the system. This is the same pattern as audit studies of bias in hiring.
• Caveat on the lunch study. The original effect (Danziger et al., 2011) has been criticised (see slip 2). It is used here as an illustration of the kind of bias statistics can reveal, not as proof. A more careful example would be sentencing disparities by race or judge assignment.
• Data journalism as a model. It works because the reporter decides the question and checks the output. The method is inverted when an LLM drafts the story. See the generative-AI context in Part 1.
• Routine vs judgment, and the improvisation test. A practical rule from the talk: work that is predictable will be predicted. This is a forecast about firms with data, not an abstract claim. A job with little data trace is safe until someone starts logging it.

Where others push back.

• Algorithmic fairness researchers say humans are noisy and biased (Kahneman, Sibony, Sunstein, Noise, 2021), so “snack before lunch” understates how much better a rules-based or statistical judge could be. Pasquale concedes that bias audits help; he resists replacing the judge.
• Legal-tech builders reply that open questions are a shrinking class as systems read more case law.
• Skeptics of “complementarity” say the complement phase is transitional.

What he leaves out. Who owns the “system-wide” statistics about judges, and whether judges would accept being nudged. Also, whether giving the judge a snack is a trivial fix that avoids deeper bias in the law itself.

Check yourself

• Why is “version 2.1” a strong objection, and how does Pasquale reply?
• What is the regress in “meta-experts”?
• How does the judges-and-lunch example show complementarity rather than replacement?

---

5. Cyber-libertarians, microlending and the limits of “more data”

🎙 In the talk. The host asks about libertarians who say AI should be free to think and access all the data. Pasquale: this “has this logic of ever increasing data accumulation. To me, it becomes alienating. Alienating in the sense of meaninglessness and powerlessness” [25:03]. His example is algorithmic lending: the argument is that free contracts produce more data on good and bad risks and allocate credit “optimally” [25:03]. His objection is based on talking to people in fintech microlending: success is judged by repayment, and “They don’t really get into how the repayment is made” [26:09]. A loan repaid by another loan, or by skipping meals, signals desperation, not success [26:09]. He cites a study of Kenyan microlending that found people repaying but struggling [26:57]. Finance, with media, is where AI “has gone the furthest” and has “so many ongoing problems” that regulators and firms have not addressed [26:57].

📚 Deeper theory

• The ideology. The “Californian Ideology” (Barbrook and Cameron, 1995) and Barlow’s 1996 Declaration of the Independence of Cyberspace joined counter-culture and free-market faith. “Data wants to be used” is its descendant. Hayek’s argument (1945) that markets aggregate dispersed knowledge is the intellectual base for “more data, better allocation”.
• Why repayment is a poor measure. A single outcome measure ignores the cost of reaching it. This is Goodhart again (Deep dive 2) and also the problem of externalities that the lender does not bear. Economists in microfinance (Banerjee, Duflo and others, randomized trials from 2015) found modest average gains from microcredit, not transformation (from memory), which fits the sceptical view.
• Predatory inclusion (Louise Seamster and Raphaël Charron-Chénier, 2017): including a previously excluded group on worse terms can extract more than it helps. Digital lending is a modern case.
• Alienation. In the Marxian sense: separation from the products and meaning of one’s activity. Pasquale uses the word loosely, as meaninglessness and powerlessness [25:03].
• Surveillance capitalism (Zuboff, 2019) is the broader frame. Data accumulation is a business model, not a neutral process.
• Counter-view: credit access. Alternative-data scoring does bring credit to people without a bank history. The M-Pesa success story (Kenya) is the usual example. The strongest reply to Pasquale is that over-lending is a rule problem (interest caps, disclosure), which can coexist with expanded access.

Where others push back. Fintech advocates say that data-driven credit is fairer than relationship lending, which was often discriminatory. Libertarians say regulators have their own failures.

What he leaves out. What a good metric would be. A natural candidate is repayment with measures of borrower welfare (income stability, missed meals), but those are costly to collect, which is why they are rarely used.

Check yourself

• What does Pasquale mean when he says data accumulation is “alienating”?
• Why is repayment alone a bad measure of a loan’s success?
• What is the best argument in favour of algorithmic microlending?

---

6. Intelligence augmentation, explainability and the FDA

🎙 In the talk. The host asks whether “AI” is even the right word, or whether it should be intelligence augmentation [27:37]. Pasquale agrees and credits Markoff’s Machines of Loving Grace for the history [27:37]. Engelbart pushed IA in the 1960s, against people whose goal was “something that’s in silico, a replacement for, or assimilation of, or goes beyond what the human can do” [28:16]. He describes his work with Barbara Evans on FDA treatment of software as a medical device. The FDA has “developed a distinction between explainable and unexplainable software” [28:16]. His proposal: to the extent software is unexplainable (“spitting out recommendations in a black box sort of style”), it “should be subject to more lawsuits than if it is just a tool that is walking the doctor through” [28:16]. IA “perhaps comes closer to what a real machine conversing with a human might look like”, because it explains its reasoning [28:16]. He sums it up: “legal regimes can play a role in helping push the development of technology in one direction or another” [29:35].

📚 Deeper theory

• The regulatory background (from memory; not checked). In the US, the 21st Century Cures Act (2016) carved out some clinical decision support software from device regulation if clinicians can independently review the basis of the recommendation. That is the legal home of the explainable/unexplainable split. The FDA has since issued guidance on clinical decision support and on AI-enabled device software.
• Liability as design pressure. Using tort law to reward explainable tools is a classic law-and-economics move (2.5). It makes the vendor bear costs when it chooses opacity.
• Explanations are not all equal. Cynthia Rudin, “Stop explaining black box models for high-stakes decisions” (2019) argues for inherently interpretable models, because post-hoc explanations can mislead. Others argue that accuracy matters most and that people are not explainable either. Pasquale’s category is institutional (does the tool give reasons a clinician can use), not technical.
• Who is protected? If an explainable tool is used and the doctor follows it, the doctor is liable; if the black box is used, the vendor is. This moves the liability relationship and may change which tool hospitals buy.
• Large language models raise the stakes. They produce fluent explanations that may not reflect the true cause of the output. Fluent is not the same as faithful. See the medical-practice sibling page for more.
• IA language now. “Copilot” and “assistant” are IA words; the test is the objective and the pricing (does the vendor profit when staff are cut?).

Where others push back.

• Performance first: if a black box saves more lives, demanding explanation costs lives. Responses: the evidence of real-world performance is often thinner than trials suggest.
• Liability chills innovation in small firms.
• Explanation can be theatre: clinicians may not read it.

What he leaves out. How a doctor should weigh a black-box tool that outperforms humans on average. His answer here is liability, not a rule on use.

Check yourself

• Define explainable vs unexplainable software, in the FDA sense Pasquale describes.
• How does liability steer design?
• Why might a fluent explanation from a language model not meet the explainability standard?

---

7. War, drones and the limits of “lawfare”

🎙 In the talk. The host notes that governance gets exciting only when guns come up [29:48]. Pasquale calls this “the hardest part of the book to write” [30:27]. For the rest of the book he could imagine a national or provincial regulator; for killer robots and cyber-security “we really do need a global perspective” [30:27]. He describes a mixed group with Chinese government representatives, Australian academics and Americans, where it was hard to agree even on facial recognition, because one side stressed harms and the other talked about stopping child kidnappings [31:06]. He hopes for “a level of global alarm” and compares the task to nuclear non-proliferation [31:44].

On lawfare [32:27]–[35:14]: drones that “can potentially watch people 24:7” and strike. Normally war’s tolerance for killing rests on mutual risk. With a fleet operated from Nevada, operators “can’t be hurt by the people that they’re patrolling” [33:32], so it looks more like policing, which has rules on excessive force. Sam Moyn: war is “longer lasting, more pervasive, and more humane” [34:20]. He reads the international norms as evidence that we are “intuiting that mass technological asymmetry is something that needs to be regulated and recognised, and not pushed too far to one side’s advantage” [34:20]. He suggests two laws of war: one for occupation, another for matched enemies where “almost anything can go” [35:14].

📚 Deeper theory

• The risk-reciprocity argument (2.7: Walzer, Kahn). Pasquale’s chain is: reciprocity of risk justifies killing → asymmetry removes it → policing norms apply. It is an argument from the moral structure of war, not just from treaty text.
• Is it descriptive or normative? He says norms show we “intuit” a need for limits. That is an empirical claim about practice. Critics (and Moyn) note that the practice also licenses endless low-intensity war with fewer constraints on where and whom.
• The two-tier idea. Occupations/counter-insurgency with policing norms; great-power war with few limits. It matches a pattern in international law (the law-enforcement vs. conduct-of-hostilities paradigms, a standing dispute in the debate over targeted killing; from memory).
• Why global agreement is hard. Dual use (the same vision system finds missing children and tracks protesters), different threat perceptions and verification. Compare the chemical-weapons regime, which works partly because inspection is possible.
• Where the topic stands. See Part 1: the 2026 treaty target, 166-3 General Assembly vote in 2024, and the CCW’s stall since 2014.
• How this links to Law 3 and Law 4. An attribution rule (a registered owner for every drone) is the cheap, civilian-side version. Pasquale’s claim at [07:13] that attribution blocks the autonomy forms of most concern to existential-risk theorists is a bridge from this military topic to the long-term debate.

Where others push back.

• Military-ethics realists argue autonomous systems could be more discriminating than stressed soldiers (Ronald Arkin).
• Pacifist/abolitionist critics say the “humane war” framing legitimises war.
• Strategists say bans are unverifiable and the powers will not sign.

What he leaves out. Cyber conflict and information operations, which he mentions but does not discuss, and who enforces any rule against a nuclear state.

Check yourself

• Why does asymmetry make war look like policing, in Pasquale’s argument?
• What is the difference between Pasquale’s and the common meaning of “lawfare”?
• What does Moyn’s phrase imply, and how does it complicate an optimistic reading?

---

8. Counterfeiting humanity — deception, devaluation and markers

🎙 In the talk. Law 2 [04:34]: no “massive investment in tools that are meant to deceive users into thinking that the tool itself has had an emotional response. He wants generated text from models like GPT-3 labelled, and a GAN-generated face not presented as just a person” [05:15]. The host says he had not realised that humanising and counterfeiting are different, and asks for the dividing line [35:23].

Pasquale: “This is one of the most controversial of the laws” [35:55]. It has a pragmatic side (limits on affective computing used to manipulate) and a “metaphysical commitment” [35:55]. He prefers “emotion attribution” to “emotion recognition” because emotions are “ephemeral and ineffable”, and “the mere suggestion goes a long way towards creating the thing” [35:55]. Examples: ads matched to your face, “vulnerability marketing” [37:20]. Proponents say avatars humanise; he replies “there’s something that’s really deceptive there, because there isn’t a human there” [37:20]. The OS in Her is his vivid case: it lacks the emotions of the people it talks to yet “taking advantage of a world in which that was happening” [38:11]. The second harm: simulation “will inevitably lead to a devaluation of the authentic thing itself” [38:11]. His analogy is robocalls: people stop answering unknown numbers, which has hurt “doctors not getting reached” and COVID-19 contact tracing [38:57]. The metaphysical side: a world of humanoid mimics would place overwhelming “claims on resources and attention” [40:04], though he allows that more positive visions may exist [40:04].

On markers [40:44]–[43:06]: Felten’s red recording light; green lights on Macs [40:45]; visceral notice from Ryan Calo [41:51]; a robot emoji in a Twitter bio to mark a bot [41:51]; links on ads for attribution, and his early article “Asterisk Revisited” proposing a small asterisk linking to a person’s reply to a search result [43:06].

📚 Deeper theory

• Two harms, two theories. Deception is a consent harm (the user would object if they knew). Devaluation is a commons harm: a shared signal (expressed feeling, a voice on the phone) loses informational value when it is cheaply forged. The second is the more original claim and needs the Gresham-style reading in 2.4. Compare spam wrecking email and robocalls wrecking the phone.
• Signalling theory. Spence’s signalling model: a signal is useful only if it is costly to fake. AI makes faking human warmth cheap, so the signal weakens, and honest people lose.
• Dark patterns and manipulation law. US Federal Trade Commission actions on deceptive design; the EU AI Act bans manipulative techniques that exploit vulnerabilities (Art. 5; from memory; not checked). Vulnerability marketing at [37:20] fits the second.
• The “Her” example. Turkle-style worry (2.4): relationships without demands. Also related to the 2024–26 debate on companion apps; see the SB 243 and New York laws in Part 1, which require not-human notices and safety protocols.
• The metaphysical claim, restated. He is not saying machines cannot have feelings. He says if they seem to, humans will be pressed to respond, and our attention and care are finite. This is a point about moral attention, close to Peter Singer’s and Paul Bloom’s discussions of scarce empathy (from my reading).
• Why the marker examples are good. They use existing affordances (lights, emoji, links) that do not annoy. The tension: disclosure laws often produce banners nobody reads. Ryan Calo’s visceral notice is aimed at that failure.
• Is the law enforceable on open models? Watermarks can be stripped. Hence the EU code’s mix of marking, metadata and detection (Part 1).

Where others push back.

• Humanising design is useful: for people with disabilities or loneliness, a warm interface helps.
• Fiction has always simulated feeling; the difference is knowledge. If users know, is there deception? Pasquale’s answer relies on exploitation and scale.
• Definition problem: he calls it “a rather difficult principle to apply sometimes” himself [04:34]. Where does a polite voice assistant end and a counterfeit begin?
• Speech rights: he notes US bot speech has First Amendment protection claims [38:57]. Courts have not settled this. The state companion-bot laws (Part 1) now test it.

What he leaves out. A test for “counterfeit” that does not depend on intent. The state laws chose a narrower trigger: whether a reasonable person could be misled.

Check yourself

• Separate the two harms of counterfeiting humanity. Which is about a shared commons?
• Why does he prefer “emotion attribution” to “emotion recognition”?
• What are two markers he proposes, and what problem does Calo’s “visceral notice” target?

---

9. The corporate analogy, scenario analysis and stories

🎙 In the talk. The host notes that corporations are already artificial persons and asks how that framework helps [43:38]. Pasquale: the analogy “has both promise and pitfalls” [44:16]. The promise: liability, and the rule that “we’re supposed to know who owns it”, though people have found ways around it [44:16]. The second layer: trace actions to “particular people who made certain decisions”, through record keeping, as in the Volkswagen diesel scandal where key actors were identified [44:54]. For robots: keep track “both of the initial algorithms and data and of the people that interacted with them who might have interacted in a malicious way”, as with Microsoft’s Tay [45:40].

On action [46:24]–[48:47]: the dialogue is stuck between “a very unambitious form of economics and engineering” and “a very long term view that is not really translatable into current policy” [46:24]. His proposal: scenario analysis by sector, 10–15 years out, with concrete steps [47:12]. Example: robotic caregiving. Workers should be treated well, get tools that lessen the burden of the hardest parts, and keep developing the skills they find meaningful [47:12]. That involves licensing requirements for both the robots and the people, certification, and worker involvement in improvement through reports and post-marketing surveillance [48:10]. He points to existing “ongoing quality improvement in a learning healthcare system” as a base [48:10].

On stories [49:26]–[50:14]: “taking seriously the culture” (art, novels, films), including works that resonate with few people. Ex Machina is not just “another techno-thriller”: it shows ethical limits “when the rubber hits the road” [50:14].

📚 Deeper theory

• Pitfalls of the analogy. Corporations are legal fictions run by humans; their decisions pass through people. An autonomous system may have no human decision to point to. That is the responsibility gap (2.5). Also, the corporate form is used to shield people, so extending it to machines could give another shield (a shell AI). The EU’s “electronic personhood” debate (2.5) turns on this.
• “Know who owns it”. Beneficial-ownership registers (2.5) are the model; a drone or model registry would work likewise. Compare aircraft registration marks and car number plates. Licensing drivers and registering cars was how cars were governed.
• Post-marketing surveillance is the medical-device and drug idea of tracking real-world performance after approval. Applied to AI, it is the case for continuous monitoring and incident reporting (similar to aviation’s safety reporting).
• Scenario planning comes from Herman Kahn at RAND and Pierre Wack at Shell (1970s). Collingridge’s dilemma is the reason for starting early (2.8). Backcasting (start from the desired end state, work back) is the structure Pasquale describes for caregiving.
• Participatory design. Scandinavian “Utopia” project of the 1970s–80s, where unions co-designed technology with computer scientists. Shows worker involvement in technology can be institutionalised, and is a precedent for “involve them in the improvement of it” [48:10].
• Why stories matter in policy. Beckert, Shiller and Jasanoff (2.8). Fiction rehearses ethics: Ex Machina is about a creator who treats his creation as a test subject and a possible consumer of her own deception; critics also read it as about gender and power (from my reading).
• Tay, in more detail. The incident shows the attribution problem has three parties: builder, maintainers and the users who attack the system. Record-keeping has to cover interaction logs.
• Present-day check. The Part 1 items are the nearest real-world tests: transparency rules (Art. 50), companion-bot laws, and so on.

Where others push back.

• Scenario planning is only as good as who is in the room. Industry-dominated panels produce industry-friendly scenarios.
• Records can be falsified or lost, and logging everything raises privacy problems.
• Stories can mislead policy (killer-robot imagery dominates attention while quieter harms such as scoring and ranking go unaddressed).

What he leaves out. The institutions that would run sectoral scenario work and carry the results into law, and the funding.

Check yourself

• Give one strength and one weakness of treating AI like a corporation.
• What does Pasquale say attribution requires in practice? Use Tay or Volkswagen.
• What is scenario analysis, and what is his caregiving example?

---

Putting it together

The mental model in one paragraph. Technology does not decide its own direction. Firms, regulators and stories do. Pasquale reads the present as a fork: one path builds machines to augment experts (and so makes labour more valuable), another builds them to replace or imitate (and so concentrates power and cheapens human signals). The four laws are levers on that fork. Complementarity steers investment. No counterfeiting protects the signals (feeling, voice, face) that trust runs on. No arms races blocks spending that adds nothing but relative advantage. Attribution ensures that someone is answerable. Underneath is a view of expertise: it is not rule-following that data can capture, but judgment under uncertainty, owed to a person, and kept honest by institutions (professions, courts, regulators). So the practical agenda is institutional: notice, records, liability, licensing, and sector-by-sector scenario planning, all informed by culture and stories.

Open tensions

1. Complement or transition? Is complementarity a stable end state or a stage before replacement? The 2026 entry-level data (Part 1) is a warning.
2. Defending expertise or defending incumbents? The strongest critique of Law 1 (Susskind, access arguments, licensing research).
3. Simulation: harm or help? Law 2 depends on contested claims about deception and devaluation; the first laws now in force are narrow.
4. Global norms. He admits war is the hardest case. The 2026 treaty goal shows how far it is.
5. Who enforces? The talk names tools but not an enforcer, and several tools (notice, records) are known to underperform in practice.

Reading path

• Start: Pasquale, New Laws of Robotics (Harvard UP, 2020), chapters 1–2 and 5 (professions, medicine, and the counterfeit law). Markoff, Machines of Loving Grace (IA vs AI history). Brynjolfsson, “The Turing Trap” (2022).
• Then: Susskind and Susskind, The Future of the Professions (the position he argues against). Acemoglu and Johnson, Power and Progress (direction of technology). Freidson, Professionalism: The Third Logic. Ryan Calo, “Robotics and the Lessons of Cyberlaw”.
• Deeper: Weizenbaum, Computer Power and Human Reason. Dreyfus, What Computers Can’t Do. Collingridge, The Social Control of Technology. Moyn, Humane. Chamayou, Drone Theory. Pasquale, The Black Box Society.
• Next on Pasquale: pasquale-machines-judging-humans, pasquale-future-of-ai-in-law, pasquale-ai-changing-medical-practice.

---

Q&A

(Add questions and answers here only if worth keeping.)