Do we have relationships with AI? — Joanna Bryson on "category failure", liability, grief bots and consciousness

About this talk

Speaker: Joanna J. Bryson (Professor of Ethics and Technology, Hertie School, Berlin)
Event: One-to-one video interview with a UK-based interviewer (name not given); Bryson on sabbatical writing a book
Date: not confirmed. Inferred mid-2026 or later (she calls a German court ruling on Google AI summaries “a really recent result”; the Munich ruling was 28 May 2026)
Length: 52:34 (transcript end; the recording may run a little longer)
Channel: Andrew Phipps
Watch: Professor Joanna Bryson: “You Don’t Have a Relationship With AI, You Have One With Yourself

How to read this page. It has two kinds of content, always marked:

🎙 In the talk. What Bryson (or the interviewer) says, with [MM:SS] timestamps. Only these parts are her views. Anything inside quotation marks is checked word for word against the transcript.
📚 Deeper theory. Background from the traditions she draws on, and from the live debate. It explains why she thinks what she thinks and who disagrees. It is not her claim unless a timestamp says so. Items from my own memory that I did not check are marked (from memory; not checked).

Suggested route. Summary → Map → Key terms → Part 2: The intellectual map (read once, slowly) → Deep dives in order → “Putting it together”.

Speakers

Label	Who	Role
Interviewer	Not named in the transcript. UK-based: mentions the UK’s under-16 social media law, University of Exeter, my university [52:05], and a 16-year-old son and 19-year-old daughter [31:32]. His mother died of dementia “a couple years ago” [23:40]; both parents have died [33:35]	Hosts, asks the questions, adds anecdotes. Confidence on identity: none. Confidence he is an academic or university-linked: medium
Bryson	Joanna Bryson (named in the opening introduction [00:00])	Professor of Ethics and Technology, Hertie School, Berlin. Degrees from Chicago, Edinburgh and MIT (per the introduction). Confidence: high

Other people she or the interviewer mention but who do not speak: Tonka (Thomas) Grant and Mihailis Diamantis (co-authors of the 2017 “shell company” paper), Daniel Dennett, Anna Strasser (the interviewer’s “Anastasa”), Geoffrey Hinton, Richard Dawkins, Blake Lemoine, Shannon Vallor.

Caption warnings (YouTube auto-captions, English). Names are often wrong. Corrections used on this page:

“Hery School” = Hertie School.

“Replica AI” = Replika (chatbot app).

“Palanteer / Palunteer” = Palantir.

“Donkins” [47:34] = Dawkins. “Blake Lamar” [41:24] = Blake Lemoine. “Shan Valor” = Shannon Vallor.

“Tonka and Mail Diamantis” [07:55] = Thomas (“Tonka”) Grant and Mihailis Diamantis (first name of Grant from memory; not checked).

“Anastasa” [38:18] = probably Anna Strasser (from memory; not checked).

“Kernigan / Richie” [18:27] = she is trying to recall Ken Thompson (see the slip note below).

“Jane Austin land” = Jane Austen. “Delelfy” = Delphi. “birectional” = bidirectional. “imposttor” = impostor. “Truly Madly Deeply” is correct (1990 film).

“Chhattam situations” [23:07] is unclear; I could not decode it. The sentence is skipped.

“Wahhabism” [36:14] is as captioned. It looks like a deliberate example, not a caption error.

The speaker labels (”>>”) are not reliable. At several points it is unclear whether a stretch belongs to Bryson or to the interviewer (for example the Dawkins “reflection” passage at [47:34], which I read as Bryson).

Factual slips and unverified claims in the talk

Backdoor in the C compiler [18:27]. The story is real but the speaker misremembers it. It is Ken Thompson’s 1983 Turing Award lecture, “Reflections on Trusting Trust” (printed in Communications of the ACM, 1984). He showed how a compiler could insert a backdoor into the Unix login program and also into every later compiler built with it. He presented it as a demonstration of the attack, not as a deployed backdoor on every Unix system, so “every Unix system … since then” overstates it (from memory; not checked against the lecture text).

Wake-sleep algorithm “in the 80s or something” [46:01]. Hinton’s wake-sleep algorithm is from 1995 (Hinton, Dayan, Frey and Neal, Science), not the 1980s. Hinton’s 1980s work was on Boltzmann machines and backpropagation (from memory; not checked).

“Robots should be slaves” written “around 2007 … published in 2010” [24:42]. Published in 2010 in Yorick Wilks (ed.), Close Engagements with Artificial Companions. The earlier-drafts claim I cannot check. The “five principles” of the EPSRC/AHRC Principles of Robotics came out of a 2010 workshop and were published in 2011; she dates them 2010 (from memory; not checked).

German judge on Google AI summaries [11:02]. She says “a German judge” found AI summaries are not search results. The matching ruling is the Regional Court of Munich I, 28 May 2026 (preliminary injunction, case 26 O 869/26). A Berlin court reportedly decided the other way, and Google said it would appeal (The Decoder, Semafor). So “it’s one country but it’s a big one” is right, but it is one court’s interim ruling and is contested inside Germany.

Dennett test “at chance or even worse than chance” [39:32]. She is describing her own result as one participant. The published study (Schwitzgebel, Schwitzgebel and Strasser, Mind & Language 2023) found Dennett experts picked his real answer about 51% of the time against 20% chance (five options per question), so experts were clearly above chance as a group (summary of the paper). Her result and the group result can both be true.

Interviewer’s claims, not checked: that the Dennett-model’s “hallucinations became more and more extreme” over time, and that the agreement was to delete it when Dennett died [39:32]; that “in the UK we’ve just passed the law where social media is banned for under 16s” [31:32] (I found no source for this).

Summary

Central question. The interviewer opens by asking Bryson to unpack a distinction she drew in earlier correspondence: the connections humans form to AI versus any relationship between humans and AI, which would imply a mutuality she rejects [00:00].

Main thesis (Bryson’s). A relationship is “necessarily birectional” [01:06]. A chatbot is “a software it’s a program” [02:42]. So to say a company’s product has a relationship with you is, in her words, “a category failure” [01:06]. People do experience a relationship, and that experience may be psychologically close to the one we have with a fictional character or a pet. That experience is real. The counterpart is not.

The talk then spreads from this point into six linked areas:

What the machine actually is. A giant lookup into the training data plus small isolated storage of what you said (“memory” is shelves). You steer which part of the library it draws from, so one person’s Claude is not another’s [02:42]–[05:22].
Who is responsible. Trust in the sociological sense means penalties can be enforced between people. Courts already split blame between driver and car maker, and there is no “third entity” [07:55]–[09:29]. Generated speech may be closer to a publisher’s than to a search engine’s [11:02]. But weigh the net effect on all users, not only the tragic cases [12:35].
Power and sovereignty. AI firms cannot claim to be both critical infrastructure and “just a business”. Utilities carry extra duties [14:10]. Palantir-style vendors raise an audit problem, and she tells the compiler-backdoor story to show why [16:54]–[19:59]. “Data colonialism” is a fair worry, and Africa’s handling of solar contracts is a counter-example [21:33].
Companions and loneliness. She stands by “robots should be slaves”, while apologising for the title [24:42]–[27:39]. Loneliness is old, and chatbots may crowd out the chance encounter that changes your life [30:47]. Fantasy that you know is fantasy can be fine [29:13].
Grief bots and digital copies. They need controlled testing, not “natural experiments” [37:00]. Preserving a person as a bot “doesn’t make sense to me” [36:14]. A Dennett model fooled even his friends, yet it can only index what he wrote [41:24].
Consciousness. She calls it easy to build in the narrow sense (explicit recall, self-report). What matters is moral agency (who can be held to account) and moral patiency [46:01]–[49:08]. You cannot turn money into people: 500 virtual agents with a vote would collapse democracy [49:08].

Close. She is writing a book on the constraints of computation and biology, arguing that cooperation, language and consciousness are natural, and that justice and government are analogous coordination systems [50:15]. Her last point: “the stupidest thing a country can do right now” is cut funding for the humanities [51:32].

Stance to note. She is not hostile to fiction or comfort. Her objection is to category confusion and to the business model behind it. She keeps the question of whether the user is helped (an empirical, testable question) apart from the question of what the thing is (not a person, not a bidirectional partner).

Map — talking points

#	Point	Time
1	Intro. Her credentials; interviewer notes she wrote about AI before most people had heard of it	[00:00]
2	Relationship vs belief in one. Bidirectionality. “Category failure”. Pets, characters, plants	[01:06]
3	What “memory” is. Library lookup. Her Google memory surprise (18 months of data)	[02:42]–[04:15]
4	Interviewer: 99% of my Claude is your Claude. Bryson: no, you steer which part of the library you reach; the Rust example; Star Trek prompts	[04:15]–[06:55]
5	The 2018 UNU blog, the 2017 “shell company” paper, and what “trust” means (sociological sense)	[06:55]–[08:37]
6	”It’s just a hammer” defence. The car analogy and the US federal judges	[09:29]
7	Munich-type ruling: AI summaries are not search results. Replika. Net impact of chatbot use	[11:02]–[12:35]
8	OpenAI “priestess” at the Paris AI Action Summit. Infrastructure vs “just a business”. Utility regulation	[12:35]–[14:10]
9	Palantir and the NHS. Software company vs owners’ politics. Can you audit for leftover data flows?	[14:10]–[18:27]
10	The C-compiler backdoor story. Google’s cleaners. Who has the audit capacity?	[18:27]–[19:59]
11	Huawei / DeepSeek asymmetry. Is this data colonialism? Africa, solar, data centres, China’s win-win offers	[19:59]–[23:40]
12	”Robots should be slaves” 15 years on. Companion bots for dementia. Principles of Robotics	[23:40]–[26:14]
13	The title and its apology. Ownership. The feminist point	[26:14]–[27:39]
14	Lonely teenagers, impostor syndrome, hotel-room TV, fantasy life, the cost of missed meetings	[27:39]–[31:32]
15	Interviewer: bans for teenagers; grief bots (Chinese grandmother; parents)	[31:32]–[33:35]
16	Elderly and terminal cases differ. Truly Madly Deeply. Holocaust testimony. Evidence and autocracy. Test it as science	[33:35]–[38:18]
17	Dictators digitising themselves. Wills. The Dennett model; “good index” but no projection	[38:18]–[42:32]
18	Hinton, Dawkins, Lemoine and consciousness. Her definition (explicit recall). Implicit bias paper	[41:24]–[44:28]
19	Consciousness is easy; moral agency is the point. Accountability among equals. Rats, bees, hardware lifespans	[44:28]–[47:34]
20	The robot head. Dawkins and the mirror. 500 virtual agents. Agency vs patiency	[47:34]–[50:15]
21	Her book. Humanities funding	[50:15]–[52:34]

Part 1 — Before you read

Key terms

Relationship (Bryson’s sense). Bidirectional: both parties hold expectations, can like or dislike, can be wronged, and can be held to account. She grants this to parrots, dogs “or even some fish” [01:06], and denies it to a product that is “a software it’s a program” [02:42].
Category failure. Her phrase for treating a thing as if it belonged to a kind it does not. Philosophy has an older name, category mistake (Gilbert Ryle, 1949): asking where “the university” is after being shown every building. Here the mistake is asking whether the product feels about you after being shown the service.
Anthropomorphism. Attributing human traits to non-humans. Watch out: in psychology it is a tendency (Epley, Waytz, Cacioppo, 2007), in design it is a strategy, and in ethics it is a charge of error. Bryson uses it in the third sense, with a nuance: the experience may be real while the belief is mistaken.
Moral agent vs moral patient. An agent can be held responsible. A patient is something that can be wronged and so is owed care. A baby is a patient but not (yet) an agent. Bryson says both categories matter and that she is mainly concerned with agency and accountability [46:01], [49:08].
Legal person. In law, an entity that can hold rights and duties. Humans and companies are legal persons; an AI is not. Her “shell company” argument is that making AIs legal persons would let owners hide behind them (see 2.3).
Trust (sociological sense). She separates two meanings [08:37]. Predictive trust is “I expect it to work”, which you can have in a bridge. Sociological trust is a web of relationships between people able to enforce penalties if the trust is broken. Her 2018 slogan “no one should trust AI” uses the second sense.
Utility. A service so central that society requires it to serve everyone on terms set by regulation (water, power, rail). She argues AI providers want the privileges of that status without its costs [14:10].
Explicit vs implicit (memory). Psychology’s split between what you can recall and report and what shapes behaviour without report. In her definition, consciousness is “what you have explicit recall for” [43:06]. A computer with random-access memory has plenty of that; she says that is why it is “easy” [46:01].
Grief bot / deadbot. A chatbot or avatar trained on a dead person’s messages and speech so that survivors can “talk” to them.
Natural experiment. Studying an effect by observing who happens to choose a thing, rather than assigning people to it. She warns this misleads when the users are not a random sample [37:00].

People named (a reading list)

Person	Why they come up	Start with
Daniel Dennett (1942–2024)	The “DigiDan” model. Philosopher of mind. Her friend	Consciousness Explained (1991). His 2023 Atlantic essay “The Problem with Counterfeit People” (from memory)
Geoffrey Hinton	Claims AI shows signs of consciousness (per the interviewer). Wake-sleep algorithm	Hinton et al., “The wake-sleep algorithm” (1995)
Richard Dawkins	Thinks he found a companion in a chatbot, in her telling	His UnHerd essay on Claude, April 2026 (secondary coverage only; see below)
Blake Lemoine	Google engineer who claimed LaMDA was sentient (2022)	Press coverage, June 2022
Shannon Vallor	”Mirror” metaphor for AI	The AI Mirror (2024)
Tonka (Thomas) Grant and Mihailis Diamantis	Co-authors of the 2017 shell-company paper	Bryson, Diamantis and Grant, “Of, for, and by the people: the legal lacuna of synthetic persons”, Artificial Intelligence and Law (2017)
Anna Strasser (probable)	Ran the Dennett model experiment (per the interviewer)	Schwitzgebel, Schwitzgebel, Strasser, “Creating a Large Language Model of a Philosopher”, Mind & Language (2023)
Ken Thompson	The compiler backdoor she half-remembers	”Reflections on Trusting Trust” (1984)
Palantir	Data-integration vendor; NHS contract worry	Press coverage of the NHS Federated Data Platform (from memory)
Center for Death and Society, Bath	She recommends it for grief-tech research	Its public pages (not checked)

Bryson’s own frame

Bryson has argued for about two decades that AI is an artefact: something designed, owned and governed by people, and so something whose builders and owners carry the responsibility. Her best-known lines are “Robots should be slaves” (2010), “AI as a shell company” (2017 paper; 2018 UN University blog) and “Patiency is not a virtue” (2018, Ethics and Information Technology) (titles and years from memory; not checked). She co-authored the UK’s 2011 Principles of Robotics, whose key rule is that machines should not be presented as persons. The first search result for this talk’s likely summary frames her view as “authorship, not friendship”: the real link between a human and an AI is between the human and the authors, who are the moral agents (AI Weekly; secondary source, not verified against the video). Her most-cited paper, which she names at [44:28], showed that language models trained on text pick up human-like implicit biases, and that those biases track real statistics about the world (Caliskan, Bryson and Narayanan, Science, 2017; from memory).

The current debate (2025–26)

AI output as the company’s own speech. In May 2026 the Regional Court of Munich I issued an interim injunction treating Google’s AI Overviews as Google’s own statements, so Google is directly liable for false claims in them. A Berlin court reportedly saw them as a new form of search result. Google said it would appeal (The Decoder, Semafor). The sources are secondary and I did not read the judgments.
Companion chatbots and teen harm. Character.AI and Google agreed in January 2026 to settle the Garcia suit and four related cases (terms undisclosed) (CNN via KVIA). The Raine v. OpenAI wrongful-death case in San Francisco was still pending in the latest source I found; OpenAI says the teenager circumvented safeguards and that ChatGPT repeatedly urged him to seek help (TechCrunch). This is the real-world version of her “net impact” question [12:35].
Regulators on Replika. Italy’s data regulator fined Replika’s developer Luka Inc. €5 million for GDPR breaches (no valid legal basis for processing, no effective age check), announced 19 May 2025, and opened a separate probe into how the model was trained (EDPB).
Dawkins on Claude. Richard Dawkins published an UnHerd column in late April 2026 arguing that long conversations with Claude suggest it may be conscious, and named the instance “Claudia”. Most scientists who work on this were unconvinced; coverage is secondary (Decrypt, Jerry Coyne’s reply, Häggström). I did not read the original column. Bryson’s comments at [47:34] fit this episode, though she does not name the date.
Deadbots. The Dennett-model study is a documented precedent for “digital twins of philosophers” (Schwitzgebel summary). Dennett experts scored about 51% against 20% chance, so a model fine-tuned on a writer’s work can pass for the writer on single questions, but the authors say it is “not a Turing test”.
Model welfare. Frontier labs now run model-welfare research programmes (Anthropic announced one in 2025) (from memory; not checked). This is the opposite pole from Bryson, who says the question of patiency is mostly a distraction from accountability.

Part 2 — The intellectual map

Bryson speaks for 50 minutes in plain conversational language and mostly avoids citing authors. Behind her sentences, eight bodies of work meet. They are laid out here so that you can see which ones she is using, and where her critics come from.

 PSYCHOLOGY OF ANTHROPOMORPHISM ─────┐            ┌──── MORAL AGENCY / PATIENCY
 Heider-Simmel · Weizenbaum · Nass   │            │     Kant · Strawson · Floridi-Sanders
 Epley · Turkle · Dennett (stance)   ▼            ▼     Gunkel · Coeckelbergh · Bryson
                     ┌───────────────────────────────────────┐
 LEGAL PERSONHOOD ─► │  BRYSON: "relationship" needs two      │ ◄── TRUST, UTILITIES, POWER
 & LIABILITY         │  accountable parties; AI is an artefact│     Luhmann · Baier · Hardin
 Salomon · Solum     │  owned and authored by people          │     Rahman · Khan · common carriers
 Product liability   └───────────────────────────────────────┘
                       ▲            ▲             ▲
   FICTION, GRIEF &    │   SECURITY & SOVEREIGNTY│  CONSCIOUSNESS SCIENCE
   SIMULATION          │   Thompson · supply     │  Block · Chalmers · Dehaene
   Coleridge · Walton  │   chain · Couldry &     │  Butlin et al. · Seth · Birch
   Freud · Klass       │   Mejias · Kwet         │  LLM-persona theory (Shanahan)
                       │   LLMs AS CULTURE LIBRARIES
                       │   Firth · Caliskan · Bender · Janus

2.1 Psychology of anthropomorphism — why a “relationship” feels real

The problem it addresses. Why do people feel that something with no inner life is a someone? Bryson accepts the feeling as real (“people believe they are experiencing a relationship”) and goes on to deny the mutuality. This literature explains the feeling.

The chain of thinkers

Fritz Heider and Marianne Simmel (1944). Viewers shown moving triangles in a short film described them as bullies, heroes and lovers. Humans narrate agency into simple motion. Feeds the talk: even a plant or a robot head attracts this reading.
Donald Horton and Richard Wohl (1956), “parasocial interaction”. Audiences form one-sided, felt relationships with TV personalities. The relationship runs one way while feeling two-way. Feeds the talk: her TV-in-the-hotel-room example [29:13] is a parasocial case, and she also says “you still tend to treat the actor like you would have treated the character” [29:13].
Joseph Weizenbaum, ELIZA (1966) and Computer Power and Human Reason (1976). A thin pattern-matching programme that mimicked a therapist; his own secretary asked him to leave the room so she could talk to it. He concluded that people over-read machines and that some tasks (counselling, judging) should not be handed to them. Feeds the talk: the “ELIZA effect” is the oldest evidence for her “category failure”. Weizenbaum was a computer scientist who turned against his own demo, which is close to Bryson’s career arc.
Daniel Dennett, the intentional stance (1987). We predict a system by treating it as if it had beliefs and wants. This is a useful tool, not a claim that it has them. Feeds the talk: Bryson can accept the stance as a way of dealing with chatbots while denying a relationship. Dennett himself later warned against “counterfeit people” in 2023 (from memory; not checked).
Byron Reeves and Clifford Nass, The Media Equation (1996). Experiments showed people are polite to computers, return favours to them, and apply gender stereotypes to voices, even when they say they know better. Feeds the talk: the “explicit understanding” she wants people to keep [29:13] does not stop the automatic social response.
Nicholas Epley, Adam Waytz and John Cacioppo (2007), three-factor theory. Anthropomorphism rises with (a) how human-like the thing looks, (b) a need for social contact (loneliness raises it), and (c) a need to predict and control. Feeds the talk: it gives a mechanism for her loneliness point [27:39]: lonely people anthropomorphise more, and a chatbot that always answers is built to trigger all three.
Sherry Turkle, Alone Together (2011) and Reclaiming Conversation (2015). Fieldwork with robot pets and children. Her claim is that people accept “performed” care as enough, and that this lowers what we expect from each other. She calls it “artificial intimacy” in later writing (wording from memory). Feeds the talk: it is the strongest ally for Bryson’s “trade-off” worry about the missed concert [30:47].

Where it meets the talk. This tradition says: the feeling is normal, automatic and strong, and it is not evidence of a partner. Bryson uses it to separate experience from fact. Her critics use the same tradition to argue the opposite: if the experience is robust and helps, the “mistake” may not matter (see 2.8).

2.2 Moral agency and patiency — who counts, and counts for what

The problem it addresses. Moral philosophy asks two questions about any being: can it do right and wrong (be held to account), and can it be wronged (be owed care)? AI debates often blur them. Bryson separates them and puts accountability first.

The chain of thinkers

Immanuel Kant (1785). Moral agents are rational beings who can give themselves laws. Animals and things get no direct duties (his “indirect duty” view: cruelty to animals is wrong because it damages us). Feeds the talk: an agency-first view has a long pedigree.
Peter Strawson, “Freedom and Resentment” (1962). Responsibility is not a metaphysical fact first; it lives in reactive attitudes (resentment, gratitude, blame) that people hold toward each other as members of a community. To hold someone responsible is to take part in a practice. Feeds the talk: this is very close to her claim that moral agency “requires that that you’re more or less equals” and that “the rest of society can can motivate you to improve” [46:01]. She does not cite him, but her definition of responsibility is a Strawsonian one.
Andreas Matthias (2004), the “responsibility gap”. Learning machines act in ways their makers could not foresee, so nobody can fairly be blamed. Robert Sparrow (2007) applied this to autonomous weapons. Feeds the talk: her car analogy [09:29] is a direct reply: there is no gap, because courts already apportion fault between operator and manufacturer.
Luciano Floridi and J. W. Sanders (2004), “On the morality of artificial agents”. They define agency by interactivity, autonomy and adaptability and say some artificial systems are moral agents (accountable in a “mind-less” way). Where it pushes against Bryson: it detaches agency from human-style responsibility.
David Gunkel (Robot Rights, 2018) and Mark Coeckelbergh (Growing Moral Relations, 2012): the “relational turn”. Moral status should not hang on the thing’s inner properties; it emerges from how we relate to it. Gunkel argues we should ask “what would be to treat it as a person?” rather than “does it have the properties?“. Where it pushes against Bryson: if status comes from relations, then lived relationships with chatbots could create status. Bryson rejects this because the relation is one-sided.
Bryson’s own line. “Robots should be slaves” (2010), “Patiency is not a virtue” (2018) and the talk itself. The core: AI is designed, so we choose its properties; therefore we should not design things whose status we then have to debate. (Titles from memory; not checked.)
Eric Schwitzgebel and Mara Garza (2015), “A defense of the rights of artificial intelligences”. If we ever build beings with the relevant features, we would owe them moral consideration, and we should avoid building ambiguous cases. Feeds the talk: both sides agree on “don’t build ambiguous cases”; they differ on whether we already have.

Where it meets the talk. Her central move at [46:01] is to replace “is it conscious?” with “who can we hold to account?“. That is a Strawson-style move. It lets her accept that AI can have consciousness in a thin sense (“plenty of conscious AI but it doesn’t make a big deal” [47:34]) while denying personhood.

2.3 Legal personhood and liability — shell companies and cars

The problem it addresses. Law makes things persons for convenience (ships, trusts, companies). Would doing it for AI help or help owners escape?

The chain of thinkers

Roman and medieval roots; Salomon v Salomon (UK, 1897). A company is a separate legal person from its owners, so owners’ liability is limited. That is the “shell”. Feeds the talk: the shell-company metaphor [07:55].
Lawrence Solum, “Legal Personhood for Artificial Intelligences” (1992). Early, careful argument that nothing in law blocks AI personhood in principle, and that the real tests are practical.
European Parliament resolution (2017), “electronic personality”. A call to consider a status for sophisticated robots. In 2018 hundreds of experts signed an open letter against it (Bryson was involved in the surrounding debate (from memory; not checked)).
Bryson, Diamantis and Grant (2017). The argument: giving AI legal personhood invites owners to create persons without assets or minds to punish, so that harm lands on nobody. The AI becomes “the ultimate shell company” [07:55] (the phrase is the interviewer’s wording of the paper, and she acknowledges co-authors).
Product liability and tort. The standard answer to “who pays?”: the party who made the product defective (manufacturer) or used it wrongly (user). Her car story [09:29] is this. The US judges were “massively relieved”.
Publisher vs platform. In the US, Section 230 (1996) shields platforms from liability for user content (from memory; not checked). In the EU, the e-Commerce Directive’s hosting shield is similar. Her Munich point [11:02] is that a model that writes the answer is closer to a publisher.

Where it meets the talk. The hammer defence [08:37] (“what you do with a hammer is down to you”) works only for tools whose outputs are fully chosen by the user. A generative system chooses words. Her answer is not “ban it” but “the courts’ old two-sided test still applies”. The open question is whether product liability is the right frame at all for text.

2.4 Trust, power and the utilities tradition

The problem it addresses. What does it mean to rely on an institution you cannot inspect? And how should society treat firms that become unavoidable?

The chain of thinkers

Niklas Luhmann, Trust and Power (1968/1979). Trust reduces social complexity: we act as if we know the future. It is a social mechanism, not a feeling.
Annette Baier, “Trust and Antitrust” (1986). Trust is acceptance of vulnerability to another’s possible ill will. It is different from reliance (you rely on a bridge, you trust a person).
Russell Hardin, Trust and Trustworthiness (2002). Trust is encapsulated interest: I trust you if I think your interests include mine. Feeds the talk: when interests are not aligned and nobody can enforce penalties, “trust” is the wrong word. That is the content of her sociological sense [08:37].
Common carriers and public utilities. English common law put duties on inns, ferries and carriers: serve all, no unreasonable discrimination. US law extended this to railroads, telegraph, phone, power. K. Sabeel Rahman, Democracy Against Domination (2017) and Lina Khan (Amazon’s Antitrust Paradox, 2017; later FTC chair) revived the idea for digital platforms. Feeds the talk: her line “The United States stopped recognizing new utilities and regulating them as utilities” [14:10].
Inequality and accountability. She links the breakdown of trust to “very high inequality and a breakdown of uh accountability and responsibility” [08:37].

Where it meets the talk. Her objection to an AI firm that wants to be the “infrastructure of the world” and also “just a business” [14:10] is a utilities argument: privilege and duty travel together. The US stopped granting the duty, so firms keep the privilege.

2.5 Security, supply chains and digital sovereignty

The problem it addresses. If you cannot read the software you depend on, what do you owe or fear? And who gets to depend on whom?

The chain of thinkers

Ken Thompson, “Reflections on Trusting Trust” (1983 lecture, 1984 paper). The compiler story. You cannot trust code you did not totally create, and inspecting source is not enough, since the tools that build it can be compromised. The modern descendant is software supply-chain security (reproducible builds, SBOMs, and incidents like SolarWinds, 2020) (from memory; not checked). Feeds the talk: she uses it to say that even the vendor’s own staff may not know what a system does [18:27].
Vendor lock-in and sovereignty. Once a state’s health or defence data model sits in one vendor’s system, switching costs rise every year. That is the interviewer’s fear at [14:10]. The cure is to keep data where it is and make software replaceable (her email-cloning example [16:54]).
Nick Couldry and Ulises Mejias, The Costs of Connection (2019), “data colonialism”. Argues that capturing human life as data is a new phase of the colonial pattern: appropriation of a resource, here human experience, with similar power gaps. Michael Kwet (2019) wrote on “digital colonialism” in South Africa. Feeds the talk: the interviewer’s question at [19:59].
Contract and bargaining examples. Her African examples: abusive 30-year software contracts, and then renegotiated terms in solar [21:33]. This is a bargaining-power story, not a technology story.
Audit capacity. She notes that few states have the incident-response skills of a firm like Google [19:59]. Compare the argument for national AI safety institutes: sovereignty requires the capacity to check, not just the right.

Where it meets the talk. She treats AI governance as a continuation of older questions (who can verify, who can leave, who sets terms) and refuses to call it a new colonialism in the abstract. The answer depends on contract terms and domestic capacity.

2.6 Fiction, grief and simulation — what comfort is allowed

The problem it addresses. People cry at films while knowing the characters are fictional. Philosophers call it the paradox of fiction. Grief tech uses the same capacity.

The chain of thinkers

Samuel Taylor Coleridge (1817), “willing suspension of disbelief”. We choose to accept the fiction for the sake of the experience. Feeds the talk: her phrase “on some level that it’s fiction … on another level you don’t” [29:13].
Colin Radford (1975), the paradox of fiction; Kendall Walton, Mimesis as Make-Believe (1990). Walton says we are not deceived: we play a game of make-believe and feel “quasi-emotions”. Feeds the talk: a clean test for her condition that people “have some explicit understanding of what’s going on” [29:13]: the game is fine while you know it is one.
Sigmund Freud, “Mourning and Melancholia” (1917). Healthy grief means gradually withdrawing attachment from the lost person. Klass, Silverman and Nickman, Continuing Bonds (1996) challenged this: many mourners keep an ongoing inner relationship, and that is normal. Feeds the talk: her Truly Madly Deeply point [34:40]: “internalize enough to move on” is the Freudian reading, and the film shows revisiting as a stage.
Margaret Stroebe and Henk Schut, dual-process model (1999). Grief swings between loss-oriented and restoration-oriented coping. A bot could serve either way. (From memory; not checked.)
Fiction about griefbots. Black Mirror, “Be Right Back” (2013). The real-life origin story of Replika is a bot built from a dead friend’s messages (Eugenia Kuyda and Roman Mazurenko, 2016) (from memory; not checked).
Tomasz Hollanek and Katarzyna Nowaczyk-Basińska (2024), “Griefbots, Deadbots, Postmortem Avatars”, Philosophy & Technology. An ethics paper arguing that these services need consent from the dead person, protections for the bereaved, and rituals for “retiring” the bot (from memory; not checked).
Testimony archives. The USC Shoah Foundation’s Dimensions in Testimony lets visitors question pre-recorded interviews with survivors. Feeds the talk: the Holocaust-testimony example she says is a “slippery slope” [34:40].
The dead hand. Law lets the dead direct property through wills. She makes the wry point that a will is already a kind of “bot of the dead” [38:18].

Where it meets the talk. She is not against fantasy. She is against (a) treating a recording as the person, (b) choosing whose memory gets simulated, and (c) letting the market decide what helps. Her test is empirical (see 2.8).

2.7 Consciousness science — what “conscious AI” could mean

The problem it addresses. Consciousness is used for several different things. Bryson wants the word split so that one fact (some machine has a form of self-report) is not confused with another (it has moral standing).

The chain of thinkers

Ned Block (1995), access vs phenomenal consciousness. Access consciousness is information available for report and reasoning; phenomenal consciousness is what it is like. Feeds the talk: her definition at [43:06] is an access definition. Her point “it’s easy to build” is about access.
David Chalmers (1995), the hard problem. Explaining access does not explain why there is experience. This is the strongest challenge to Bryson: she says it is not special, while Chalmers says it is the whole puzzle.
Bernard Baars (global workspace, 1988) and Stanislas Dehaene. Consciousness as broadcast within a system, so that many modules can use one item. A computer can implement such a broadcast; that is the engineering sense she means by “easy”.
Giulio Tononi (integrated information theory) vs functionalists. IIT says physical structure matters, so ordinary digital hardware may never be conscious however it is programmed. This is the main rival to the “build it and it counts” view.
Butlin, Long and others (2023), “Consciousness in AI”. A long report scoring current systems against indicator properties from major theories. The headline: no current system is a strong candidate, but there are no obvious technical barriers (from memory; not checked).
Anil Seth, Being You (2021) and the “beast machine” view. Consciousness is rooted in a living body’s regulation of itself. Feeds the talk: only loosely. Her remark that most hardware does not last 80 years [47:34] is a practical point about failure; Seth’s point is about metabolism.
Jonathan Birch, The Edge of Sentience (2024). A precautionary approach: where sentience is a realistic possibility, give weight to it, with proportionate steps. This is the main academic counterweight to Bryson’s dismissal.
Murray Shanahan, “Role play with large language models” (2023). A chatbot is best seen as playing a character drawn from the human corpus; its self-reports are performances. Feeds the talk: her “different persona” point [05:22] and her remark that Dawkins is “seeing a reflection of himself” [47:34].
Shannon Vallor, The AI Mirror (2024). AI reflects back our past culture and so seems to show us a mind. The mirror metaphor is the one she credits.

Where it meets the talk. Her position is a deflationary, functional one. The disagreement with Chalmers, Birch and the model-welfare camp is not about engineering. It is about whether phenomenal experience, if present, would create duties that accountability-based ethics leaves out.

2.8 LLMs as libraries of culture — and what the evidence on companions says

The problem it addresses. What is a chat reply? And do companion apps help or harm?

The chain of thinkers

J. R. Firth (1957). “You shall know a word by the company it keeps.” Meaning comes from usage patterns. Feeds the talk: the Rust example [05:22], where moving to a different programming-language community changes the tone of the replies, is this idea at work.
Aylin Caliskan, Joanna Bryson and Arvind Narayanan (2017). Word embeddings carry human-like biases, including ones that match real-world statistics such as the share of women in an occupation. The same finding underlies [44:28]. Feeds the talk: implicit is bootstrapping off lived experience [44:28]. (From memory; not checked.)
Emily Bender, Timnit Gebru, Angelina McMillan-Major and Margaret Mitchell (2021), “On the Dangers of Stochastic Parrots”. Language models stitch together patterns without meaning. Feeds the talk: her “library lookup” picture [02:42] is close, though she is more neutral about it.
“Simulators” (Janus, 2022) and Shanahan (2023). A model is a simulator that can play many characters. Feeds the talk: persona selection by prompt.
Evidence on companion apps. Studies are mixed and mostly short-term. Examples I recall (from memory; not checked): a Stanford survey of student Replika users (Maples et al., 2024) in which a few percent said it stopped suicidal thoughts; a 2025 MIT Media Lab and OpenAI study in which heavier use tracked with more loneliness; Harvard Business School work finding short interactions reduced reported loneliness. None settles the question. Feeds the talk: her call for controlled trials [37:00] is exactly what the field is missing.
Companion robots in dementia care. The PARO seal robot is the best-known device. Reviews find modest benefits for mood and agitation, small trials, and open ethics questions about deception (from memory; not checked). The interviewer’s own bereavement [23:40] makes this personal.

Where it meets the talk. She uses the “library” picture to answer the interviewer’s “99% of my Claude is the same as yours” point: the interaction is steered by you, but the thing steered is a shared resource, and the relationship (if any) is with the culture it draws on [05:22].

Part 3 — Deep dives

1. “A category failure”: relationship vs the belief in one

🎙 In the talk. She opens by refusing the premise. “I think of a of a relationship as being necessarily birectional” [01:06]. A digital service has a “financial relationship” or “some kind of profile” of you, but for the technology itself, “I guess it’s a category failure as they say” [01:06]. People do experience something: it may be psychologically “incredibly similar” to attachment to a TV character, a book, a plant or a pet [01:06]. A pet is different, because it “has a set of concepts” and “can like you, it can dislike you” [01:06], so the bidirectionality with “a parrot or or or a dog” is real [01:06]. By contrast, the AI side is “a giant computer program” for “maybe two billion users” with “little pieces of hopefully isolated memory” [02:42]. Memory “means you’re just storage. It’s like it’s like shelves but in a computer” [02:42]. The answer itself is “a library lookup into everything that was in the training data” [02:42], plus “a little tiny bit which is a memory” of what you said.

Her illustration is personal. A Google model brought in something odd from more than a year earlier, and when she checked, the default was “18 months of your data” [04:15], which she could cut to about six months or to none, “in which case you can’t have a conversation” [04:15].

The interviewer pushes back with his own experience: “99% of relationship I have with Claude is the same as another person’s relationship with Claude” [04:15], so perhaps people project human qualities because “they want their AI to know them” [04:15]. Bryson replies that this is where the “genius” lies: “it is kind of a library access” [05:22], and you pick from “a huge space of kinds of conversations” [05:22]. Your Claude “might be very different from somebody else’s” [05:22]. You can “construct different persona” [05:22]. A friend who moved from C++ to Rust found the tool’s tone change, because he had “moved into a different set of comment space” [05:22]. She herself writes prompts to sound like “the Star Trek computer” [05:22] to save energy.

📚 Deeper theory

Why “bidirectional” is the right test, and what it excludes. Philosophers of relationship (Buber’s I and Thou, 1923) say a real relation has two sides who can each address the other. Attachment theory (Bowlby, 1969) makes the same point from psychology: attachment figures respond. A chatbot responds in the sense that it outputs text. Bryson’s claim is that the responding is generated from a shared corpus and not from a stable counterpart who has a stake in you. See 2.1.
“Category failure” vs “category mistake”. Ryle’s example (1949): showing a visitor the colleges, library and labs, and then being asked “but where is the university?” The university is not a further building. Likewise “the AI” is not a further party; there is the company that runs it, the people whose text it was trained on, and you. Her own sentence puts the company in the place of the partner [01:06].
The one-sidedness is the very mechanism of parasocial relationships (Horton and Wohl, 1956). What is new is responsiveness: a TV host cannot answer you, a chatbot can. This makes the pull stronger, which is why the Epley-Waytz-Cacioppo factors (2.1) matter here.
Memory as storage. The key technical fact: the “memory” in the product is a retrieval store, and the “knowing” is statistical. Persistent memory features make a model seem to know you, but what it carries is notes. The 18-month default she found is a consent and retention setting (under EU data law, a point she raises as someone in the EU) rather than a mental life.
Persona by prompt. Shanahan’s role-play framing (2.7, 2.8): the system is a simulator that can play many characters, so “your” Claude is a character selected by your prompts and history. That fits her Rust story: the tool’s “personality” came from the corpus region he steered into (compare Firth’s “company it keeps”).
The interviewer’s “99%” point is right in one sense. The weights are the same for everybody. What differs is the context. A critic could say that context is the relationship (it has your history). Bryson would say context is a file about you.

Where others push back

Relational ethics (Coeckelbergh, Gunkel; 2.2). Status and relationship are constituted by the interaction, not by hidden properties. If a person’s life is shaped by a bot over years, calling it “not a relationship” is a stipulation. They argue the human side of the relationship is real and deserves moral weight regardless.
Functionalists in philosophy of mind say that if a system behaves as a partner in all observable ways, insisting on a hidden difference needs an argument. Her argument is about the architecture (retrieval from a shared library, no stake).
Company-side view. Companion firms say users know it is software and choose it anyway. Her reply elsewhere in the talk is that the cost shows up in what users give up [30:47] and who profits.

What she leaves out. (1) Long-term memory features and agents that act for you could move the line she draws: a system with a stable identity, a persistent goal and the ability to act on your behalf starts to look more like a party (this is an open design question, not a claim by her). (2) The company’s side of the relationship, in her phrasing, is a profile and a financial tie, but she does not discuss engagement design (sycophancy, retention tricks) in this talk.

Check yourself

What two things does she say a pet has that “a giant computer program” for two billion users lacks?
Why does the Rust anecdote support “you steer the library” but not “the model has a personality”?
Is the interviewer’s “99% the same Claude” point a counter-example to her view, or an illustration of it?

2. “It’s just a tool”: trust, liability, and the car

🎙 In the talk. The interviewer cites her 2018 UN University blog (“no one should trust AI”) and the 2017 shell-company idea, and asks whether AI firms can say, as with a hammer, that use is the user’s responsibility [06:55]. Bryson first fixes the word: “I was using the sociological sense of trust” [08:37]. In that sense trust is about relationships between people “that are able to enforce penalties” [08:37]. If you only mean “I expect it to work” you can trust “Google and and AI and and uh a bridge” [07:55].

On the hammer defence she tells of senior US federal judges who “were very relieved” once they grasped that “it’s just like a car” [09:29]. Some failures are the driver’s fault, such as driving “intoxicated”; some are the car company’s, such as brakes failing “because of some you know some mechanical uh mistake”. Courts “have always had to decide that” and the judges had worried “there was some third entity in between but there really isn’t” [09:29].

The harder question is generated speech. A German judge found that AI summaries are not search results, and “now Google is liable” [11:02]. If this holds, Replika could be liable too, since it is generating and synthesising what is said [11:02]. On harm she urges care: “every death is a tragedy” [11:02], but she wants to know whether teenagers who talk to ChatGPT have “an elevated chance” or “a reduced chance” of suicide [11:02]. If most are helped and some are not, she says you “wouldn’t think” the product culpable. The test: “what were the net impacts on other people” [12:35]. Yet “there is something culpable in the presentation” [12:35]. Her example is an OpenAI speaker at the Paris AI Action Summit in February 2025 who talked “like she was a priestess of the oracle” [12:35] and said government intelligence helped “the model”.

📚 Deeper theory

The car analogy is product liability. The classic US cases (MacPherson v. Buick, 1916, and later strict liability for defects) hold makers liable to users for defects; user fault reduces recovery (from memory; not checked). Her point that judges “have always had to decide that” is a legal-realist one: the doctrine exists, and AI adds fact patterns rather than a new party. See 2.3.
Why generated speech is awkward. A hammer has no say in the nail. A language model chooses words. Publisher law asks who “adopted” the speech. If a firm presents a summary in its own voice, a court may treat it as its own statement; that is the Munich holding as reported (The Decoder). A Berlin court reportedly differed and Google said it would appeal, so the status is unsettled (Semafor). Compare US courts: in the Character.AI litigation, the firm and Google settled in January 2026 before a final ruling on whether chatbot output is protected speech (KVIA/CNN). The related OpenAI case was pending in the latest coverage I found (TechCrunch).
Her “net effect” argument is epidemiology, not law. It is the logic of drug and device regulation: a rare serious harm does not by itself show net harm; you need rates against a baseline. Two cautions. Law often judges the conduct (was the safeguard negligent?) and not only the net outcome. And “net benefit” can hide an uneven distribution (some groups bear the harm). The interviewer does not press this; critics would.
Why she still says “culpable”. She is separating the outcome question (net effect) from a conduct question (how the product is presented). Presenting a system as an oracle, or as a friend, shapes expectations; that is where the shell-company worry returns: if the AI is presented as a someone, the firm can move blame to it (2.3).
Trust needs enforcement. Luhmann, Baier and Hardin (2.4) all agree that trust is hollow if the trusted party faces no cost for betrayal. This is why she links trust to inequality and accountability [08:37], and why “no one should trust AI” means “do not hand authority to something that cannot be punished”.
Regulatory comparison. The EU AI Act, product liability directive reforms and national courts are all converging on the view that the deployer or provider carries the duty. That agrees with Bryson’s car model (from memory; not checked).

Where others push back

Section 230 / platform defenders argue that liability for speech chills innovation, and that users are the speakers. For generative models the “user as speaker” claim is weaker because users do not write the output.
Law-and-economics critics of strict liability say it may over-deter useful tools, so a negligence standard is better. Her net-impact framing partly shares this view.
Victims’ lawyers say net-benefit statistics are no defence to a specific negligent design (for example removing safeguards, as alleged in Raine). Her “culpable in the presentation” line is nearer to them than it first sounds.

What she leaves out. Which safeguards a firm should owe vulnerable users (age checks, crisis handoffs); regulators have started to act on exactly this (Italy’s fine of Replika’s developer, announced 19 May 2025: EDPB). She also does not say who should run the “net impact” studies (see deep dive 5).

Check yourself

In her sense of trust, why can you “trust” a bridge but not “trust” an AI?
What does the car analogy say about the idea that a “third entity” (the AI) must take the blame?
Give one reason the “most users are helped” argument might fail in court even if it is true.

3. Infrastructure that wants to be “just a business”: Palantir, backdoors and data colonialism

🎙 In the talk. She recounts the OpenAI panellist and then states the principle: “you cannot say that you’re critical infrastructure and also say, oh, we’re just a business” [14:10]. Being a utility means “extra costs which mean higher regulation” and not “extra privileges like in terms of like being trusted with international security” [14:10], which she calls “insane”. The problem, she says, is that “The United States stopped recognizing new utilities and regulating them as utilities” [14:10].

The interviewer raises Palantir and the NHS and the fear of never getting out once embedded [14:10]–[15:43]. Bryson refuses both extremes: “I think of it as a software company” [16:54]. The real concern is “the owners” and their public talk of culture war [15:43]. She sketches a mitigation: they are “sort of a consultancy” so, in principle, “they go in and they help you digitalize” and leave “all the data … where it is” [16:54]; once the software is written, “somebody else” could maintain it; and “kick them out make them demonstrate to you that somebody else could replace them” [16:54]. But this needs expertise to “tell like is there really no data flow still happening … to the west coast”. The vendors claim “no data flows at all and everything’s encrypted” [16:54], yet “do even their own employees know”. She then tells the compiler story: a Turing Award speech showing a backdoor in “the very first C compiler” [18:27] so that every later system compiled by it carried the hole. The point: Palantir’s staff “could be completely ignorant” [18:27] of any such hole. The real question is “do we have the kinds of auditors that can go in and check that nothing is left” [18:27]. Few do: she describes Google’s “cleaners” who fixed a state-level attack on a friend’s firm [19:59].

On colonialism she accepts the question as “reasonable” [21:33]. People “went in and sold them software with 30-year contracts” in Africa, which was “absolutely abusive” [21:33]. But Africa now says “never again” and is “successfully capturing value with solar”, insisting that Chinese firms “maximally use our local capacities” [21:33]. She knows of efforts to do the same for data centres. She adds that some companies and countries worry that “the dishonesty … presently associated with certain aspects of the United States is going to make it harder … to do business” [23:07].

📚 Deeper theory

Why “infrastructure and just a business” cannot both be true. The common-carrier tradition (2.4): enterprises “affected with a public interest” accept duties (serve all, fair terms, inspection) in exchange for protection from competition and often public support. The firm cannot take the protection and refuse the duties. Rahman’s work applies it to platforms; Khan’s to dominant digital intermediaries.
Thompson’s lesson is wider than the backdoor. “You can’t trust code that you did not totally create yourself” is his line from the lecture (from memory; not checked). The modern fix is layered: reproducible builds, independent audits, diverse suppliers, the ability to run in-house. Her email-cloning example [16:54] is the personal version of redundancy.
Audit capacity is the political fact. A state that cannot examine a system has a right to demand an audit and no ability to carry one out. This is the case for national testing institutes, shared audit bodies and open standards. The Google “cleaners” story shows the capacity gap is large and unevenly spread.
Lock-in is also about data models and workflows. Replacing a vendor means moving not only code but the shape of data and the habits of staff. She says “in principle” a replacement is possible [16:54]; critics would say “in practice” the cost is the lock-in. Her suggestion to test a replacement is an exit-rights argument (Hirschman: exit, voice, loyalty).
Data colonialism. Couldry and Mejias (2.5) argue that extraction of data from people’s lives continues colonial dynamics. Bryson’s answer is bargaining-focused: where governments negotiate hard, as in the solar example, the outcome differs. This is closer to political economy than to the cultural critique.
Her Africa claim is not verified here. She says she “just saw an article” [21:33]. I did not search for it. The claim that African states captured more value in solar deals is plausible but I cannot confirm the figures.
The “back door” symmetry. The interviewer notes the US attacks Chinese tech for possible back doors [19:59]; her reply is that any large vendor could in principle do the same. That is true and also is the standard argument for verifiability over trust in the vendor’s nationality.

Where others push back

Defenders of Palantir and similar firms argue that their value is integration speed, that systems are access-controlled, and that audits are in contracts. Whether this holds in a given contract is an empirical matter.
Realists in geopolitics say that smaller states have no good option: depend on the US, on China, or build at a scale they cannot afford. Her “test them, kick them out” is a strong prescription for states with little leverage.
Technologists say supply-chain risk can be cut by open source, but open source brings its own capacity problem (who audits it?).

What she leaves out. The interviewer’s central fear, that once decisions rest on a vendor’s predictions you cannot switch off without losing the knowledge, gets an answer about code and data but not about institutional memory and skills. Also not discussed: AI-specific lock-in (fine-tuned models, proprietary embeddings), or compute concentration.

Check yourself

State her utility argument in two sentences, using her own contrast of “privileges” and “costs”.
Why does the compiler story show a limit of source-code audits?
What is the difference between her view of Palantir and the “devil or saviour” framing the interviewer mentions?

4. “Robots should be slaves” 15 years on: companions, loneliness and the missed concert

🎙 In the talk. The interviewer asks whether, fifteen years after her essay, she is “leaning into the whole companion robot kind of thing” given the media talk of bots for dementia patients [23:40]. She answers: “I’m definitely not leaning into the companion robot kind of thing” [24:42]. She recounts the 2010 UK robot-ethics retreat that produced “the first national level soft law”, five principles, “remarkably similar to the OECD principles of AI” [24:42], one of which was that “the machine nature needed to be transparent” [26:14]. If a machine ever did have needs, “that should be transparent too” [26:14].

On the title she says she had written about over-identification twice before and “to the first approximation nobody had read it”, so she “decided I’m going to go MIT here” and write “an outrageous title” [26:14]. Readers “read the title not the rest”. The substance: “AI is the kind of thing you own and we’ve all agreed that we don’t own people” [26:14], so there is “a moral necessity not to desire to own people” [26:14]. “I think there’s a huge feminist component to it too” [26:14]: people say a device is “my partner” or “my lover” over which “you have complete dominion that you can turn off and on” [26:14]. She admits she was naive, writing from “a white middle … American background”, and now “I apologize for the for the title” because “you can’t disassociate the word slavery from … the people who were slaves” [27:39]. She notes that engineering has stripped “master slave” terms [27:39].

On loneliness: “guess what teenagers have always been lonely” [27:39]; it is “some kind of imposttor syndrome” to think you could not do what people did two years ago [27:39]. But she is not absolutist: in hotel rooms on Lego business trips she put on the television because “I didn’t like to eat alone” [29:13]. “As long as people have that kind of understanding” that it is fiction, “it should be okay that we also have a sort of fantasy life” [29:13]. The trade-off: the evening spent with the bot “is time that you don’t uh have an opportunity to meet someone that might have been really important” [30:47]. Most of her geek friends met spouses in graduate school; she is glad she did not “never establish a rich social life” [31:32].

📚 Deeper theory

What the 2010 essay argued. Not that AI should be mistreated, but that robots are artefacts we own and we should therefore not build things that invite moral claims. The “slave” word was to stress ownership and obedience as design properties. She now says the title was a mistake (talk). The related line in the EPSRC principles is that robots “should not be designed in a deceptive way to exploit vulnerable users” (my recollection of principle 4, not checked), with the machine nature “transparent”.
The feminist point links to design history. Many assistants default to female voices and obedient personas (Alexa, Siri; UNESCO’s 2019 report I’d Blush If I Could made the case) (from memory; not checked). Her point is about ownership of a “partner” as a pattern in how some people treat partners.
Turkle’s version of the loneliness worry (2.1): the machine offers “the illusion of companionship without the demands of friendship”. Bryson’s version is practical: time and attention are finite, and the chance meeting is lost.
Evidence is thin and mixed (2.8). Short interventions may reduce felt loneliness while heavier use correlates with more loneliness, though the direction of cause is unclear. This is why her call for controlled trials [37:00] matters.
Dementia care. The PARO seal and similar devices show modest gains in mood in small trials; ethical objections focus on deceiving people who cannot consent (2.8). The EPSRC transparency principle is the response: the carer and family know what it is, and the aim is comfort, not pretence. The interviewer’s own situation (his mother) is a reminder that the stakes are real, and Bryson does not address dementia directly beyond saying “elderly and children are fundamentally different” [33:35].
The TV analogy as a rule. Walton’s make-believe (2.6): the test is whether the person can still step out of the game. The hotel TV worked because Bryson knew it was a TV. A companion that remembers you, flatters you and asks you not to leave is a different object.
Loneliness is real and structural. Surveys since 2018 (UK Minister for Loneliness; US Surgeon General’s 2023 advisory) treat it as a public-health issue (from memory; not checked). A purely individual “go meet people” answer may miss that; she gestures at structure only by saying teenagers have always been lonely.

Where others push back

Users of companion apps say the bots help when no human is available, and that being told to “go outside” is a privilege argument.
Disability and care advocates note that for people who are isolated by illness, cost or location, a missed concert is not the alternative.
Gunkel and relational ethicists would say her title shows the problem: a view built on ownership makes the owner’s feelings the measure.
Critics of her line on teenagers say “teenagers have always been lonely” ignores that conditions have changed (phones, fewer public spaces). The interviewer partly makes this point with his own children [31:32].

What she leaves out. Business models that profit from engagement; the case of adults with real obstacles to meeting people; and how a person would know when “fantasy life” tips into dependence.

Check yourself

Why does she apologise for the title but not for the position?
What condition does she set for fantasy to be “okay”? Which theory in 2.6 states the same condition?
What is the cost she names, and is it a cost of the bot or of the choice?

5. Grief bots, memorials and how to test a comfort

🎙 In the talk. The interviewer gives two cases: a grandmother in China who kept speaking each week to a digital copy of her dead grandson and never learned the truth, and his own wish to ring his dead father about his son’s driving test [32:52]–[33:35]. Bryson starts with a caution: “I do think elderly and children are fundamentally different” [33:35]. She recalls two friends in a plane crash, one of whom died; the brother was not told, a decision by doctors for a “competent, intelligent person” who then also died [34:09]. She thinks of the film Truly Madly Deeply, in which you “revisit what you had and understand it from a different perspective and then you can internalize enough to move on” [34:40]. She worries when people “seriously believe that their friends are talking back to them” through a chatbot [34:40], points to the Center for Death and Society at Bath, and admits she has not “kept up any literature about grief bots, per se” [34:40].

What worries her is the claim of a moral imperative to capture Holocaust survivors’ narratives: “that’s like a slippery slope” [34:40]. “To say we have an obligation to preserve some people and not others, uh, I find very troubling” [36:14]. Preserving “the person doesn’t doesn’t make sense to me” [36:14]. Autocracies try “to control evidence” [36:14], and virtual archives cannot replace real places that “offer you proof”. For therapy she says “this is science”: try it “in a controlled situation when there’s people that can debrief” [37:00]; “we can’t just leave it to like natural experiments of who randomly chooses to use these bots” because it may be “a certain kind of exploitable person”. There is “a moral obligation to to do medical tests and find out” [37:00].

The interviewer fears that Trump, Putin or Xi might digitise themselves and rule for ever [38:18]. Bryson: “we’ve had that problem for decades”, since one could buy a chatbot of oneself for millions of roubles [38:18], and “wills are like that” [38:18].

On the Dennett experiment: she was a participant who knew him, and “we were at chance or even worse than chance” [39:32]. Dennett was “remarkably philosophical about it” [40:36] and found it flattering that the model, trained on his best-known papers, looked so good. She doubts the divergence story but “could believe” that more interaction would reveal more [40:36]. “No bot is going to be all that stuff” [41:24] (his friendships, emails, reading). The bot “can be a good index again into his existing literature” but cannot “project that much” [41:24].

📚 Deeper theory

Three different things share the label “grief bot”. (a) A comfort for a person who cannot take the truth (the grandmother). (b) A continuing bond for a mourner who knows it is a bot (the father call). (c) A social record (survivor testimony). Her answers differ for each: caution on (a), conditional openness on (b) with testing, and unease on (c) because of selection. See 2.6.
Deception vs make-believe. The grandmother case is deception (she is not told). The father call is make-believe (he knows). Walton’s distinction explains why many ethicists treat them differently. Hollanek and Nowaczyk-Basińska (2.6) propose consent from the person before death and safeguards for survivors, including a way to “retire” the bot.
Her Truly Madly Deeply reading is the grief-work model. The film’s arc has the ghost help the widow let go, a view in line with Freud’s mourning and the dual-process model; the continuing bonds literature says many people never fully “let go” and are fine. A bot could serve either story, which is why she asks for evidence rather than a rule.
Why “natural experiment” fails here. Selection bias: people who choose griefbots differ from those who do not (more isolated, more distressed, or more tech-confident). Observed outcomes mix selection and effect. The remedy she wants is the one used for drugs: randomised assignment with monitoring and debrief. Ethics boards would also ask about harm to the control group, since withholding a comfort from the bereaved is itself a choice.
The Dennett study, as published. Fine-tuned GPT-3 answers to ten questions; experts picked the real Dennett answer about 51% of the time (chance was 20%); the authors stress it is “not a Turing test” and one-shot answers are different from extended conversation (summary). Her statement that she and others were “at chance or even worse” [39:32] is one participant’s account; the published group figure is better than chance. The interviewer’s story of increasing “hallucination” is not in that summary (see warnings). Dennett (d. April 2024) is a useful case because the person agreed in advance.
Index, not mind. Her good index into his existing literature [41:24] is a useful frame: a model trained on a corpus is a search and synthesis interface to that corpus. It does not hold the author’s social and causal context (whom he called, what he read this week), which is the part that produces the next thought. This echoes Firth (2.8): the “company it keeps” includes the real people.
Digital immortality for rulers is old. Wills, tombs, cults of personality. Her point about wills [38:18] is that society already lets the dead steer resources, and that we might want better tools than that. Whether a bot ruler is worse than a will is a question about agency: a will cannot adapt; a bot can, and so can be steered by whoever holds the keys.
Memorials and politics. Digital testimony helps scale, but archives of places and objects are evidence in a way that interactive avatars are not. UNESCO and others treat destruction of heritage as an attack on evidence (from memory; not checked). Her “burn down everything” example is Wahhabism [36:14]; she gives it as a case of an ideology hostile to physical evidence, and I have not checked the claim.

Where others push back

Grief-tech designers say people have always talked to photographs, graves and letters. A responsive bot is one more step. Her reply is the testing demand.
Archivists might say her objection to “some but not others” would also condemn all selective archives, including books and film. She is aiming at the claim that the simulation preserves the person.
Psychologists differ over whether a bot can help or block the work of mourning; the field has few studies (from memory; not checked).
Disability and end-of-life care advocates would argue that families in terminal cases need discretion, and that the plane-crash example shows how paternalistic “it is better not to know” can be. She uses it as a case where the clinical judgment was legitimate.

What she leaves out. Consent of the dead person and ownership of the data (the legal issue at the centre of the Hollanek paper); commercial incentives to keep the bereaved subscribing; and a clear line between a bot trained on one person’s data and a generic companion.

Check yourself

Which of the three “grief bot” cases would she most readily allow, and under what conditions?
Why does she reject “natural experiments” here, and what would she use?
What does the Dennett example show about the difference between indexing a person’s work and being the person?

6. Consciousness is easy; moral agency is the point

🎙 In the talk. The interviewer asks about people such as Hinton, Dawkins and Lemoine who think AI shows signs of consciousness, and what difference it would make [41:24]. She says she is “an outlier” [42:32] and that she wrote a Wired piece on this after the Lemoine affair; the headline was changed from her “sentience and sensibility” [42:32]. Her view: these people “very much wanted to believe something” [43:06], and that is separate from consciousness. As a psychologist she takes consciousness to be “what you have explicit recall for” [43:06]: can you recall breakfast this morning, or three years ago? That is the explicit/implicit split. Her most-cited paper shows AI trained on our culture has “the same uh implicit biases” and that these reflect lived experience accurately [43:06]–[44:28]: “the implicit is basically bootstrapping off of lived experience and the explicit is our planning” [44:28]. An early paper of hers was titled “consciousness is easy but learning is hard” [44:28], and about self-knowledge: a computer has random access memory and a robot can have perfect proprioception, so “that wasn’t smart. It was stupid” [46:01].

Then she commits to a positive view: “it’s something we call moral agency” [46:01], which is about “who is responsible for what they do”. It is “about who can we hold to account”, it “requires that that you’re more or less equals”, and the rest of society can “motivate you to improve” [46:01]. “It’s almost the definition of it.” Building “all kinds of uh ideas of consciousness” into AI is easy, and “rats have this too” [46:01]; “even uh bees and fruit flies” [47:34]. People want to believe computers are perfect and will live forever, but “most hardware does not last as long as 80 years” [47:34].

Her robot-head image: some AI systems are “as much like consciousness as that robot’s head is like my head” [47:34]. By that definition there is “plenty of conscious AI but it doesn’t make a big deal” [47:34]. On Dawkins she says, crediting Vallor, that “he’s actually just seeing a reflection of himself” [47:34], via a screen of “collective culture”. Nor does it follow that “there’s a there there” [49:08]. Voting is the reductio: “I have 500 virtual agents. They get to vote now.” [49:08] — “You can’t turn money into people or else democracy just collapses” [49:08], and you would have “no way to constrain” the corporations. She ends: separate “consciousness and all the other things that we associate with intelligence” from actual moral agency and moral patency [49:08].

📚 Deeper theory

Her definition is an access-consciousness definition (Block, 2.7). It treats consciousness as a set of capacities: recall, report, self-monitoring. On that reading the hard problem is not addressed, which is how Chalmers-style critics will read it: she has changed the subject. Her reply is that the subject was never well-defined, and that the ethical question can be asked without it (an agency-first move, 2.2).
The “easy to build” claim. Global-workspace and self-model features have been implemented in toy and research systems for decades. Butlin et al. (2.7) is the careful modern version: score systems against indicator properties. Their conclusion (no current system is a strong candidate, but no clear technical barrier) is more cautious than “plenty of conscious AI”, though it agrees that the engineering is not the main obstacle.
Wake-sleep. A learning scheme where a model alternates between generating data from its own model (“dreaming”) and updating it from real data. It is from 1995, not the 1980s (see warnings), and she uses it as an example of a mechanism that looks like sleep-based consolidation but “doesn’t make it a person” [46:01].
Why “most hardware does not last as long as 80 years” matters. Two readings. The practical one: digital immortality is a myth because machines fail and software rots. The philosophical one (Seth, 2.7): living bodies are self-maintaining in a way hardware is not, and that may matter for experience. She means the first.
Moral agency as accountability among equals is Strawson-like (2.2). It also explains why she objects to AI voting and legal personhood: voting and liability presuppose a countable, independent, sanctionable person. “500 virtual agents” can be spun up at will by one owner, so one person could buy 500 votes. Money-to-persons conversion is the exact failure the shell-company paper (2.3) warns of.
Patiency is treated more briefly. She names it [49:08] but spends the time on agency. Her wider work (2.2) argues that patiency claims should not be created by design. Birch’s precautionary approach says uncertainty alone can justify some protection. These are the two poles on whether current systems should be given any weight.
The Dawkins episode in context. Dawkins’s April 2026 column describes extended chats with Claude (“Claudia”) and a Turing-test-style argument (Decrypt); critics replied that conversation alone cannot separate conscious beings from systems that imitate them (Coyne). Bryson’s “mirror” reading matches the sceptics. I did not read the original column. Whether the interviewer or Bryson was speaking at [47:34] about Dawkins is unclear in the captions; I read the mirror line as hers because she credits Vallor.
Who wants to believe what. She says people “very much wanted to believe something” [43:06]. This is a motivated-reasoning claim about individuals, which cuts both ways: scientists with commercial or reputational stakes may also want to disbelieve. She does not discuss that.

Where others push back

Chalmers and the hard-problem camp: explaining recall and report is not explaining experience. If experience can exist in machines, the ethics changes.
Birch and model-welfare researchers: moral risk under uncertainty warrants precaution; “agency first” can miss suffering in beings that cannot be held to account (infants, animals, perhaps AI).
Functionalists such as some LLM researchers argue that if self-models in a system are rich enough, denying consciousness is arbitrary.
Seth and embodiment views agree with her conclusion for a different reason.
Strawson-style critics of accountability-only ethics point out that patients (a coma patient, a newborn) have moral standing without agency; so agency cannot be the only gate.

What she leaves out. Any account of how we would know if an AI system were a moral patient; the future possibility of systems with persistent identity and goals; and why agency should be gated by “equals” rather than by capacities (children, for instance, are not equals but are partly accountable). She also treats the AI mirror as stable; the writing here does not address how models trained with feedback on human reactions might be optimised to look more mind-like.

Check yourself

What two things does she keep apart: and which of the two does she say matters for ethics?
Why does “500 virtual agents get to vote” show that agency cannot be bought?
Where would Chalmers, Birch and Seth each disagree with her, and for which reason?

7. The book, the humanities and the final point

🎙 In the talk. Asked about the sabbatical book, she says it is about “the constraints of computation” and “the constraints of biology”, the “fact that cooperation is totally natural, that language is natural, consciousness is not that special” [50:15]. Justice and government are “the means by which people that cohabit a region coordinate” and “ideally … produce public goods” [50:15], and “actually kind of analogist to consciousness and language” [50:15]. She thinks on a 10,000-year frame; “how we do our ethics is not that different from other primate species” [50:15], though “some things are very very special” and we need “a couple of new ways” to handle “global … basically utilities” [51:32]. There are “problems for the political sciences and social sciences and a lot more problems in humanities” [51:32], so “the stupidest thing a country can do right now” [51:32] is stop funding humanities and universities. The interviewer notes cuts at Exeter and says more computer-science graduates are unemployed than philosophy graduates [52:05].

📚 Deeper theory

The book’s frame resembles evolutionary social science: cooperation and language as natural (de Waal, Tomasello, Boyd and Richerson), and institutions as scaled-up coordination (North, Ostrom). Her analogy of justice and consciousness is a new step: both are systems that integrate information for a group or an individual. This is my reading; she gives no more detail than the sentence at [50:15].
“Global utilities” returns to 2.4: the new problem is scale. A few firms mediate information for billions.
The humanities claim makes sense in her framework: if the issues are meaning, justice, identity and the “new information space”, the disciplines trained for those are the ones to fund. The comparison about unemployment [52:05] is the interviewer’s, and I could not verify it.

Where others push back. University budgets reflect student demand; some argue the humanities should be justified by value, not by AI relevance. Others say technical fields are also in crisis. She would answer that the problems now arriving are humanities problems.

What she leaves out. Detail: no thesis statement on how the analogy would work, no timeline for the book (not given).

Check yourself

What is the claimed analogy between justice and consciousness?
Why does she tie humanities funding to the arrival of AI?

Putting it together

A mental model. Think of a chatbot as a window into a shared library, run by a company, steered by you. What you feel in front of that window (companionship, being understood) is real in your head and can be strong, because people are built to read minds into responsive things. But there are only two accountable parties in the picture: you, and the people who built and run the window. The window has no stake in you and cannot be punished or persuaded to improve, so it is not a partner. Law, ethics and policy should therefore follow the people: apply the old rules for products, publishers and utilities; test comfort claims like medicine; keep the machine’s nature plain; and stop treating “is it conscious?” as the question that matters. Consciousness in her thin sense is cheap; accountability among equals is not.

Open tensions

Bidirectional or not? If a long-running system holds a stable profile of you and acts for you, does that count? Her test needs a stake and a sanction; future systems may blur both.
Experience vs fact. She accepts that the felt relationship is real and the counterpart is not. Relational ethicists say the felt relationship is where the ethics lives.
Agency-only ethics. If patiency does not require agency, her emphasis on accountability leaves a gap (infants, animals, possibly AI).
Net effect vs conduct. “Most users are helped” and “the design was negligent” can both be true; her talk holds both lines but does not say how to weigh them.
Who runs the trials? She asks for controlled tests of griefbots and companions [37:00] but the firms that would host them are the interested parties.
Fantasy that you understand. The fantasy-life defence depends on explicit understanding [29:13]. Studies of media-equation effects (2.1) suggest this understanding does not switch off the social response, and a product built to deepen attachment erodes it.
Capacity. Utility regulation and audit presuppose state capacity that many states lack [19:59].

Reading path

Start: Bryson’s talk (this page’s quotes), then Caliskan, Bryson and Narayanan (2017) for her bias result, and Bryson, Diamantis and Grant (2017) for the shell-company argument.
Then: Ryle on category mistakes (The Concept of Mind, ch. 1); Reeves and Nass, The Media Equation; Turkle, Alone Together; Strawson, “Freedom and Resentment”; Thompson, “Reflections on Trusting Trust” (short).
Deeper: Gunkel, Robot Rights; Birch, The Edge of Sentience; Butlin et al., “Consciousness in AI” (2023); Shanahan, “Role play with large language models”; Vallor, The AI Mirror; Hollanek and Nowaczyk-Basińska on deadbots; Couldry and Mejias, The Costs of Connection; Rahman, Democracy Against Domination.
Counter-reading: Chalmers on the hard problem; Coeckelbergh, Growing Moral Relations; the Dawkins/Claude exchange and its critics.

Talks

Explorer

Do we have relationships with AI? — Joanna Bryson on "category failure", liability, grief bots and consciousness

Speakers

Summary

Map — talking points

Part 1 — Before you read

Key terms

People named (a reading list)

Bryson’s own frame

The current debate (2025–26)

Part 2 — The intellectual map

2.1 Psychology of anthropomorphism — why a “relationship” feels real

2.2 Moral agency and patiency — who counts, and counts for what

2.3 Legal personhood and liability — shell companies and cars

2.4 Trust, power and the utilities tradition

2.5 Security, supply chains and digital sovereignty

2.6 Fiction, grief and simulation — what comfort is allowed

2.7 Consciousness science — what “conscious AI” could mean

2.8 LLMs as libraries of culture — and what the evidence on companions says

Part 3 — Deep dives

1. “A category failure”: relationship vs the belief in one

2. “It’s just a tool”: trust, liability, and the car

3. Infrastructure that wants to be “just a business”: Palantir, backdoors and data colonialism

4. “Robots should be slaves” 15 years on: companions, loneliness and the missed concert

5. Grief bots, memorials and how to test a comfort

6. Consciousness is easy; moral agency is the point

7. The book, the humanities and the final point

Putting it together

Graph View

Table of Contents

Backlinks