How AI is changing medical practice — Frank Pasquale, Justin Skowno, Kimberlee Weatherall

About this talk

Speaker: Frank Pasquale (with Justin Skowno; moderator Kimberlee Weatherall)
Event: Sydney Ideas, University of Sydney (online, with the ARC Centre of Excellence for Automated Decision-Making and Society)
Date: 2022-07-25 (event date per the description; the video was uploaded 2022-07-26)
Length: 58:32

How to read this page. It has two kinds of content, always marked:

• 🎙 In the talk. What the speakers say, with [MM:SS] timestamps. Only these parts are their views.
• 📚 Deeper theory. Background from the traditions behind the talk. It explains why they say what they say and who disagrees. It is not their claim unless a timestamp says so.

Suggested route. Summary → Map → Key terms → Part 2: The intellectual map (read once, slowly) → Deep dives in order → “Putting it together”.

Sibling pages (same author, other talks): pasquale-new-laws-of-robotics (the book this talk draws on), pasquale-machines-judging-humans, pasquale-future-of-ai-in-law.

Format note. Pasquale talks for about 22 minutes [04:40]–[26:25]. The rest is a three-way conversation, so the deep dives mix his claims with Skowno’s and Weatherall’s. The page marks who says what.

---

Speakers

Label	Who	Role	Confidence
Host	”vanella curabo” in the captions (probably a different spelling)	Head of programming, Sydney Ideas. Opens and closes	Low on the name; role is stated in the talk
Weatherall	Kimberlee Weatherall (captions: “kimberly wetherol”)	Professor of law, University of Sydney; chief investigator at the ARC Centre of Excellence for Automated Decision-Making and Society (ADM+S). Moderator	High (video details name her)
Pasquale	Frank Pasquale	Professor, Brooklyn Law School. Author of The Black Box Society (2015; the host says 2016) and New Laws of Robotics (HUP, 2020)	High
Skowno	Justin Skowno (captions: “scono”)	Associate professor and paediatric anaesthetist; director of research in paediatric anaesthesia at the Children’s Hospital at Westmead; lead on “Big Data for Small People”, a critical-care informatics project	High (video details name him)

Caption warnings. Auto-captions, no punctuation. Corrections used on this page:

• “dr eric copel” = Eric Topol (cardiologist, author of Deep Medicine).
• “carolyn creato perez” = Caroline Criado Perez, Invisible Women.
• “hal wang” = Hao Wang, “Transparency as Manipulation?” (see Part 1).
• “nicholson price sarah gurkey and glenn cohen” = W. Nicholson Price, Sara Gerke, I. Glenn Cohen.
• “ziad obermeyer” is right. “ben schneiderman” = Ben Shneiderman. “chris slavogan” = Christopher Slobogin, Just Algorithms. “d medicine” at [24:51] is unclear; Pasquale’s list there is of books and papers on unrepresentative databases, and I could not tell which one this is.
• “radical people of the urination” = Gadigal people of the Eora Nation. “calm practice”, “pax love it” = Paxlovid, “hits” = probably “its” (drug interactions). “cell phone” at [09:20] is a mobile-phone search card.
• “prepare robot” at [18:42] = Paro robot. “covert” = Covid. “adm” = ADM+S.
• Both “fhir” and “hl7” are right ([37:20]).

Factual slips in the talk (checked by search or from memory).

• [35:46] Skowno: Obama “got it going in 2008” for electronic health records. The US HITECH Act, which funded EHR adoption, was part of the 2009 stimulus. (From memory; not checked.) The 2008 financial crisis context is right.
• [37:20] Skowno calls HL7 “a specific code” and FHIR “a background governance architecture”. HL7 is a standards body and family of standards, and FHIR is a standard it publishes for exchanging records. (From memory; not checked.)
• [54:30] Skowno calls the swamping of small groups by large ones “regression to the mean”. That term means extreme values drift toward the average on re-measurement. What he describes is closer to under-representation or majority-dominated aggregation. His point stands under the right name.
• [40:28] Pasquale says the Obermeyer study used “insurance records” and that minorities “were not getting the care they needed”. The study’s algorithm used health-care costs as a proxy for need; less money was spent on equally sick Black patients, so the model read them as healthier (Berkeley CHMI summary). His version is a fair short form; the proxy mechanism is the part to remember.
• [03:06] The host dates The Black Box Society to 2016. I recall a 2015 first edition (Harvard UP). From memory; not checked.
• [56:02] Pasquale flags his own NHS claim as unverified (“i’ve not verified it”). I did not check it.

---

Summary

Central question. What should law and policy do about AI in medicine, if the realistic goal is help for clinicians and not replacement of them?

Main thesis (Pasquale). The old dream of AI medicine was full automation, the Star Trek tricorder. That dream is “delayed and is perhaps indefinitely delayed” [06:15]. The new goal is “decision support advice error reduction quality assurance” [09:20]: human-computer cooperation. Even tasks that look simple (post-surgery coaching, chatbot therapy) raise hard legal questions: licensure, institutional liability, malpractice, informed consent, privacy [10:54]–[14:01].

He then walks through five cases where AI changes a clinician’s job:

1. Robotic and avatar caregivers (the Paro seal, Care Coach) [17:09].
2. “Death algorithms” that predict recovery in intensive care [14:01]–[21:47].
3. Dermatology image tools, which also create direct patient demand [15:35].
4. Drug-interaction decision support, already old [15:35].
5. Triage, where patients may game the system [21:47].

Two framing tools run through the talk:

• the iron triangle (cost, quality, access; you cannot improve all three) and Nicholas Terry’s trilemma (automation, value, empathy) [20:15];
• the 8-way liability grid of Price, Gerke and Cohen (standard or non-standard care × AI correct or not × doctor follows or not) [23:19].

He closes with three metaphors for AI and the professions: meta-expert (AI judges who is an expert), melting pot (the two merge), peaceable kingdom (each keeps its identity and learns from the other). He chooses the third [26:25].

The conversation adds (mostly Skowno and Weatherall):

• Quantifying “bedside” care [27:59].
• Automation versus augmentation, and the handoff problem (a self-driving car hands the crash to the driver) [31:05].
• The data is uneven: imaging is standardised, a GP’s clinical notes are not [33:46].
• Regulators lack capacity [38:55].
• Amazon’s purchase of One Medical, and de-skilling of doctors [43:35].
• Gut feeling versus protocol [49:50].
• Exclusion, especially Indigenous data and a “two-tier” system where AI screens first [53:00]–[56:02].

Stance. Pasquale: sceptical of automation, constructive on decision support, and focused on institutions, labour and political economy more than on model accuracy. Skowno: pro-protocol, pro-data, openly says reliance on gut feeling “probably causes more harm than good on average” [49:50]. Weatherall: moderates, and raises the sharpest institutional questions (who sets standards, what Amazon wants from doctors’ data).

---

Map — talking points

#	Point	Time
1	Welcome, access, Slido questions. Weatherall introduces Pasquale and Skowno	[00:01]–[04:40]
2	The tricorder: the dream of full automation. Delayed, perhaps indefinitely	[06:15]
3	Dr Google: search results for colloidal silver; Google brought in medical experts for its health “cards”	[07:48]–[09:20]
4	New dream = decision support. Topol’s “AI for the simple things” (Alexa, drug delivery, post-op care)	[09:20]–[10:54]
5	Are the simple things simple? Five legal issues: licensure, institutional liability, malpractice, informed consent, privacy	[10:54]–[14:01]
6	Four concrete cases: caregivers, ICU “death algorithm”, dermatology, drug-interaction software	[14:01]–[17:09]
7	Paro seal and Care Coach avatar: recording, labour in the Philippines, cortisol as a measure of “it works”	[17:09]–[20:15]
8	Labour shortage may be a low-wage problem. Iron triangle. Terry’s trilemma	[20:15]
9	Death algorithm: comfort and objectivity, but also rigid reimbursement and rationing	[21:47]
10	Triage: gaming (“say an eight or nine”); Wang on transparency as manipulation	[21:47]–[23:19]
11	Price/Gerke/Cohen 8-way liability grid; two red boxes	[23:19]–[24:51]
12	Unrepresentative databases (skin types, women); liability for failing to fix them	[24:51]
13	Three metaphors: meta-expert, melting pot, peaceable kingdom	[24:51]–[26:25]
14	Skowno: how can law quantify bedside care? Pasquale: risk, personalisation, evidence-based medicine, Dartmouth Atlas	[26:25]–[29:32]
15	Automation vs augmentation; the handoff problem; Shneiderman and human-computer interaction	[29:32]–[32:39]
16	Data: imaging is standardised (DICOM), notes are messy; EHRs; ICD-10, HL7, FHIR; who sets standards	[32:39]–[37:20]
17	Self-referential data loops; regulator capacity and pay gaps; Obermeyer bias study	[37:20]–[40:28]
18	Political economy: profit-led vs socialised systems	[41:42]–[43:35]
19	Amazon and One Medical; EHR as unpaid labour; Carr’s Glass Cage and de-skilling; competition law	[43:35]–[48:16]
20	Skowno: big tech skills are valuable; policy settings decide the outcome	[48:16]–[49:50]
21	Gut feeling vs protocol; expert systems of the 1980s; Slobogin’s “just algorithms”; documenting deviations	[49:50]–[52:57]
22	Exclusion: Indigenous data, regression to the mean (sic), two-tier screening, NHS radiology example, AI to surface existing bias	[52:57]–[57:36]
23	Close	[57:36]

---

Part 1 — Before you read

Key terms

• Decision support (clinical decision support, CDS). Software that advises a clinician and leaves the decision with them: drug-interaction alerts, risk scores, image flags. Pasquale’s “new dream” [09:20]. Compare with automation, where the software acts alone.
• Licensure. In many health systems, only licensed people may do some tasks. A licence implies training, an exam, ethics duties and a body that can strike you off. Whether a chatbot or robot needs an equivalent is Pasquale’s first legal question [10:54].
• Institutional liability vs malpractice. Malpractice (a type of negligence) is a professional’s failure to meet the standard of care. Institutional liability is the hospital’s or company’s responsibility. The two can split: a hospital may carry blame for a vendor’s error because the vendor’s contract disclaims it [12:28].
• Standard of care. The level of skill a reasonable practitioner would show. It decides negligence. If AI becomes common, it may start to define “reasonable”.
• Informed consent. A patient must understand the risks and alternatives. With AI, “understand” is harder.
• “Death algorithm.” Pasquale’s phrase for a model that predicts chance of recovery from monitoring data, used to help decide whether to continue life support [14:01]–[15:35].
• Iron triangle. US health-policy idea: cost, quality and access trade off [20:15].
• Trilemma (Terry). Automation, value, empathy: you can push one or two, rarely all three [20:15].
• Automation vs augmentation. Replace the human task or enhance the human who does it. Weatherall’s framing [29:32]. Watch for a trap: augmentation can still shift risk onto the human (see “moral crumple zone” in Part 2).
• Interoperability standards. DICOM (imaging), ICD-10 (disease coding), HL7 and FHIR (record exchange) [33:46]–[37:20]. These are the plumbing that decides what data AI can use.
• Protocolisation. Turning a clinical practice into a fixed checklist. Skowno’s example is antibiotics before surgery [51:23].
• Regression to the mean. Skowno uses it loosely [54:30]; the statistical term means something different (see slip above).
• Word that means different things by tradition: “bias”. In statistics, bias is systematic error of an estimator. In law and ethics, it is unfair treatment of a group. In medicine, it also means cognitive bias in clinicians. The talk uses all three, often within one answer. Check which one each speaker means.
• Word that means different things by tradition: “transparency”. For engineers it is interpretability of a model. For lawyers it is disclosure of rules or reasons. For Wang (cited at [23:19]) it can itself be a tool of discipline. Pasquale holds that secrecy is a problem and that full disclosure invites gaming.

People and works named (a reading list)

Person	Why they come up	Start with
Eric Topol	”AI for the simple things”: Alexa, drug delivery, post-op coaching	Deep Medicine (2019)
Ryan Abbott	Treat equivalent AI and human acts in parallel	The Reasonable Robot (2020)
Nicholas Terry	Trilemma of automation, value, empathy	his health-law and AI papers (no single title checked)
Hao Wang	Transparency can normalise behaviour	”Transparency as Manipulation? Uncovering the Disciplinary Power of Algorithmic Transparency”, Philosophy & Technology 2022 (UvA record)
W. Nicholson Price, Sara Gerke, I. Glenn Cohen	8-way liability grid	their work on liability for medical AI (title as said in the talk: “Liability for Use of AI in Medicine”)
Adamson & Smith	Dermatology AI and skin-type disparities	”Machine Learning and Health Care Disparities in Dermatology”, JAMA Dermatology 2018 (from memory; not checked)
Caroline Criado Perez	Women missing from medical data	Invisible Women (2019)
Ziad Obermeyer et al.	Cost-as-proxy bias	”Dissecting racial bias in an algorithm used to manage the health of populations”, Science 2019
Ben Shneiderman	Human-computer interaction for human-centred AI	Human-Centered AI (2022)
Nicholas Carr	De-skilling through automation	The Glass Cage (2014)
Christopher Slobogin	”Just algorithms” in sentencing	Just Algorithms (2021)
Carlos Bustamante	Diversity in DNA databases	his population-genomics papers
Pasquale’s own earlier work	Liability for unrepresentative data	a Columbia Law Review article he cites at [24:51] (probably “Data-Informed Duties in AI Development”, 2019; from memory; not checked)

The speakers’ own frame

Pasquale’s frame is political economy plus professional ethics. The Black Box Society (2015) argued that finance, search and reputation systems run on secret rules. New Laws of Robotics (2020) argues that AI should complement professionals, not replace them, and that law should protect professional judgement where stakes are high (the book’s four laws, as Weatherall previews at [03:06]: robots should not counterfeit humanity; and see pasquale-new-laws-of-robotics). This talk is that book applied to medicine.

Skowno’s frame is clinical informatics: a practising anaesthetist who builds data projects, favours protocols, and cares more about data quality than about philosophy.

Weatherall’s frame is ADM+S: automated decision-making in society, with a regulator’s eye on standards and capacity.

The current debate (checked by live search, 30 Oct 2026)

• AI devices are mostly radiology. Counts differ by source, and the FDA says its list is not comprehensive. One tracker reports 1,524 FDA AI-enabled device authorisations through 30 Mar 2026, about 76% radiology (ai2.work); another secondary source claims 1,614 by June 2026, which I could not confirm (Complizen). This supports Skowno’s point that imaging is where data is standard [33:46] and Weatherall’s guess that FDA approvals cluster there [34:12]. Both are secondary sources, so treat the exact numbers as approximate.
• Ambient AI scribes are the real mass deployment, and they address Pasquale’s “unpaid labour” point. A 2026 multi-site study reported about 13 fewer minutes of EHR time per encounter (summary at 2 Minute Medicine); an emergency-department comparison found human scribes saved more time than AI scribes and neither raised throughput (Mass General Brigham). Evidence is mostly single-site and self-reported. This was not in the 2022 talk.
• Liability is still unsettled. Commentary in 2025–26 says courts have not settled whether following or rejecting AI advice breaches the standard of care; AI malpractice cases are “in their infancy”; one study cited suggests jurors may see following AI as reasonable (Medical Economics, Oct 2025, Physicians Practice, May 2026). That is trade press and law-firm commentary, not case law. It is the open version of the 8-way grid at [23:19].
• Wang’s argument has grown. Wang followed up in 2023 and others (Franke, Klenk) have replied, so “transparency as manipulation” is now a small literature (Springer lists the related papers).
• The Obermeyer finding is still the standard example of proxy bias: at the same risk score, Black patients were sicker, and fixing the proxy would have raised the share of Black patients flagged for extra help from 17.7% to 46.5% (Berkeley CHMI).
• Amazon and One Medical. The deal closed in February 2023 after the FTC chose not to sue before closing; the FTC said its investigation continued (Gulf News report). One Medical later added a generative-AI assistant per Wikipedia (One Medical); the date was not confirmed.
• Not searched; from memory; not checked: large language models changed the picture after this talk (late 2022). Chatbot “therapists” and symptom checkers, which Pasquale raises at [10:54], now run on general LLMs. Studies of LLMs answering clinical questions, and the EU AI Act’s treatment of medical devices as high risk, are live. Verify before quoting.

---

Part 2 — The intellectual map

Pasquale spoke for 22 minutes to a mixed legal, medical and policy audience, so he names a handful of thinkers and leaves the traditions behind them unstated. Here they are laid out. Seven families of ideas meet in this talk:

 PROFESSIONS & EXPERTISE ───┐                 ┌─── LIABILITY & REGULATION
 Abbott(Andrew) · Larson    │                 │    Calabresi · Price/Gerke/Cohen
 Freidson · Illich          ▼                 ▼    FDA devices · licensure
                     ┌───────────────────────────────┐
 AUTOMATION & HUMAN  │  PASQUALE: AI should support  │  HEALTH ECONOMICS
 FACTORS ──────────► │  clinicians, under law that   │ ◄── Iron triangle · Baumol
 Bainbridge · Carr   │  protects judgement, labour   │     Dartmouth Atlas
 Elish · Shneiderman │  and fair access              │
                     └───────────────────────────────┘
        ▲                      ▲                     ▲
 EVIDENCE & PROTOCOL     DATA, STANDARDS,       BIAS, POWER, TRANSPARENCY
 Cochrane · Meehl ·      INTEROPERABILITY       Foucault · Wang · Obermeyer ·
 Gawande · Tetlock       DICOM · ICD · FHIR     Criado Perez · Benjamin

2.1 Professions and expertise — why law protects judgement

The problem it addresses. Why do some jobs (doctor, lawyer) get licences, self-regulation and legal duties, and what happens when a machine can do part of them? Pasquale’s whole stance, “AI complements, it does not replace”, comes from this tradition.

The chain of thinkers

1. Max Weber and Émile Durkheim (c. 1900–1920). Expertise as a basis of authority; professional groups as moral communities with duties. Durkheim saw occupational groups as a source of ethics in modern society.
2. Talcott Parsons, “The Social Structure of Medicine” (1951). The sick role and the doctor’s role: trusted, disinterested, bound by a code. Medicine as a profession built on trust.
3. Eliot Freidson, Profession of Medicine (1970). Medicine’s power is autonomy granted by the state: control over its own work and standards. The bargain is trust in return for self-policing.
4. Magali Sarfatti Larson, The Rise of Professionalism (1977). Professions are “projects” to win a market monopoly through credentials. This is the sceptic’s view: licensure also protects incomes.
5. Ivan Illich, Medical Nemesis (1975). The harshest critique: medicine creates dependency and harm. Useful to remember when Pasquale defends doctors.
6. Andrew Abbott, The System of Professions (1988). Professions fight over jurisdiction: who owns which tasks. New technology reopens the boundary. Not to be confused with Ryan Abbott (the lawyer Pasquale cites at [14:01]).
7. Richard and Daniel Susskind, The Future of the Professions (2015). The opposite view: professions are a 19th-century solution to scarcity of knowledge, and technology will dissolve them.
8. Pasquale, New Laws of Robotics (2020). Takes the Freidson side: keep professional judgement, and use law to channel AI toward complementing it (see pasquale-new-laws-of-robotics).

How it feeds the talk. The three metaphors at [24:51]–[26:25] are three answers to a jurisdiction fight. Meta-expert = AI wins the whole territory (Susskind). Melting pot = the border dissolves (radiology and pathology). Peaceable kingdom = the border holds and both sides learn (Pasquale, Freidson). Licensure for chatbots [10:54] is the same fight in legal form: does a bot enter the licensed jurisdiction, or stay outside it?

2.2 Automation and human factors — the handoff problem

The problem it addresses. When a machine does most of a job, what happens to the human who is left, and who is blamed when it fails? Pasquale’s self-driving example [31:05] and Carr’s pilots [46:42] come from here.

1. Lisanne Bainbridge, “Ironies of Automation” (1983). The classic paper: automate the easy parts and you leave the human the hardest parts, with less practice and less awareness. The human must monitor a system they are bad at monitoring, and take over in a crisis, when they are least ready.
2. Parasuraman and Riley, “Humans and Automation: Use, Misuse, Disuse, Abuse” (1997). Automation bias (over-trust) and disuse (under-trust). Both occur with clinical alerts: alert fatigue makes clinicians ignore real warnings.
3. Nicholas Carr, The Glass Cage (2014). The talk names this at [46:42]. Autopilot erodes pilots’ skills (“de-skilling”). Carr’s contrast: automation that keeps the human skilled and in control vs automation that does not.
4. Madeleine Clare Elish, “Moral Crumple Zones” (2019). Named after the part of a car that absorbs impact. In complex systems, the nearest human absorbs blame for a system failure. This is exactly Pasquale’s worry that a vehicle “turns over responsibility to the driver” just before a crash [31:05], and that doctors may be left “with a great deal of responsibility but then without a lot of support”.
5. Ben Shneiderman, Human-Centered AI (2022). Named at [32:39]. Argues against the one-dimensional “more automation = better” scale: you can have high automation and high human control. His proposed design goal is “reliable, safe and trustworthy” systems with strong human oversight.
6. Levels of automation (SAE J3016 for cars; Sheridan–Verplank for general tasks). A ladder from full manual to full auto. Medical devices are starting to use similar ladders (from memory; not checked).

How it feeds the talk. Weatherall asks automation nor just augmentation [29:32]. The answer from this tradition is that augmentation is not automatically safe. It only is if the handoffs, the interface and the institution let the human override. Pasquale’s HCI pointer [32:39] is this line.

2.3 Liability and regulation — who pays when software errs

The problem it addresses. Tort law assigns losses. Medical AI breaks the usual fit: is the software a product (strict liability), a service, a reference book, or a colleague?

1. Tort basics. Negligence: breach of the standard of care. Product liability: a defective product harms someone, and the manufacturer pays whether or not it was careful. Software has been hard to treat as a “product” (from memory; not checked, and it varies by jurisdiction).
2. The “learned intermediary” idea (US). Drug makers warn the doctor, not the patient. The doctor is the filter. If AI advice reaches the doctor, the vendor may argue the doctor is the learned intermediary who takes responsibility. This is the tension in the hospital signs a disclaimer point at [12:28]. (From memory; not checked as applied to AI.)
3. Guido Calabresi, The Costs of Accidents (1970). Put liability on the “cheapest cost avoider”: the party best placed to prevent the harm cheaply. The question for AI errors is who that is: the vendor, the hospital or the clinician.
4. Price, Gerke and Cohen. Their 8-way grid [23:19] applies the question to cases. The two red boxes are (a) the doctor rejects correct, standard AI advice and the patient is hurt, and (b) the doctor follows incorrect, non-standard AI advice and the patient is hurt. In both, the doctor departs from, or follows, the machine against the norm. The grid shows liability pressure pushes toward “do what is standard” (defensive medicine).
5. Regulation of medical devices. The FDA (US) and TGA (Australia) classify software as a device by risk. Weatherall notes that FDA AI approvals so far are largely imaging [34:12]. A hard problem is adaptive systems that change after approval [38:55]; the FDA has proposed “predetermined change control plans” for them (from memory; not checked).
6. Ryan Abbott, The Reasonable Robot (2020). Argues for neutrality: if AI does the same task better, the law should treat it like the human, including holding it to the same standard. Pasquale notes the objection that responsibility is institutional [14:01].
7. Pasquale’s own addition: liability for data. His argument [24:51] is that developers who ignore unrepresentative data should be liable, even when the algorithm itself is not “wrong”. This ties tort law to the bias literature (2.6).

How it feeds the talk. The five issues at [10:54]–[14:01] are an inventory of this tradition’s open questions, applied to AI.

2.4 Health economics — cost, access and the low-wage care sector

The problem it addresses. Healthcare cannot be understood as engineering only; money and labour shape what gets built.

1. The iron triangle (Kissick, Medicine’s Dilemmas, 1994). Cost, quality, access. A change in one pushes the others. Pasquale’s rule of thumb [20:15]: when AI is sold as a cost cut, ask what is traded away.
2. William Baumol, “cost disease” (1965, with Bowen). Services that need human time (care, teaching) get dearer relative to goods because they cannot be made more productive per hour. This is the deep motive for automating care: a way to beat the cost disease. Pasquale’s counter-point is that the “labour shortage” may be a wage problem [20:15].
3. Dartmouth Atlas of Health Care (Wennberg, Fisher, from the 1990s). Large, unexplained regional differences in treatment rates. Named at [29:32]. Used to argue for evidence-based standardisation and also to question whether more care means better care.
4. Kenneth Arrow, “Uncertainty and the Welfare Economics of Medical Care” (1963). Medicine is unlike ordinary markets: the patient cannot judge quality, so trust and professional norms substitute. If AI weakens the professional’s role, the basis of trust changes.
5. Care work and the “global care chain” (Hochschild). Care labour moves from richer to poorer places, as in Care Coach’s avatar voiced from the Philippines [17:09]. (Hochschild’s term from memory; not checked.)
6. Electronic health records as a cost shift. The 2009 US stimulus funded EHRs; clinicians experienced it as a rise in unpaid data entry [43:35]–[45:09]. This is the labour side of Pasquale’s political economy.

How it feeds the talk. The caregiver avatar, the death algorithm used for rationing [21:47] and the Amazon discussion are all cases where a cost story hides a distribution story.

2.5 Evidence, protocol and the limits of judgement

The problem it addresses. Skowno’s gut-feeling answer [49:50] touches a long debate: how much to trust expert intuition versus rules.

1. Paul Meehl, Clinical vs Statistical Prediction (1954). In dozens of studies, simple statistical rules matched or beat clinicians’ judgement. A foundation of the “use the algorithm” camp.
2. Kahneman and Klein, “Conditions for Intuitive Expertise: A Failure to Disagree” (2009). Intuition can be trusted when the environment is regular and the expert had fast, clear feedback (a chess master, a firefighter), not when the environment is noisy. This answers Skowno’s puzzle: gut feeling may be reliable in some medical settings and not in others.
3. Evidence-based medicine (Sackett, Guyatt, 1990s; Cochrane). Medical practice should rest on systematic evidence, not authority. Pasquale’s “evidence-based medicine and the dartmouth atlas” [29:32] is this line.
4. Atul Gawande, The Checklist Manifesto (2009). Checklists reduce errors in surgery and ICU lines. This is Skowno’s protocolisation, with antibiotics before surgery and line cleaning [51:23]–[52:57]. Best evidence: the WHO surgical checklist and the Keystone ICU line-infection programme (from memory; not checked).
5. Expert systems and their failure. MYCIN (1970s) and the 1980s systems Skowno mentions [49:50] failed partly because experts could not state their rules. Michael Polanyi: “we can know more than we can tell”; Hubert Dreyfus argued from this that expertise is not rule-following. Skowno’s “very difficult to codify why the senior clinicians ended up being correct” is Polanyi’s paradox in a ward.
6. Tetlock, Expert Political Judgment (2005). Experts are often poorly calibrated; track records matter more than status.
7. Slobogin’s Just Algorithms (2021). Weatherall cites it [51:23]: use algorithms for sentencing (and “just” in both senses). Her suggested middle path is documentation duty: if you go against the standard or the AI, write down why.

How it feeds the talk. The disagreement between Skowno (“more protocol”) and Pasquale (“protect judgement”) is Meehl against Freidson, with Kahneman and Klein as the referee.

2.6 Data, standards and bias

The problem it addresses. AI is only as good as the data and the labels. Medical data is uneven in format, coverage and meaning.

1. Standards stack. DICOM (imaging, 1980s onward), ICD (disease codes, WHO), HL7 v2/v3 and FHIR (exchange of records). Skowno [33:46]–[37:20]. Standards are political: whoever sets a code decides what can be counted. (From memory; not checked beyond the talk.)
2. Geoffrey Bowker and Susan Leigh Star, Sorting Things Out (1999). Classifications look neutral but embed values and hide the residual. ICD is one of their cases. Weatherall’s “who develops those standards?” [35:46] is their question.
3. Representativeness. Adamson & Smith on skin types; Criado Perez on women; Bustamante on genomic databases; the Indigenous data question at [53:00]. For Indigenous data, see CARE principles and Indigenous data sovereignty (Te Mana Raraunga in Aotearoa; Maiam nayri Wingara in Australia) (from memory; not checked). They stress collective control, not only individual consent.
4. Obermeyer et al. (2019). Proxy bias: the algorithm predicted cost, not need. The lesson Pasquale takes is that bias can enter without any racial variable.
5. Feedback loops. Weatherall’s “self-referential cycle” [37:20]: the model changes behaviour, which changes data, which retrains the model. The ML literature calls this performative prediction or dataset shift (from memory; not checked). Goodhart’s law is the folk version: a measure used as a target stops being a good measure. Pain scores [21:47] are the clinical example.
6. Ruha Benjamin, Race After Technology (2019). The “New Jim Code”: neutral-looking systems reproduce hierarchy. Close to Skowno’s last line that AI might help surface bias [56:02].

2.7 Transparency, discipline and gaming

The problem it addresses. Should a triage rule be public? If secret, it is unaccountable; if public, people game it.

1. Foucault, Discipline and Punish (1975). Power works by making people visible and normal. Wang uses this: disclosing an algorithm can normalise behaviour, giving patients a script to perform for the machine (Wang 2022 record; abstract only checked).
2. Goodhart (1975) and Campbell (1976). Any indicator used for decisions is corrupted by pressure. Pasquale’s “say an eight or nine” [21:47] is this law with an anecdote.
3. Pasquale, The Black Box Society (2015). The case against secret systems. The talk’s twist is that he does not conclude “publish everything”. He holds both horns: opacity is unaccountable, transparency is gameable.
4. Possible exits, which the talk does not name (from memory; not checked): audit by trusted third parties instead of public release; disclose reasons and rights, not parameters; random audits to deter gaming; and a right to human review.

How it fits. This tradition explains why Pasquale ends the triage point on a dilemma and not a rule [23:19].

---

Part 3 — Deep dives

1. The vanished tricorder — from full automation to decision support

🎙 In the talk. Pasquale shows the Star Trek tricorder as the dream of “full automation” a decade ago [06:15]. The dream was noble because many people “are not able to access the medical system”, yet it “has been delayed and is perhaps indefinitely delayed” because of “the ongoing need for intervention and advice and mediation” [06:15]. Search is his test case. Searching “colloidal silver” shows a messy result list; Google then “brought in medical experts” to build the health card, so these cards are “human computer interaction and cooperation” [09:20]. His claim: the real goal is “decision support advice error reduction quality assurance” [09:20]. Topol’s version is AI for the simple things: Alexa for symptom talk, drug distribution, post-op coaching [09:20]–[10:54]. Pasquale then asks whether those “simple” tasks are simple [10:54].

📚 Deeper theory

• Why the dream stalled. Medicine mixes narrow, rule-like tasks (read a scan) with open ones (a sick patient “walking in off the street”, as Skowno says at [34:12]). Systems do well on the first and badly on the second. That is the 1980s expert-system story and Polanyi’s paradox (2.5).
• Search as medicine. Google’s expert-built cards show a pattern: general-purpose tools get specialised, supervised layers in high-stakes fields. Domain curation beats raw scale where error is costly. The later LLM era challenges this (see Part 1, current debate), though the reliability gap is the same concern.
• The jurisdiction view (2.1). “Decision support” is the diplomatic answer: AI enters the task but not the licence. It leaves the border of the profession where it was.
• Topol’s optimism is conditional. In Deep Medicine his hope is that AI gives clinicians time (“the gift of time”) to be human. Whether the time is returned to care or taken as extra throughput is the political economy question (2.4). The scribe studies in Part 1 show that tools save minutes but do not raise throughput in some settings.

Where others push back.

• The Susskind camp: decision support is a halfway house; once the system outperforms clinicians, the human in the loop is a cost and a risk.
• Meehl-style critics: if the algorithm is better, “support” lets the clinician’s worse judgement override it.
• Access advocates: “indefinitely delayed” ignores places with no clinicians at all, where imperfect automation is better than nothing (Pasquale concedes the dream is noble [06:15]).

What they leave out. Patient-facing use. Pasquale mostly speaks of professionals, but the demand he notes for dermatology tools [15:35] is consumer-led, and LLM chatbots have made that the main story since 2022 (from memory; not checked).

Check yourself

• Why does the Google health card count as evidence that automation is stalled?
• What is the difference between decision support and automation in terms of licence?
• Name a task where the Meehl argument beats “decision support”.

---

2. Five legal questions the “simple” tasks raise

🎙 In the talk. Pasquale takes post-surgery care coordination by a voice assistant, and chatbot therapists, and lists five questions [10:54]–[14:01]:

1. Licensure. Humans in these roles “would need to get a license from the state”; should the system get “some level of licensure or quality assurance” [10:54]?
2. Institutional liability. Even without licensure, a glitch can harm. A large vendor would likely make the hospital sign an agreement that would “disclaim all liability”, leaving the hospital liable though it “may not be their fault” [12:28].
3. Malpractice. Will physician-liability law transfer, or will AI be treated like “a book that you might consult” [12:28]?
4. Informed consent. Do people “truly consenting to this ai understanding it or not” [14:01]?
5. Privacy “will always be a major concern” [14:01].

On parity, he cites Ryan Abbott, who argues for treating AI and humans “in parallel ways”; others say institutional dynamics prevent that [14:01].

📚 Deeper theory

• The book-versus-colleague analogy matters. If AI is a reference book, the doctor is fully responsible for using it, as with a textbook. If it is a colleague or consultant, responsibility spreads. The law has treated clinical guidelines as evidence of the standard of care, not as the standard itself (from memory; not checked), which is why AI output could be admitted as evidence without being the rule.
• Contract allocates what tort leaves open. The disclaimer Pasquale describes shows how a strong vendor moves risk by contract. That is a bargaining-power story: the hospital cannot refuse Amazon’s terms. It links to the competition-policy point at [46:42].
• Licensure is about accountability, not only skill. A licensed person can be sanctioned and can lose the right to practise. A bot cannot be struck off, so licensing it means licensing its maker or deployer, a different legal object. (Argument is analytic; no source checked.)
• Consent. Informed consent doctrine asks what a reasonable patient would want to know. Whether “an algorithm was used” is a material fact is open. Pasquale’s New Laws adds a related rule that AI should not pretend to be human (see pasquale-new-laws-of-robotics).
• Privacy. The Paro seal’s recordings [17:09] raise questions about storage and sensing that data-protection law (GDPR in the EU; Australia’s Privacy Act) handles unevenly for health data (from memory; not checked).

Where others push back.

• Innovation-first view: licensure for software would freeze small developers out; ex-post liability and market reputation are enough.
• Standards view: regulate the process (testing, monitoring) via device rules, not by treating a bot as a licensee.
• Abbott’s neutrality: do not hold AI to a special, higher bar; hold it to the same standard as a human, then let it win by being safer.

What they leave out. Insurance as the real regulator: in practice, malpractice insurers and payers may decide what gets used long before courts do (my addition; not argued in the talk).

Check yourself

• Why might a hospital be liable for a vendor’s error even when the vendor disclaimed liability?
• What is lost if AI is treated as “a book” and not a colleague?
• Who would a “licence” attach to for a chatbot?

---

3. Death algorithms, caregivers and the cost story

🎙 In the talk. Four cases [14:01]–[17:09]. The first is the ICU “death algorithm”: a model comparing a patient’s vitals with “say a hundred thousand or a million” cases to predict likelihood of recovery [15:35]. Pasquale lists upsides and downsides. Upside: “objectivity” for hard decisions, and comfort for families if “there’s never been a recovery of a patient given a certain set of vital statistics” [21:47]. Downside: it “could rigidify funding and reimbursement guidelines” (an insurer might refuse to pay below a score), cause “clashes of culture”, and enter “rationing” [21:47]. The second is dermatology: comparing a mole with a million cancerous and a million non-cancerous images; patients will demand it to avoid waiting months [15:35]. The third is robotic and avatar caregivers: the Paro seal that bats its eyes, which records the environment; and Care Coach’s animated pet voiced by workers in the Philippines, who ask “how are you doing” and “have you taken a walk today” [17:09]–[18:42]. He asks about pay parity with direct caregivers, and whether recorded conversations “may well become a corpus” for later automation [18:42]. One study measured cortisol in urine to test if care robots reduce stress [18:42], and he asks about “how it works and should it work” and whether it treats people “in a sort of behavioristic way” [18:42]. He then notes that a “labor shortage” may mean “low wages for human caregivers” [20:15].

📚 Deeper theory

• Prognosis is statistical, decisions are not. A model that says a group of patients with such vitals mostly died within weeks speaks about a reference class, not about this patient. The leap from class to individual is the standard problem of prognostic statistics (from memory; not checked). Clinicians use it with judgement, which is Pasquale’s point about leaving the decision to a person.
• Calibration and base rates. For rationing uses, what matters is whether the predicted probabilities are calibrated for this hospital’s population. Models trained elsewhere can shift (2.6). A threshold such as “never recovered” is a claim about the training data.
• Quantification as governance. Once a score exists, payers can attach rules to it. This is what Pasquale calls rigidifying. It is the same pattern as Goodhart’s law (2.7): the score becomes a target.
• Paro and “relational” machines. Sherry Turkle, Alone Together (2011), warned that machines that perform care may lower our expectations of people. The talk raises behaviourism and cortisol; Turkle gives the long argument (not cited in the talk).
• Hidden labour (Gray and Suri, Ghost Work, 2019). Care Coach is a case of a human behind an “AI” face. The legal questions on the pay gap, and on whether recordings can train later systems, are data-labour questions (from memory; not checked as applied here).
• The cortisol measure shows how “it works” is chosen. A biomarker is a convenient proxy for well-being. Compare the cost-as-proxy error in Obermeyer (2.6): convenient measures can mislead.
• The labour-shortage argument ties to Baumol (2.4): automation looks attractive in sectors with stubbornly rising labour cost. Pasquale’s response is to question whether the shortage is a wage artefact.

Where others push back.

• For the algorithm: human prognosis is itself erratic. If a calibrated model is better than clinicians’ guesses, withholding it is also a harm (Meehl, 2.5).
• For the robots: some isolated elderly people have no other companion, and a pet-like device may be better than nothing. The evidence base is thin either way (from memory; not checked).
• For the avatar model: offshore voice work is a job; the pay gap is a labour-law issue, not a reason to ban the model.

What they leave out. Voices of patients and carers. The talk treats the older person as an object of design; none of the speakers asks what residents prefer. Also missing: how a “no recovery” score would be explained to a family, which is a communication task as much as a legal one.

Check yourself

• Give one benefit and one harm of the death algorithm from the talk.
• Why does the cortisol study show a general problem with “does it work?”
• Why might Pasquale’s labour-shortage point not apply in a country with a real shortage of trained nurses?

---

4. Triage, gaming and the liability grid

🎙 In the talk. If hospitals use AI to triage ER patients, “people may alter their behavior to be recognized by the algorithm” [21:47]. His example: asked to rate pain from one to ten, a doctor said the way to get strong pain drugs is to “say an eight or nine” [21:47]. Pasquale adds i’m certainly not endorsing that [23:19]. Wang’s point is that secrecy is opaque and “not really that responsible to larger citizenry”, but revealing the method invites gaming [23:19]. Then the Price, Gerke and Cohen grid [23:19]: is the AI recommendation for standard or non-standard care, is it correct, does the physician follow it? That gives “at least eight different possibilities”, and the “two red boxes” are rejecting correct, standard advice, and following incorrect, non-standard advice [23:19]–[24:51]. He asks how “courts around the world” will treat them [24:51].

📚 Deeper theory

• Why only two boxes are red. If AI advises standard care, the doctor follows and it is wrong, the doctor is covered by the standard of care. If AI advises non-standard care and the doctor follows, the doctor has left the safe harbour. If the AI is right and non-standard, and the doctor rejects it, the doctor stays within the standard, so there may be no liability. The grid’s practical message: the law rewards conformity. (My reading of the grid from the talk’s description; the underlying paper was not checked.)
• Defensive medicine. If liability turns on deviating from the norm, clinicians have a reason to ignore useful non-standard AI advice. This could slow useful AI as much as it slows harmful AI.
• Automation bias (2.2). The second red box is a human factors problem too: following a confident wrong output.
• Gaming is Goodhart’s law with a human face (2.7). The pain scale is a long-known gameable measure, and the same will be true of any triage score people can learn. The design response is to avoid single, public, high-stakes scores, or to audit them.
• Wang’s argument. Disclosure is not neutral. If a hospital publishes how it triages, patients learn to present as the algorithm wants, and the algorithm’s norm becomes the norm. This is Foucault’s normalisation applied to disclosure (abstract of Wang checked; full paper not read).
• Triage is rationing. Medicine has long accepted triage rules (battlefield, ER), and the ethics literature distinguishes rules agreed in public from hidden ones (from memory; not checked). The AI case turns on whether a model’s cut-offs are as contestable as a published protocol.

Where others push back.

• For transparency anyway: gaming of a pain scale is a feature of a bad scale, not of openness. Better to publish, then design rules that are hard to game.
• For secrecy: in some adversarial settings (fraud), secrecy is normal.
• Against the grid: it assumes a clear “standard of care” and a clear “correct” answer, which are often contested ex post.

What they leave out. A third actor: the patient’s right to ask for human review, and the patient who is mis-sorted without knowing it. Also whether staff can see why the model scored a patient low, which matters for overriding it.

Check yourself

• Explain the two red boxes in one sentence each.
• Why might the grid make doctors ignore good non-standard advice?
• What would an audit-based answer to the transparency dilemma look like?

---

5. Automation versus augmentation — handoffs, data and regulators

🎙 In the talk. Weatherall asks whether we want “automation nor do we just want augmentation” [29:32]. Pasquale agrees the divide is critical and gives a self-driving example: vehicles designed to “turn over responsibility to the driver” right before a crash, so “it was the driver’s fault” [31:05]. The medical worry: professionals “with a great deal of responsibility but then without a lot of support” and without “institutional support to say contradict the advice” of an AI [31:05]. He names Shneiderman and human-computer interaction [31:05]–[32:39].

On data, Skowno says there is “a huge diversity of data types” [32:39]. Imaging is standardised (DICOM) and so suits early automation; clinical notes and “recording human decisions” are “phenomenally difficult” [34:12]. EHRs boomed in the US, but “we’re quite a way away from saying all of the data is there” [35:46]. Weatherall asks who sets standards. Skowno names ICD-10, HL7 and FHIR [37:20] and says hospitals differ widely in how far they have adopted them.

On regulators, Pasquale points to state capacity: ML professors get offered “10 times their salary in the private sector” [38:55]. He suggests industry fees, as the FDA uses [38:55]. On bias he cites Obermeyer [40:28] and calls for “a pretty coordinated regulatory response” nationally and perhaps multinationally. Skowno adds that a socialised system may support a better-funded regulator [41:42]; Pasquale agrees that bias issues are “embedded in larger political economic systems” [41:42].

📚 Deeper theory

• Moral crumple zone (2.2). Pasquale’s driver story is Elish’s concept in plain words. Without authority to override, “human in the loop” becomes a liability sponge.
• Bainbridge’s irony. The more reliable the AI, the less practice the human gets, and the less able they are to catch its rare failures. A diagnostic assistant that is right 99% of the time may erode the skill needed for the 1%.
• Why the data gap is a policy gap. Imaging gets AI first because it has standard formats and abundant labelled data. Unlabelled, free-text, context-heavy care (a GP’s first visit) gets less. So capability follows data infrastructure, which is a public investment choice (2.6).
• Regulatory capacity. Weatherall’s question whether regulators “are equipped” [38:55] is about the adaptive device problem (2.3): a model that changes after approval needs continuous monitoring, which needs staff who understand it. Salary gaps make that hard, and the political economy of the system (profit-led or socialised) matters.
• Comparative systems. Skowno’s point [41:42] is that in a more socialised system, the data is held by the public sector and the regulator is not competing with a hospital chain’s pay scale. Evidence for this is not offered in the talk and is plausible, not shown.
• Standards as a commons. FHIR’s growth shows a technical standard becoming de facto law for what data can flow (2.6).

Where others push back.

• Techno-optimists: bad handoffs are an engineering problem, solved by better interfaces and training, not a reason to slow adoption.
• Market view: regulators don’t need to match private pay; they need to buy audit services and rely on third-party testing.
• Skowno’s own limit: he says the right answer “depends on which area of medicine” [34:12], which resists a single policy.

What they leave out. Low-resource settings. The talk assumes hospital systems with EHRs; the access argument for AI (Pasquale’s own opening, [06:15]) is where data and standards are weakest.

Check yourself

• Why is the self-driving “handoff” a liability problem and not only an engineering one?
• Why does imaging get AI before general practice?
• Why might a regulator need adaptive-device skills that it cannot afford?

---

6. Big tech, de-skilling and the doctor as a data source

🎙 In the talk. Weatherall raises Amazon’s purchase of One Medical, a primary-care provider, and asks what it means for doctors, given Amazon’s “reputation for monitoring its employees” [43:35]–[45:09]. Pasquale answers that US EHRs were experienced as “a sudden increase in unpaid labor” with “terrible user interfaces”, and bets “apple or google would come up with a more usable user interface” than the incumbents [45:09]. He sees promise in “data aggregation and … long-term investment”, and says “to the extent that … the state shrinks from this long-term investment” some will welcome big tech [46:42]. But he warns of de-skilling, citing Carr’s The Glass Cage and the pilots case, where some automation “preserve[s] a pilot’s sense of skill and autonomy” and some does not [46:42]. He wants competition law to consider “conglomeratization”, not just cartels [46:42]. Skowno says one can’t be strongly for or against: big tech has “a vast skill set”, and “it would be crazy” for health systems not to benefit; policy settings decide whether it is “open and transparent” or a “dystopian take on data” [48:16].

📚 Deeper theory

• The vertical-integration worry. A firm that owns the clinic, the pharmacy, the cloud and the AI can use data from one layer in another. Antitrust has historically focused on market share in one market; “conglomerate” harm (Lina Khan, “Amazon’s Antitrust Paradox”, 2017) is the recent challenge to that narrow frame (from memory; not checked). The FTC chose not to sue before the deal closed and said it would keep investigating (Gulf News).
• Workplace surveillance. If doctors are employees, their decisions become data for management. That is “algorithmic management” (as studied for warehouses and gig work). It is Weatherall’s point, and Pasquale says it deserves “to be brought front and center”.
• De-skilling is old. Carr’s pilots echo Bainbridge (2.2) and, much earlier, Marx on machines and craft skill and Braverman, Labor and Monopoly Capital (1974), on “deskilling” of work. The medical version also has a training problem: if juniors never practise the task, seniors of the future will lack the skill.
• Usability as politics. The EHR story is about who the software is built for: billing and compliance, not clinicians. Epic and Cerner sold to hospital administrators. Pasquale’s bet that Apple or Google would do better is a claim about incentives (they sell to users) and is not shown (no evidence offered).
• The public-investment alternative. Pasquale’s line about the state shrinking from investment is a quiet argument: big tech enters where public funding recedes.

Where others push back.

• Efficiency view: integration lowers cost and reduces friction (one app for booking, records and prescriptions). The dystopia is a regulatory failure, not an inevitability (Skowno’s view).
• Doctors’ view: some physicians welcome salaried employment by large firms for the relief from running a practice.
• Pasquale’s own caution: he is careful to say his UI bet is a guess.

What they leave out. The patient side: whose consent covers using visit data to train models, and whether patients can opt out. Also non-US cases: how this plays out where the public system holds the data (Skowno’s AWS work is public-sector [48:16]).

Check yourself

• Why might EHR complaints support the case for big tech, and what is the counter-argument?
• What is “conglomerate” harm, and why does Pasquale ask competition law to consider it?
• Give one example of automation that preserves skill and one that erodes it.

---

7. Gut feeling, protocols, bias and exclusion

🎙 In the talk. Asked about “the paradox of the gut feeling”, Skowno says it is the sense that “that child over there is just not quite well enough in the ed”, and that it “accumulates over time” and is hard to put into words, which is why 1980s expert systems found it “incredibly difficult to codify” [49:50]. His own view is blunt: “our reliance on that gut feeling in medicine probably causes more harm than good on average” and he’d “favor a more open protocol” [49:50]. But we must keep it, as doctors’ and nurses’ autonomy, and not “over trust it” [51:23]. Weatherall brings in Slobogin’s Just Algorithms, and suggests a middle ground: require documentation for going against evidence-based practice or the AI [51:23]. Skowno: protocolisation of simple processes (antibiotic before surgery, cleaning skin for a drip) gives “better results”, but where it fails and gut instinct is needed is very hard to determine [51:23]–[52:57].

On exclusion, Weatherall raises Indigenous data and a “two-tier system” [52:57]. Skowno warns about large data swamping small groups, and says “the most lethal biases are the ones that are invisible” [54:30]. Pasquale points to ongoing “protocolization” of dataset representativeness, mentions Bustamante’s DNA work [54:30], and raises AI as a screening tool before scarce professionals: in a system where “it might take months to see a radiologist”, AI could triage scans, which is “a more rational allocation of resources than just brute luck” but risks missing people the AI fails to flag [56:02]. Skowno ends hopefully: AI can “identify those little areas of bias” [56:02].

📚 Deeper theory

• Kahneman and Klein (2.5). Gut feeling is reliable in high-validity environments with fast feedback, unreliable otherwise. A paediatric nurse who sees many sick children and learns outcomes has the former. This helps explain why both Skowno’s scepticism and his trust can be right in different tasks.
• Documentation duty as a design pattern. It puts friction on deviation without banning it. It also creates a record that can later be audited and fed to the model, and a record that can be used against the clinician in court (my point, not made in the talk). Compare the “learned intermediary” idea (2.3).
• Protocolisation’s evidence. Gawande’s checklists and the surgical-safety evidence (2.5). Limits: checklists suit simple, repeated steps and fit badly where cases vary.
• Invisible bias is the dangerous kind. Skowno’s line matches Obermeyer: the cost proxy passed standard accuracy checks, so no one saw the disparity until it was audited from outside (2.6).
• Screening-first as two-tier care. If AI screens before humans see the patient, the people the AI scores low may never reach a person. This is a design of who gets human attention, an allocation of scarce expertise. The test: is the AI’s error rate similar across groups, and is there a route to appeal? (My framing.)
• Indigenous data sovereignty. The question is not only accuracy but control: who owns, governs and benefits from the data. In Australia and elsewhere, this is a distinct policy field (from memory; not checked). The panel raised it and moved on within a minute, so the talk gives little.
• The use of AI to find bias (Skowno’s closing hope) is real but conditional: you need a reference to compare against, and the model’s own training data may contain the bias it is asked to find.

Where others push back.

• Against protocols: rigid pathways can harm unusual patients and demote clinical skill (Illich-style and Freidson-style critics, 2.1).
• Against Skowno on gut feeling: some of the most valuable clinical signals (the “sick child” look) are real and not yet capturable; discarding them has a cost.
• For a rights-based reading of exclusion: representation is a matter of justice and consent, not only of model accuracy.

What they leave out. Concrete mechanisms for the exclusion problem (how Indigenous communities would govern data use); accountability for the “AI screen first” design; and any evidence on how often the documentation duty changes decisions.

Check yourself

• When is gut feeling likely to be reliable, by Kahneman and Klein’s test?
• What does a documentation duty add to, and take from, a clinician’s autonomy?
• Why is an AI that screens before a human visit a two-tier risk?

---

Putting it together

Mental model. Medical AI is a fight over jurisdiction inside a political economy. Pasquale’s claim is that the work should stay with licensed humans, with software as a supporting tool, because accountability lives in professions and institutions. The fight is decided less by model accuracy than by five things the talk keeps returning to: who is liable when advice is followed or rejected (the 8-way grid), who bears the handoff when automation fails (the crumple zone), whose data counts (imaging yes, notes and Indigenous data less so), whether regulators can keep up (pay gaps, adaptive devices), and who owns the infrastructure (big tech, EHR incumbents, public systems). The metaphor “peaceable kingdom” is the political choice that follows: keep AI and clinicians distinct and let each correct the other, and use law to stop the cost story from hiding quality and access losses.

Open tensions.

• Transparency vs gaming. Pasquale names it and does not resolve it [23:19].
• Protect judgement vs protocolise it. Pasquale defends professional expertise; Skowno wants more protocol and thinks gut feeling does net harm [49:50]. The talk shows the disagreement more than it settles it.
• Parity vs institutional responsibility (Abbott vs the sceptics) [14:01].
• Big tech as a UI fix vs a data monopoly [46:42], [48:16].
• AI as screen vs as bias detector [56:02]: the same technology could widen or narrow gaps.
• Timing. Recorded July 2022, before ChatGPT. The tricorder was “indefinitely delayed” then; the patient-facing chatbot arrived within months. The talk’s legal questions have held up better than its technology picture.

Reading path.

• Start: Obermeyer et al. 2019 (short, concrete); Elish, “Moral Crumple Zones” (2019); Pasquale’s talk on his own book, pasquale-new-laws-of-robotics.
• Then: Shneiderman, Human-Centered AI; Carr, The Glass Cage; Price, Gerke and Cohen on liability; Topol, Deep Medicine. For Pasquale’s wider argument on automated judgement see pasquale-machines-judging-humans; for law and the professions, pasquale-future-of-ai-in-law.
• Deeper: Freidson, Profession of Medicine; Andrew Abbott, The System of Professions; the Susskinds, The Future of the Professions; Bainbridge, “Ironies of Automation”; Kahneman and Klein (2009); Bowker and Star, Sorting Things Out; Wang (2022) with its replies.