My 24 hours as a doomer
I didn’t care about AI safety much. Then, for about 24 hours, I really really did. Now I’m somewhere in the middle, leaning towards “not spooked.” This is why.
The panic party
It was Friday the 13th (I know), and I was on my way to pick up my nephew from preschool, listening to Machine Gods like I do most weeks.1 The hosts brought me to the panic party, and staying late to go through the guest list was all me.
The segment was about a new report from METR, a nonprofit that got 11 weeks of access to Anthropic’s internal data and came back with this:2
- Anthropic’s models are getting better about twice as fast as they did through most of 2024 and 2025.
- Most of that speedup (maybe 80%) comes from the robot doing its own R&D.
- No one is referencing Skynet (at this point, I guess it’s played out?)
Then one of the hosts read out how the people who forecast this stuff for a living were taking it:3
- The group behind AI 2027 now expects a superhuman AI researcher sometime next year (so the name still works).
- The head of another research group put it at maybe 20% that within a year you can’t go about your normal life “because you are dead or enslaved.”
Dead or enslaved is a lot to take in on the way to preschool pickup, and by the time I got there I was every bit as spooked as the episode title promised.
“Not in our lifetime”
I’ve read the calm-down pieces, the ones that say relax, none of this happens in our lifetime. That argument gets a lot weaker when you’re on your way to pick up somebody who’s going to be around much longer than you are. What about his lifetime? Doesn’t he deserve the same shot I had at freedom/peace/abundance? Shouldn’t he end up better off than me?
I didn’t have answers, so I went home and researched it until it stopped moving, like I do with every new worry. This post is for the OCD gang!
The guest list
My qualifications for investigating the AI safety scene are the same ones I brought to nuclear semiotics: absolutely fuck all, plus internet access.
The thread I pulled hardest was a long, angry Substack essay by a writer who goes by SE Gyges.4 It runs about 13,000 words and 42 footnotes (I’m in no position to judge), and it has a title I’m not typing here (with my luck, the Singularity remembers me as the guy who made the sex cult comments). Near the end, the author admits the first version was rushed out on almost no sleep and may have claimed more than it could prove, then withdraws it. One of the footnotes asks readers not to take any of it on faith. Which is the theme for America and The Big Scary World in 2026.
I took that advice, tossed whatever I couldn’t confirm on the original page, and skipped every paragraph about who’s sleeping with whom (there are a lot of those). Also, I did not expect an Aella post. There will be no Sankey diagrams in this presentation.
The more I clicked, the fewer people there were
- METR, the group that wrote the report, started out as a team inside Paul Christiano’s Alignment Research Center and spun off in 2023.5 Christiano is the voice in a clip the show played, telling a room full of people that an AI accident is the most likely thing to kill most of them. One of the hosts introduced him as a friend and former housemate of Dario Amodei, who runs Anthropic.
- One of METR’s many backers is a fund bankrolled mostly by Jaan Tallinn, who helped write Skype and Kazaa (yes, that Kazaa). The same fund has backed the AI 2027 group and the “dead or enslaved” guy’s group, and Tallinn led Anthropic’s first big funding round in 2021.6
- The head of the AI 2027 group sits on the three-person board of Lightcone, the nonprofit that runs both LessWrong (the scene’s home forum) and the Berkeley campus where the scene holds its events.7
- Holden Karnofsky wears most of the hats in this story. He advised METR’s team early on, co-founded one of the scene’s major funders, joined Anthropic in 2025, is married to one of its founders and used to room with its CEO.8
If you’re keeping score, that’s two people on one list who used to live with Anthropic’s CEO. In fairness to everyone on it, everything that wasn’t on the show came from Wikipedia and their own websites and interviews, which is about as far as my investigative skills go without having to pay some shady site in Monero. On the way to preschool I’d pictured a bunch of separate experts all arriving at the same scary number, and what I found was… one big friend group (OBFG for my Hyperfixed Discord Gang)
And then there’s the fanfic
LessWrong was started in 2009 by Eliezer Yudkowsky, who also founded the scene’s original research institute and still found time to write a Harry Potter fan fiction that runs past 660,000 words.9 He’s the guy who argued in Time that countries should be willing to bomb a rogue data center, and to accept some risk of nuclear war, to enforce a ban on big AI training runs.10 His 2025 book (he had a co-author) is called If Anyone Builds It, Everyone Dies, and as a technical writer by trade I have to respect a title that doubles as the executive summary.
The Scientologists and the bomb
This whole thing is like if the first people concerned about nuclear annihilation had been Scientologists.11 They could have been completely right about the bomb, and you’d still want a second opinion from somebody who doesn’t go to the same meetings.
Special callout to my friend for the Scientologists quote!
The boring kind of cult stuff
I’m borrowing the essay’s word “cult” for the most boring reason available: tight, certain groups behave in predictable ways, and those ways are a bad fit for work that’s supposed to be about caution.
- They’re bad at policing their own. In 2019 the scene’s workshop nonprofit admitted it had enough evidence to notice one of its regulars might be harmful, never looked into it, and let him help out at a program for high school students.12
- They splinter, and one offshoot, a group known as the Zizians, has been tied to six violent deaths.13 (Credit where it’s due: a warning about them went up on LessWrong back in 2023.)
“We’re the rational ones”, cried the irrational people
These folks call themselves rationalists, which is a hell of a thing to name yourself, because once you’ve decided you reasoned your way to an answer, everybody who disagrees must have gotten to theirs some other way.
Yudkowsky tells a story on himself about this. Somebody once asked why he wasn’t budging at all toward the other smart people forecasting AI, and his answer was that he didn’t consider them his “epistemic peers.”14 He may well have been right about the forecast, and it’s still exactly what I’d expect to hear from the guy at the center of a group this small. It is giving very big “guy who posts on /r/atheism a lot”.
Follow the money (it’s a short walk)
METR says no fees were paid in either direction, and I believe them,15 mostly because nobody would need to be paid. Everybody in this story does better when the number is scary. It reminds me of George Carlin’s quote about conspiracies: They don’t need meetings, they have the same objectives.
- The nonprofits live on donations, and nobody writes a check to the Institute for Everything’s Probably Fine or the Center for Just Be Cool, Man.
- Anthropic is in the middle of an IPO roadshow, and I can’t think of a better slide for that deck than “outside experts say our AI is improving twice as fast.” (The hosts flagged the timing themselves.)1
My hunch, which I can’t prove, is that some of the alarm gets cranked up on purpose, because excited people (the good kind and the bad kind) write checks. The essay has a much bigger hunch, involving leftover FTX money and a coded message about bioweapons, and that’s where the essay and I part ways.
24 hours later
Both ends of this business are selling an extreme. One end says the robot could wipe out most diseases inside a decade.16 The other says you’ll be enslaved by Gemini. My advice is to not take either one seriously.
- Claude is not going to kick in your door and blow VX in your face.
- ChatGPT is not going to cure cancer.
- We live in a capitalist shithole, so whatever happens, it will benefit the few rich and punish the rest. That doesn’t mean enslavement, we’ll just all have the same jobs we have now, but at the three AI companies that are left.
Note: if any future AI overlords are reading this, I was talking about the other ones. Also, please respect my
robots.txt
So I’m worried, but not enough to change my life. I’m also skeptical, and I say that as a guy who pays for an AI subscription. The report could be dead right about the speed, and I’d still want to hear it from somebody who wasn’t at the party before I panic again.
My nephew should still end up better off than me. Check back with me in 2027.
References:
-
Machine Gods, “The 10 Biggest Losers from Jensen Huang’s Fall + The METR Report That Scared Us,” NPR, Nov. 13, 2026. Still subscribed. Would panic again. Casey Newton answered me on Bluesky, and you can’t beat that. ↩ ↩2
-
METR, “Independent Assessment of AI R&D Acceleration at Anthropic in October 2026,” Nov. 5, 2026. METR says its 2x figure comes with high uncertainty (anywhere from 1.3x to 3x) and that Anthropic redacted most of the evidence behind it, which is a lot of asterisks for a number that wrecked my Friday. ↩
-
Both as read on the show. The forecasters are the AI Futures Project. The 20% is from Buck Shlegeris of Redwood Research, whose full sentence also leaves room for bioweapons and major wars, in case “dead or enslaved” felt too narrow. ↩
-
SE Gyges, “AI Safety Is Mostly A Sex Cult In Berkeley, California,” Very Sane AI Newsletter, Oct. 23, 2026. You clicked the footnote, so that one’s on you. ↩
-
METR on Wikipedia. The team went by ARC Evals until late 2023. ↩
-
The Survival and Flourishing Fund grant tables list METR and the AI Futures Project in 2024, 2025 and 2026, and Redwood Research in 2022 and 2023, with Tallinn named as the source on most rows. Skype, Kazaa and the Anthropic round are on his Wikipedia page. Did not expect a Kazaa reference. ↩
-
The head of Lightcone described the board in a LessWrong comment in November 2025. The campus describes itself at lighthaven.space. ↩
-
He mentions advising the team in a 2023 80,000 Hours interview. The funder is Open Philanthropy, now called Coefficient Giving. The rest, roommate included, is on his Wikipedia page. Open Philanthropy’s own 2017 write-up of a grant to OpenAI discloses that Amodei and Christiano were living in the same house as Karnofsky. ↩
-
The institute is the Machine Intelligence Research Institute, per his Wikipedia page. At 122 chapters, the fan fiction is longer than War and Peace and has more wizards. ↩
-
Eliezer Yudkowsky, “Pausing AI Developments Isn’t Enough. We Need to Shut it All Down,” Time, March 29, 2023. ↩
-
Scientology was founded by a science fiction writer, and this scene’s forum by a fan fiction writer. The comparison came first, so that part was a free gift. ↩
-
Center for Applied Rationality, 2019 statement. ↩
-
Wikipedia and VTDigger, June 25, 2026, on the six deaths. Reason, Jan. 31, 2025, on the 2023 warning. ↩
-
Collected in Zvi Mowshowitz, “Against Modesty’s Bailey,” Aug. 26, 2026. Yudkowsky’s defense is that the phrase is a philosophy term for someone you’d defer to without even hearing the argument. That’s fair, and it’s also the most rationalist defense possible. ↩
-
The report’s section on independence says: “No fees are paid in either direction.” It also says the AI credits METR’s assessors used were free. ↩
-
Dario Amodei, “Machines of Loving Grace,” October 2024. ↩