“Inscrutability was always inevitable, right?” by Steven Byrnes

Here's a 2022 Eliezer Yudkowsky tweet:

I find this confusing.

Here's a question: are object-level facts about the world, like “tires are usually black”, encoded directly in the human-created AGI source code?

• If yes, then (1) yo be real, and (2) even if it happened, the source code would wind up with wildly unprecedented length and complexity, so much that it becomes basically inscrutable, just because it's a complicated world (e.g. there are lots of types of tires, not all of them are black, they stop looking black when they get muddy, etc. etc.). See Yuxi Liu's ruthless “obituary” of Cyc.
• If no, then the human-created source code must be defining a learning algorithm of some sort. And then that learning algorithm will figure out for itself that tires are usually black etc. Might this learning algorithm [...]

---

First published:

August 6th, 2025

Source:

https://www.lesswrong.com/posts/ohznigkLX5CNwaLSz/inscrutability-was-always-inevitable-right

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.