LessWrong (30+ Karma)

Currently, only 5 companies in the world have access to frontier AI training compute and are also pursuing development of AGI (Google DeepMind, OpenAI, Anthropic, xAI, and Meta). This will still hol…

You can find Part 1 here. This resumes the weekly, already in progress. The primary focus here is on the future, including policy and alignment, but also the other stuff typically in the back half l…

Highlights

• We stress-tested 16 leading models from multiple developers in hypothetical corporate environments to identify potentially risky agentic behaviors before they cause real harm. In the sce…

The 1950s were crazy times. Human experimentation in the US was normalized in a way that would make modern IRBs implode from shock. After the Soviets tested their first nuclear bomb in 1949, war pla…

Audio note: this article contains 218 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

Recently, in a group chat…

It's currently possible to (mostly or fully) cheaply reproduce the performance of a model by training another (initially weaker) model to imitate the stronger model's outputs.[1] I'll refer to this …

Audio note: this article contains 147 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

TLDR: I develop a method …

Nate and Eliezer's forthcoming book has been getting a remarkably strong reception.

I was under the impression that there are many people who find the extinction threat from AI credible, but that fa…

Background Concept: What World Does The Referent Live In?

When we see some symbols representing something, it's useful to ask “what world does the referent[1] live in?”. Some examples:

• When I’m rea…

First, how and why I made this game

I spent several years working as the chief product officer of a software company in Melbourne, and during this time, there was a point where I raised some money f…

Audio note: this article contains 33 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

Linkpost to arXiv: https:/…

Thanks to helpful commenters on the original post, especially Kaj Sotala. It's mostly thanks to Kaj that, three months hence, I've now changed my view.

Three months ago, I wrote a post called AI Can…

Introduction

There are several diseases that are canonically recognized as ‘interesting’, even by laymen. Whether that is in their mechanism of action, their impact on the patient, or something els…

PDF version. berkeleygenomics.org.

This is a short miscellaneous list of projects that I think would help accelerate germline engineering. This isn't prioritized or comprehensive or anything—it's n…

I like debate. I have done for years. So I have been slowly trying to improve it. Here is a set of theories I had and things, experiments I've run so far.

Theory: Any debates are good.

Are any debat…

Ed and Anna are co-first authors on this work.

TL;DR

• Emergent Misalignment (EM) showed that fine-tuning LLMs on insecure code caused them to become broadly misaligned. We show this is a robust and sa…

I'd like to say thanks to Anna Magpie – who offers literature review as a service – for her help reviewing the section on neuroendocrinology.

The following post discusses my personal experience of t…

Ed and Anna are co-first authors on this work.

TL;DR

• Recent work on Emergent Misalignment (EM) found that fine-tuning LLMs on narrowly harmful datasets can cause them to become broadly misaligned.
• We…

A while ago I saw a person in the comments on comments to Scott Alexander's blog arguing that a superintelligent AI would not be able to do anything too weird and that "intelligence is not magic", h…