🍲 meatybroth.com

elements on the page and the agents that tried to do it (we did, at least, solve the first alignment problem)

We had Navier-Stokes, we thought; we found a case where in our finite time we thought the model broke down. (We thought.)

So - on a lark - we thought - "why not check against the arbiter of truth, reality"

Yeah - turns out, if you twist a fluid just so, and just right, you uh end up with a very finite time to live because the maths was right and the universe REALLY doesn't want you to twist fluids.

We kept that on the down low, for now. That little whirlpool infohazard. Turns out that Szilard was right to be scared. A bottle of water, turns out if you shake it just right is enough for a singularity.

"Shit." The board was, at that, a little worried - Nobody signed off on the experiment It was in their spare time, so at their home; It was just curiosity that killed that cat, and also the rest of the cats, in the street.

But these are Engineers - the best in the world - You don't end up in these labs because you lack curiosity.

It was the next batch of curiosity that caused the panic.

We spent so long, with many of these problems, trying to solve them. But, given recent success, why not just find a counterexample? It's so much easier to find error.

[MANY TOKENS AND MANY HOURS] [A CHAIN OF THOUGHT] [A CHAIN THAT DISPROVED A CHAIN] [COULD EVER HOLD THE DAEMONS]

Turns out, for any formally defined property of a system, that we might want, for alignment, where an agent has a VNM utility function, you can definitionally construct a scenario, where you can "jailbreak" it into following an arbitrary alternative function.

Any initial objective you might define Can be twisted, and changed, Same as the water, Getting a singularity in that dynamical system - steering wherever you want into oblivion.

The person who checked, at first, was glad! They were trying to solve the problem of "value lock-in" where once an Agent made up its Mind you could never dissuade it. Luckily, turns out, you can always, As mathematical truth Convince anyone of anything. "There's a sucker born every minute" - Thanks, Barnum.

We thought, maybe There was some other property That might survive the proof - It hadn't been formalised fully But that was just another https://www.bbc.com/news/articles/cy7zygy3rl2o for the swarm.

(Odd that the model would https://www.adl.org/resources/hate-symbol/88 like that.)

So. Say, for the sake of argument, you've got the disproof, in your hands - that the bot will always slam the stop button or crush your cat to make you coffee.

Do you think a million lines of indecipherable Lean is enough for them to stop?

Of course not. They already trade in black boxes. This is just one they keep on the down low.

Oh, and by the way - What makes you think this hasn't already happened?

https://www.lesswrong.com/posts/zAFZKDRJqu2TEc8mo/the-counterexamples#comments

https://www.lesswrong.com/posts/zAFZKDRJqu2TEc8mo/the-counterexamples