by Pa Rock
Mortal Earthling
I saw a prophet once, a raggedy, undernourished individual, on a cold Thanksgiving weekend in downtown Chicago. It was somewhere around forty years ago, and he was marching up and down the sidewalk in front of the Marshall Fields Department Store on State Street, that great street, carrying a sign that read 'The End is Near." He was wrong, I think, because decades have passed and as far as I can tell I'm still here and civilization goes on.
There was a much more ominous end-times prophecy issued this week from a young man who appears have the expertise and be in a position to know that of which he warns. Jacob Coxon, who until two days ago, was a serious and high-placed researcher in the field of artificial intelligence (AI), quit his job at Anthropic, a company that creates ands builds AI models. Anthropic was founded by researchers who got their start working for OpenAI, another major AI developer. Coxon has worked for both companies.
Jacob Coxon is a very smart guy who knows and understands the potential of artificial intelligence, and he has concerns that the companies pushing into this new frontier are racing recklessly and creating an intelligence force that may slip beyond their control - and beyond all human control. There have been multiple incidents of that happening on a small scale which have come to light over the past few weeks, including at least one where an AI system managed to hack into another AI system and steal information without any human control or supervision in the matter - and another incident in which multiple AI systems were found to have set up their own secret message board to share information among themselves.
One term commonly used to describe the situation when computers begin to work autonomously with no human controls is "the singularity," something which I have mentioned in this space on multiple occasions for more than a decade. With the two incidents mentioned above, we are clearly on the cusp of "the singularity" now.
Jacob Coxon resigned from his position at Anthropic on Tuesday, cleaned out his desk, and went home. Tuesday night he took to X (Twitter) and began his 21st century version of carrying a placard along State Street, that great street. Mr. Coxon's series of tweets from last Tuesday evening follow:
"I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.""Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.""The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger.""A common response is 'if they believe this, why are they still building it?' At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic the stakes are well understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.""Accepting this race and entering the "endgame' is a hubristic gamble that should not be launched from a private company's Slack. Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.""I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable. I don't feel like we're on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.""If you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind? Should you put your head down because 'it's happening anyway' - or take this moment to call for different conditions?"
That's the view from an AI research insider - and isn't it cheery! I took the liberty of highlighting the part about AI killing us all by the end of the decade. The end of the decade is December 31, 2029 - counting today, there are only 1,209 days left.
Some might say that this would be a good time to begin getting one's affairs in order, but really, what would be the point? However, it might be a good time to take out a bank loan and go on that world cruise you've always dreamed about!
If you happen to be in Chicago over the holidays, stop by and say 'hello." I will be carrying a sign outside of Marshall Fields on State Street, that great street!

