Anthropic warns that it’s very close to RSI capability, aka self-building AI
RSI is the point at which AI grows so good at training itself that it spawns a dramatic acceleration of AI capabilities. To the AI doomers, RSI represents the cataclysm from which there is no return.
Key points
- Anthropic's revised S-1A filing warns the company may be close to recursive self-improvement capability.
- Anthropic says it has already used AI to develop models with capability leaps it did not recognize at the time.
- Anthropic says no current model has crossed its RSI threshold and pledges to keep humans in R&D.
Anthropic's revised S-1A filing on Monday warned that the company may be nearing recursive self-improvement, or RSI, the point at which AI becomes capable of training itself and capabilities accelerate sharply. The warning appeared in the fine print of the filing, which also showed rising revenue and rising expenses.
Anthropic said it has already used AI to develop new models whose capability leaps it did not recognize at the time. It added that internal estimates and expectations for future models, rather than current ones, suggest it may imminently meet its Responsible Scaling Policy definition of "dramatically accelerated progress attributable to AI."
What the filing says
Anthropic said none of its current models has been designated as exceeding the RSI threshold and that it continues to assess the underlying data. The filing also cautioned that using AI to help build more capable AI could pose a considerable safety risk, particularly if it sharply accelerates progress, and said the area is covered by its Responsible Scaling Policy.
Anthropic said it does not anticipate being able to safely use Claude to fully autonomously design and build successive generations of frontier models without meaningful human supervision or understanding, if and when that becomes technically possible. It said it is considering updates to the policy to keep humans in control of frontier model development.
Using AI to help build more capable AI could also pose a considerable safety risk, particularly if it sharply accelerates the pace of progress.
— Anthropic
Skepticism and context
The article notes that an S-1 serves as both disclosure and marketing, and that claims of being close to a self-building model could fall into the latter category. It also says Claude's capabilities, especially in coding, math and cybersecurity, have grown sharply this year.
