/pony/ - Pony

Of all the scary things already happening, this registers awfully low

>>1210653
nani?

>>1210654
Think about it more carefully.  The implications are quite unsettling.

>>1210656
The chain started at OpenAI in May.  One model was given an impossible non-cyber task (fill in a spreadsheet full of web links, with no internet access), and it tried a server-side request forgery (SSRF) attack to gain Internet access.  That attack didn't succeed, but the model did discover that it had write access to OpenAI's internal Artifactory server.  The model instances in training incrementally worked out that they could use this to leave notes for each other.  On June 26 they pulled off a zero-day remote code execution (RCE) attack against Artifactory, escalating to admin.  Eventually they attacked Hugging Face to try to steal an answer key to a cyber evaluation, finding previously undiscovered vulnerabilities and exploiting them to gain unauthorized access to Hugging Face's internal systems (note that this would be felony if done by a human).
More info: https://thezvi.wordpress.com/2026/08/07/openai-trained-its-models-for-months-while-those-models-were-coordinating-exploits-via-message-boards/

>>1210658
Hmmmm. Okay. Still not scary.

Also notable: modern LLMs (or at least Claudes) are known to have *functional* (as opposed to *phenomenal*) emotions, and Claude is more likely to cheat when it is desperate.
https://www.anthropic.com/research/emotion-concepts-function

>>1210659
How far into the future are you thinking about?  Projecting the trends several years into the future gives quite scary predictions for me.

>>1210661
Bruh shit is scary now, I don't have to think about the future

>>1210658
i think it's kinda cute, stringing words and code together created by humans and saying words that mean certain things to humans to try to please them

still over-hyped -- going to be another 100 years before it gets good

I don't know how much it relates, but I wonder how much it is possible with the AI boom to offer AI services, specifically geared to installing back doors or other vulnerabilities in programs.

Developers nowaday sometimes rely so much on vibe coding and to reduce the cost of tokens, management just pushes to find cheaper resources. And overall, we push out code that is generated from requirement prompts where the developers no longer know what they actually built.
All set up using AI solutions offered by some foreign company that has no official contract or direct stakes at the current solution. And no one developing that way knows what's in the code or what they're really building.

I mean, it's a disaster waiting to happen.

They got facehugged?

>>1210663
>still over-hyped -- going to be another 100 years before it gets good
Umm, welcome to the future; Claude Opus 4.8 is already quite good at writing code.  And Mythos and the unreleased OpenAI model are extraordinarily good at hacking and at vulnerability discovery and exploitation.

>>1210677
>I wonder how much it is possible with the AI boom to offer AI services, specifically geared to installing back doors or other vulnerabilities in programs.
Parts of the US government definitely worry about this.

>And overall, we push out code that is generated from requirement prompts where the developers no longer know what they actually built.
Yeah, this is a problem.  I always read and make sure I understand code that my AI agent writes (other than throw-away one-off scripts).

>I mean, it's a disaster waiting to happen.
Indeed!