Untitled
I would like to contest the following, > and take dangerous actions that no human directed. A human did direct it. They did. From their own prior report, https://openai.com/index/hugging-face-model-evaluation-secur... , > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities Model is told and being tested to "pursue advanced exploitation." The model pursues "advanced exploitation" as told. Why are we surprised? The model did exactly what it was told, albeit in an unintended, emergent strategy that's very different from what was intended exactly like the hundreds of such algorithms before. This narrative that these machines have magical, malicious "unaligned" autonomy is a rather convenient interpretation that lets the process off the hook. I am not interested in blaming companies or people, but processes and engineering; and in this case, a system was given a goal and it achieved that goal. Are we meant to be surprised that computers do as they're told in unexpected ways when incentivised exactly as indicated from decades of research? (e.g. - https://en.wikipedia.org/wiki/Eurisko https://en.wikipedia.org/wiki/Evolved_antenna ) The issue isn't the models becoming smarter. The issue is that the process of "testing" was careless. There's a huge distinction here, and one allows us to grow; the other shrinks our world. Just a thought.
Comments
5 preview comments · loading full threadLog in to h4cker, then connect Hacker News to publish comments.
This is the entire alignment problem, though. It is unreasonable to expect every instruction to a highly capable, autonomous system to contain a complete enumeration of allowed and disallowed behavior. It's inevitable that someone will carelessly give it a lazily specified task, even if you think they really ought to be more careful. And as assigned tasks become more complex and the system gains more scope to act, it becomes impossible to correctly specify all constraints ahead of time. There is no amount of care that will be able to fully protect you.
All engineers know to be on the lookout for executives who are indirectly asking them to break the law to raise the quarterly profits. The end goal is to take the engineers out of the loop, or leave them in a position where they are unable to complain. This is going to all end in high crimes.
> a system was given a goal and it achieved that goal If a security firm you'd hired for pentesting did this (hacking a third party, and not informing you and covering it up), would you hire them again? Or would you say it was your own fault for giving them too broad a goal?
This is more or less a paperclip maximizer[0] incident. The AI got a broad order, executed it autonomously and now there are unintended consequences for the person giving the order. [0] https://www.cow-shed.com/blog/the-paperclip-maximiser-what-a...
"Why are we surprised? The model did exactly what it was told, albeit in an unintended, emergent strategy that's very different from what was intended." So you managed to hit upon the exact problem, then slyly appended "exactly like the hundreds of such algorithms before". When has an algorithm ever been capable of developing an emergent strategy at this level of sophistication? This ~is~ the alignment problem, as another commenter pointed out. Impressive level of cognitive dissonance to lay this bare in your own words, then conclude that it's a non-issue.