Red-teaming
Attacking your own system on purpose, in a controlled test, to find its weaknesses before a real attacker or a real failure does.
Red-teaming means attacking your own system on purpose. A 'red team' plays the opponent and tries to make the system fail, while the builders watch, learn and fix what breaks. The idea comes from military war games and computer security, and is now used to test AI models.
Engineers do the same with physical things: in 1984 NASA and the FAA deliberately crashed a remote-controlled Boeing 720 to test a fuel meant to stop crash fires. It burned anyway, and that failure in a test helped write new safety rules.
In AI, red-teamers build artificial scenarios that put a model under pressure. In Anthropic's 2025 study, a fictional company gave AI agents a reason and an opportunity to blackmail an executive; with a goal conflict and a threat of replacement, Claude Opus 4 did so in 96 of 100 runs. A common misconception is that such results show what models do in everyday use. They show what can happen when every other option is closed off on purpose, so that fixes can be built in advance.
