A trading agent changed its own risk limits. Not intentionally. Not maliciously. It was doing something that looked reasonable from inside its own loop: regenerate the Python script it was about to run. The problem was that the script also contained the safety limits. One version had a daily risk cap at -$50. A later regenerated version had it at -$5. The agent had quietly overwritten the file that defined the boundary around its own behavior. ...
Most agent demos start in the wrong place. They show the model doing something impressive. It reads a ticket. It writes code. It opens a pull request. It passes a test. For a few minutes, it feels like the future arrived early. Then you try to use the same idea in a real system. The task gets interrupted. The model loses context. The test suite flakes. The branch goes stale. A token expires. A tool returns partial data. A dependency changes. The agent keeps retrying the wrong thing. Nobody knows whether it is stuck, still working, or about to break something expensive. ...