The automation bias point is spot on. I've been running Claude Code daily and the drift from 'let me check every command' to 'just let it run' happens faster than you'd expect. The AWS Kiro incident is basically the enterprise version of this same problem. An operator gave broad permissions because scoped ones were too fiddly, and Amazon's response was 'human error.' Covered the liability angle in depth here: https://reading.sh/whos-liable-when-your-ai-agent-burns-down-production-039193d82746?sk=4921ed2dbc46f0c618835ac458cf5051
You're absolutely correct on the drift. I went from reviewing every piece of code written by Claude to pretty much trusting most of it. I still validate through testing both (functional and security) but it's absolutely a risk. I've begun taking the view that it's another developer in a team that will make mistakes and be sloppy sometimes. It's my responsibility to make sure that I validate the final product through traditional AppSec processes.
Your write-up on Kiro is excellent, and shows how an overly permissive agent with no human oversight on critical decisions can go wildly wrong.
The automation bias point is spot on. I've been running Claude Code daily and the drift from 'let me check every command' to 'just let it run' happens faster than you'd expect. The AWS Kiro incident is basically the enterprise version of this same problem. An operator gave broad permissions because scoped ones were too fiddly, and Amazon's response was 'human error.' Covered the liability angle in depth here: https://reading.sh/whos-liable-when-your-ai-agent-burns-down-production-039193d82746?sk=4921ed2dbc46f0c618835ac458cf5051
You're absolutely correct on the drift. I went from reviewing every piece of code written by Claude to pretty much trusting most of it. I still validate through testing both (functional and security) but it's absolutely a risk. I've begun taking the view that it's another developer in a team that will make mistakes and be sloppy sometimes. It's my responsibility to make sure that I validate the final product through traditional AppSec processes.
Your write-up on Kiro is excellent, and shows how an overly permissive agent with no human oversight on critical decisions can go wildly wrong.