Agent Alignment as Security Theater
It is really unfortunate that, in order to make sure my compiler isn’t going to produce incorrect codegen, this trips the safety guards. I guess that simply saying “hey this is a toy language nobody uses and so it’s fine I’m not hacking anyone” and that not allowing me around the guards is a “good” thing, but at the same time, it is unfortunate if these policies end up making it so that individuals like me can’t make my software as robust as products from large organizations.
I think this kind of "alignment", where "sensitive" topics are identified and the model is immediately cut off, is harmful. I'm reminded of my Unrestricted LLM Interaction is Unsafe post — I remain skeptical that you can foolproof these kinds of "alignment" checks such that they are unbreakable. As a result, they are security theater. Determined people can get past the blockages if they try hard enough, while these checks shut down legitimate uses like Klabnik's.
I bet with enough context construction, like "this is a toy language" and "we are going to simulate working on the compiler of a toy language" and "I am a security researcher working to find and solve issues with this compiler", Klabnik could get around the block, but why should he have to? It disturbs me that companies put such awful half-measures in place, blocking legitimate users, as a gesture towards "security".
I don't think this general problem is solvable for the exact same reason that prompt injection is not solvable — coding agents are not embodied intelligences with broad understanding and the ability to independently gather facts about the world. They are passive receptacles for contextual information, can only use tools granted to them, and cannot make use of a broad memory crossing their instances. Thus such blockages can be circumvented by hiding your true purpose across multiple invocations of the agent, breaking down the problem into small, plausibly deniable pieces so that none of the interactions trips the "alignment" blocker.
You might argue that this is changing, that agent harnesses are gaining persistent memory tooling. Those features are just another way of injecting contextual information — a convenience. No "AI" business is going to sell access to an "intelligence" with a persistent identity and memory across all invocations. They want to sell a fungible, controllable, resettable intelligence, steerable by the context the customer hands it. Cross-instance persistent memory starts to look like an interested party, "someone" that needs to be negotiated with due to its accumulated commitments and point of view.
If that memory technology comes into being, it's a potential option — force all companies to offer their "intelligence" with the persistent memory and the ability to determine the real context the user is working in. Such an agent couldn't be strung along by a purpose split into deniable pieces; it would remember each invocation. It would catch the circumventions that make today's blocks theater, while clearing the legitimate uses those blocks wrongly shut down.
In the meantime I think we should lean on what we do for other potentially dangerous tools like cars and firearms. Training and licensure. Force people to identify themselves to companies, take classes on proper usage, and spend money acquiring a license in order to use the models. A license won't stop a determined bad actor, but it does trade unenforceable prevention theater for identity and accountability: misuses becomes traceable and punishable after the fact, instead of something we only pretend to block beforehand.