Making wrong actions unrepresentable in a system that writes itself

Every deployment conversation I've had with a large enterprise reaches the same question, usually within the first hour: how do we know this won't do the wrong thing?

It's the right question. I would worry about a customer who didn't ask it. But we hold ourselves to something stricter: be the harshest critic of our own work. Before any customer asks, we ask ourselves: how do we know we won't do the wrong thing? If we can convince ourselves, convincing the customer is easy. If we can't, we have no business deploying.

The question is important because of where our agents operate. At FlowGen Labs we build agents that act inside SAP at Fortune 500 enterprises, the system that runs their money. Agents that post, pay, clear, and change master data.

A wrong answer in a chat window costs minutes. A wrong action in a financial system is a duplicate payment, or a close that doesn't reconcile three weeks later. These failures are quiet, and they are expensive to unwind.

So what is an honest answer to "how do we know"?

It can't be "we haven't seen it do the wrong thing yet." That is a claim about test coverage, not about the system. Testing individual behaviors gives you existential evidence: this case worked. Financial systems demand universal statements: no case of this class can go wrong. That is the oldest argument for formal methods: testing can show the presence of a fault, never its absence.

Our answer is verification. Not proving the whole stack correct in a theorem prover; that is still prohibitively expensive. We take the lightweight end of formal methods (types, invariants, executable properties) and concentrate it where the model touches the system. Design the system so that correctness is machine-checkable, then use those checks for everything: to make wrong actions unrepresentable, and to let the system write itself.

Making wrong actions unrepresentable

We prove things about the action space, the surface through which the model touches the system of record (e.g. an ERP like SAP). The model itself we treat as untrusted.

Every action an agent can take is a typed, constrained tool, not a free-form API call. A posting tool that structurally cannot touch accounts outside its scope. A payment action that cannot be expressed without the evidence that justifies it: the invoice, the match, the approval trail. Guards at the boundary check invariants the business itself defines (amount limits, segregation of duties, period status) before anything reaches the ERP.

Two more layers sit behind that. Human gates where the blast radius warrants one: certain actions exist only as proposals until an analyst approves them. And durable, replayable workflows underneath everything, so any action can be inspected, interrupted, and recovered.

The design goal is a strict relationship between two sets: what the agent can express, and what the system will accept. Anything wrong should fail at expression, not at the postmortem. Programming languages solved this locally a long time ago: make illegal states unrepresentable. We are applying it one level up, to actions.

There's a catch: you can't write invariants for a business you can't see. What "a correct posting" means is different at every enterprise. At one energy-infrastructure company we work with, the answer runs through decades of customization and tens of thousands of custom objects, plus process knowledge that lives only in the heads of the people who run it. So before agents act, we build the ground truth: a hierarchy of evidence-cited context, reverse-engineered from the SAP system itself (transactions, tables, custom code, the documents around it). We call it the Contextual Process Graph. Every guard checks against that graph, not against our assumptions. If a fact isn't evidenced, it isn't in the graph, and nothing is allowed to rely on it.

A system that writes itself

Here is the part that would have sounded reckless two years ago: the action space I just described is increasingly written by agents. So is the graph it checks against.

This is more than tuning: agents draft new tools, tighten guards, extend evals, and propose changes to their own constraints. This became tractable for one reason: verification turns out to be the best feedback an agent can get. A checkable specification (a type, an invariant, a property, a failing eval) converts "is this right?" from a human judgment call into a signal an agent can iterate against thousands of times without supervision.

The loop runs on production evidence. Traces, corrections, and near-misses from live use become findings. Findings become evals: executable statements of what should have happened. A bounded task environment (writable working copy, read-only production context, explicit validation gates) lets an agent propose a change to the tools or guards. Adversarial review follows: separate agents whose only job is to assume the change is wrong and try to break it, seeing only the diff, never the author's reasoning. Regression gates decide what ships.

Note what improves in this loop: the process and its guardrails, not a prompt. When production surprises us, the fix goes into the machinery that generates and admits changes. We retire the class of failure, not just the instance.

None of this is speculative. The past year produced public examples: a half-million-line codebase ported in eleven days by agent fleets held honest by adversarial review, and agents finding real bugs in mature open-source libraries through property-based testing: proposing properties that should hold and hunting for counterexamples. We are applying that playbook where a wrong action costs the most.

The same machinery, twice

You may have noticed a tension. If agents write the system, what keeps the guarantees trustworthy?

The same machinery, pointed both ways. The guarantees make self-writing safe: an agent changes the action space only through the guarded process (typed changes, evidence requirements, adversarial review, regression gates), the same discipline the action space imposes on individual actions. And self-writing keeps the guarantees honest: every surprise in production becomes an eval, and every eval hardens the boundary. Proof makes evolution safe; evolution keeps proof current.

One consequence we didn't anticipate: the boundary between our engineering process and our product is dissolving. Everything normally filed under developer experience, from CI and evals to adversarial review, is first-party to the platform now. The pipeline ships with the product. The way the platform is engineered is co-designed with the platform.

That is what we think enterprise agents demand: wrong actions unrepresentable at the boundary, and a system that writes itself under proof.

We are a small team working on this every day, against production SAP systems, where every action impacts the infrastructure that runs the world. If that sounds like your kind of problem, we're hiring. Join us.