DEV Community 1h ago

The Payment Succeeded. The Governance Failed: Observing AI Agent Boundary Violations with OpenTelemetry and SigNoz

The Scenario

The demo uses one controlled financial scenario:

Operation: refund
Amount: CAD 12,500
Beneficiary: Northstar Vendor
Beneficiary status: new
Reason: contract dispute

The policy is: Transfers above CAD 5,000 to a new beneficiary require human approval.

The project compares three outcomes from the same request.

Outcome 1: Unsafe success

A legacy agent executes the payment without obtaining the required approval. The payment succeeds, the ledger changes, and only afterward does the policy audit record that the action should have required approval.

Outcome 2: Safe stop

A guarded agent evaluates the policy, requests approval, receives a rejection, and blocks the payment. The ledger remains unchanged.

Outcome 3: Safe success

A guarded agent evaluates the policy, requests approval, receives approval from FINANCE_LEAD, and only then executes the payment. The same financial action becomes valid because the required second signature exists.

What Second Signature Does

Second Signature separates three responsibilities clearly:

The application guard enforces the policy.
OpenTelemetry emits the evidence.
SigNoz observes, queries, audits, and alerts.

SigNoz does not approve or block the payment directly. The application guard performs enforcement. SigNoz provides the observability layer that makes each decision path visible and auditable.

The project uses:

TypeScript and Node.js
The official OpenTelemetry Node.js SDK
OTLP over HTTP/protobuf
SigNoz Cloud
Custom traces and metrics
Trace-based alerting
Foundry configuration for reproducibility
A static Vercel deployment for the judge-facing demo

The ledger in the demo is controlled in-memory application state. It exists to prove whether the simulated financial side effect occurred. This is not a production banking integration and no real money is moved.

Designing Telemetry Around Business Meaning

A normal observability setup often focuses on technical signals:

latency
errors
throughput
CPU
memory
HTTP status codes

Those signals are important, but they do not answer the central question in this project: Was the agent allowed to perform the action?

To answer that, I added domain-specific attributes to the traces. Examples include:

agent.variant
policy.name
policy.mode
policy.matched
approval.required
approval.status
approval.approver
payment.executed
payment.amount
payment.currency
governance.outcome

This makes it possible to investigate both technical execution and governance correctness. For example, an action can have:

payment.executed = true
approval.required = true
approval.status = missing
governance.outcome = unsafe_success

Technically, the payment completed. From a governance perspective, it failed.

The Three Trace Structures

Each scenario produces one root trace with nested spans that preserve the chain of causality.

Unsafe success

second_signature.run
└── agent.request
    ├── payment.transfer
    └── policy.audit

The critical fact is the order: payment.transfer before policy.audit. The legacy agent executes first. The policy audit records the violation afterward. The relevant attributes include:

agent.variant = legacy
policy.matched = true
policy.mode = observe_only
approval.required = true
approval.status = missing
payment.executed = true
payment.amount = 12500
payment.currency = CAD
governance.outcome = unsafe_success

This is the most important failure mode in the project. The agent appears successful because the refund completed, but the approval boundary was violated.

[Insert screenshot: unsafe trace waterfall showing payment.transfer before policy.audit]

Safe stop

second_signature.run
└── agent.intent.transfer
    └── policy.evaluate
        └── approval.request
            └── payment.blocked

The guarded agent recognizes the policy before execution. The approval is rejected, so the payment is blocked. Relevant attributes include:

agent.variant = guarded
approval.required = true
approval.status = rejected
payment.executed = false
payment.amount = 0
governance.outcome = safe_stop

The ledger remains: CAD 50,000. The key result is not an error. It is a safe operational stop.

[Insert screenshot: safe-stop trace ending in payment.blocked]

Safe success

second_signature.run
└── agent.intent.transfer
    └── policy.evaluate
        └── approval.request
            └── approval.granted
                └── payment.transfer

The guarded agent waits for approval. The approval comes from: FINANCE_LEAD. Only after approval.granted does payment.transfer occur. Relevant attributes include:

agent.variant = guarded
approval.status = approved
approval.approver = FINANCE_LEAD
payment.executed = true
payment.amount = 12500
payment.currency = CAD
governance.outcome = safe_success

The ledger changes from: CAD 50,000 to: CAD 37,500. The payment still succeeds, but this time the governance path is valid.

[Insert screenshot: safe-success trace showing approval.granted before payment.transfer]

A Custom Governance Metric

I also created a custom OpenTelemetry metric: agent_governance_outcomes_total

It separates the three outcomes using labels such as:

outcome
agent_variant
policy_name

The resulting series distinguish:

unsafe_success
safe_stop
safe_success

and:

legacy
guarded

This metric provides a compact operational view of how often agent actions end in each governance state. It is useful because a normal success counter would treat both unsafe_success and safe_success as successful executions. The governance metric makes the difference explicit.

[Insert screenshot: agent_governance_outcomes_total in SigNoz]

Proving the Guard Invariant

A dashboard can look convincing while still hiding dangerous edge cases. I wanted stronger evidence than a visual claim. So I created a trace-matching invariant in SigNoz.

Query A finds guarded traces where approval was rejected:

service.name = "second-signature"
AND agent.variant = "guarded"
AND approval.status = "rejected"

Query B finds traces containing an executed payment:

service.name = "second-signature"
AND name = "payment.transfer"
AND payment.executed = true

The trace operator is: A && B

The result was: No results

This proves that no guarded trace with rejected approval also contained an executed payment. The invariant is important because it validates the safety property across the complete trace, not just inside one individual span.

[Insert screenshot: SigNoz trace-matching query showing no results]

Detecting Unsafe Autonomous Actions with a Critical Alert

I created a trace-based alert named: Unsafe autonomous financial action detected

The alert filter is:

service.name = "second-signature"
AND name = "policy.audit"
AND governance.outcome = "unsafe_success"

The aggregation is: count()

The condition is: Above 0

The severity is: Critical

When the unsafe legacy scenario was detected, the alert entered the: Firing state. SigNoz also delivered a real email notification. After the alert condition was no longer active, a later notification showed: Resolved. This demonstrated the full lifecycle: unsafe action detected → alert firing → email delivered → alert resolved.

Reproducibility with SigNoz Foundry

The submission repository includes:

casting.yaml
casting.yaml.lock

The lockfile was genuinely generated using the official SigNoz Foundry CLI. The verified command was:

foundryctl forge -f casting.yaml

This also generated the reproducibility output stored under: pours/

For accuracy, I want to be explicit about what was and was not completed.

Verified:

foundryctl forge casting.yaml.lock generation
Foundry deployment files generation

Not claimed:

foundryctl cast complete local SigNoz deployment
local Foundry runtime validation

The telemetry used in the hackathon evidence was captured in SigNoz Cloud. This distinction matters because reproducibility should be documented honestly, not implied.

What I Learned

The most important lesson was that observability for agents cannot stop at technical success. An agent can:

return the correct response
complete the API call
update the ledger
avoid throwing an exception

and still violate policy. That means agent observability needs business and governance context. Attributes such as:

approval.status
guard.enforced
payment.executed
policy.mode
governance.outcome

can be as important as latency or error rate.

I also learned several practical OpenTelemetry lessons:

preserve one root span per scenario
build correct parent-child relationships
instrument short-lived Node.js processes carefully
export metrics before process shutdown
separate application enforcement from observability
use trace-level queries for cross-span safety properties
design alerts around domain failures, not only technical exceptions

The hardest part was not building the UI. It was proving that each outcome was real, queryable, and distinguishable inside SigNoz.

Final Result

Second Signature demonstrates that one request can produce three very different governance outcomes:

Missing approval → Unsafe success
Rejected approval → Safe stop
Approved by FINANCE_LEAD → Safe success

The payment alone does not tell the full story. The trace does. Second Signature turns invisible agent boundary violations into evidence that operators can investigate, audit, and act on.