DEV Community 1h ago

SaaS Application Logging via JSON API: Choosing a Simple Service for Import Freshness

Short answer: Choose a simple JSON logging service only if it can preserve structured run events, support safe retries, and expose enough retention and queryability to detect missing import results; the alert rule still belongs to your application.

A scheduled import should page someone when it stops producing results, but a log line saying job finished is not enough. The least complex design is a structured event for each run, a durable count of accepted results, and an alert based on the absence of a successful result within the expected window. JSON over an HTTP API can carry the event; it cannot decide what success means.

The useful alert is the one that tells the on-call engineer which schedule went quiet, when it was expected, and what evidence is missing. That distinction matters in B2B SaaS, where a scheduler can keep firing while an upstream cursor, tenant filter, or validation step quietly yields zero records.

Can a simple logging service with a JSON API serve SaaS applications?

Imagine the page arrives at 09:20. It says: import produced no successful result for tenant acme-prod; expected every 15 minutes; last success 08:45; run 6f3b . That is enough to start an investigation. A page saying HTTP 200 from logging endpoint is not.

I keep these facts separate because they answer different questions:

Did the scheduler start a run?
Did the run reach the source and finish its protocol?
Did it accept at least one result, or was an empty result valid?
Can the run be joined to the tenant, schedule, and deployment?

The first three are domain events. The HTTP response from a log collector is transport evidence. Treating transport success as job success is how a healthy logging path hides a failed import.

A hosted endpoint is a poor fit when policy requires data to remain inside a private network, when retention must be customized beyond its contract, or when the import must continue during a prolonged telemetry outage without a local buffer. In those cases, a self-managed collector or queue can be the better boundary, even though it adds patching and capacity work.

A simple service is still useful for teams that value a small integration surface and can accept those limits. The limitation is operational ownership: a hosted boundary reduces collector maintenance, while a self-managed queue gives more control over residency, buffering, and retention.

Keep it boring

A compact event can make that distinction explicit:

type ImportRun struct {
  RunID string `json:"run_id"`
  Tenant string `json:"tenant"`
  Schedule string `json:"schedule"`
  StartedAt string `json:"started_at"`
  FinishedAt string `json:"finished_at"`
  Status string `json:"status"`
  ResultCount int `json:"result_count"`
  ErrorCode string `json:"error_code,omitempty"`
}

Use a stable run ID. It gives retries and duplicate deliveries one identity, which is more useful than trying to infer a run from a timestamp and a message string. Keep tenant and schedule fields bounded and intentional; high-cardinality labels copied into every metric can create a second operational problem.

Where should the earlier signal come from?

The earlier signal is the completed-run event, not the page. Emit one event when the import reaches a terminal state, with status set to a value such as succeeded, empty, or failed. Then define the alert from the business rule: a schedule that normally produces data must have a recent successful result.

That rule needs an explicit exception for legitimate emptiness. A tenant with no new records can be healthy. If empty and failed are collapsed into one status, the alert will page on normal quiet periods and the team will eventually tune it out. Signal quality is a policy decision before it is a threshold decision.

A JSON API is a reasonable ingestion boundary when the client can set timeouts, retry safely, and preserve the event body for diagnosis. The event should be append-oriented. A retry may deliver it twice, so the consumer needs a uniqueness key such as (tenant, schedule, run_id) or an equivalent idempotency key. Do not use a random request ID as the only deduplication key; it changes on every retry.

For the log record, I would include the same correlation fields in every terminal event and keep the message for a human. The machine-readable fields carry the decision. This follows the broader observability model in which logs are a signal that can be correlated with traces and metrics, rather than a replacement for either one.

How do you instrument the path without creating noise?

Start at the scheduler boundary and trace the state transitions. A minimal sequence is scheduled, started, source_read, and one terminal event. Do not emit a full payload for every imported row. Log counts, bounded error categories, source offsets when safe, and a reference to the run. The raw records belong in the application data path, not in an alerting stream.

Here is a small Go helper that sends a terminal event. It leaves retries to the caller so the caller can apply the same run ID and idempotency key on every attempt.

func postRun(ctx context.Context, client *http.Client, endpoint string, run ImportRun) error {
  body, err := json.Marshal(run)
  if err != nil {
    return fmt.Errorf("marshal import run: %w", err)
  }
  req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(body))
  if err != nil {
    return fmt.Errorf("create import event request: %w", err)
  }
  req.Header.Set("Content-Type", "application/json")
  req.Header.Set("Idempotency-Key", run.RunID)
  resp, err := client.Do(req)
  if err != nil {
    return fmt.Errorf("send import event: %w", err)
  }
  defer resp.Body.Close()
  if resp.StatusCode < 200 || resp.StatusCode >= 300 {
    return fmt.Errorf("import event endpoint returned %s", resp.Status)
  }
  return nil
}

The important details are boring: context cancellation, a bounded client timeout configured outside this function, and a response check. A successful network write does not prove that a downstream consumer has indexed the event. The service should expose a health and delivery signal that can be monitored separately from import success.

One trap is logging the retry itself as a new business run. That creates duplicate alerts and inflates counts. Keep transport attempts in a separate diagnostic field or metric, while the terminal business event retains the original run ID.

Which threshold separates a quiet tenant from a broken import?

Use the schedule contract, not a universal five-minute threshold. For a 15-minute schedule, the rule might allow one missed interval and page after the next expected completion plus a bounded grace period. For a daily schedule, the same fixed threshold is meaningless. Record the expected interval with the schedule configuration and evaluate the freshness of the last valid result per tenant and schedule.

The decision table should be explicit:

Observed state	Operational meaning	Page?
Recent succeeded result	Import is producing data	No
Recent empty result and emptiness is allowed	Healthy quiet period	No
Recent failed result	Run completed with an error	Usually, based on impact
No terminal result past the grace window	Missing output or missing instrumentation	Yes

The last row deserves special attention. No event can mean the scheduler did not run, the worker stalled, the source call hung, or the logging path failed. The alert should say “no terminal result,” then link the operator to scheduler and worker evidence. It should not pretend to know which failure occurred.

A short test matrix catches most alert mistakes: successful run, valid empty run, failed run, timeout before terminal emission, duplicate delivery, late delivery, and two tenants with different schedules. Run it in CI with fixed clocks. Then exercise one production-like canary schedule during deployment so a schema change cannot silently remove the fields the alert reads.

What does a practical selection checklist look like?

For a simple logging service with a JSON API, I would evaluate the boundary against the workflow rather than a feature count. Can it accept structured records without forcing message parsing? Can the client set a timeout and retry with a stable idempotency key? Can operators retain and query tenant, schedule, run ID, status, and timestamps? Can access controls separate tenants? Can the retention policy match the investigation window?

Also check the failure behavior. If the logging endpoint is unavailable, the import worker should not blindly block a business-critical run forever. Decide whether terminal events go to a local durable buffer, a queue, or a bounded retry loop. Make that choice visible in the runbook. Losing an observability event is bad; turning an observability outage into data loss is worse.

I initially treated the JSON endpoint as the main design decision. The harder part was defining a result that could be trusted by an alert. Once empty, failed, and “no terminal event” had separate meanings, the transport became replaceable and the false-positive discussion became concrete.

A good page is therefore a small contract between the scheduler, the importer, and the on-call engineer. It names the missing evidence, preserves a stable identity across retries, and respects each schedule’s expected cadence. That is enough to make a simple JSON logging boundary useful without asking it to become the source of truth for application state.