Operations

HTTP endpoints

Metrics

forgemirror records metrics with OpenTelemetry and exposes them in the Prometheus format. Go runtime (go_*) and process (process_*) metrics are included as well.

A failed clone counts as a failure for every target, because none of them could be updated.

Example alerts:

groups:
  - name: forgemirror
    rules:
      - alert: ForgemirrorTargetFailing
        expr: increase(forge_mirror_operations_total{status="failure"}[30m]) > 0
        labels: { severity: warning }
        annotations:
          summary: "Mirroring to {{ $labels.target }} fails"
      - alert: ForgemirrorWebhookUnauthorized
        expr: increase(forge_mirror_webhook_requests_total{outcome="unauthorized"}[15m]) > 0
        labels: { severity: warning }
        annotations:
          summary: "Webhook signature mismatch; check webhook.secret"

Logs

forgemirror writes structured logs to stderr. They are JSON by default; set log.format: text for logfmt-style lines. Mirror events carry repository, ref, delivery, and per target also target, outcome and duration.

git output in logs and errors is redacted: tokens and authorization headers show up as [REDACTED].

Graceful shutdown

On SIGTERM or SIGINT:

1. /readyz returns 503 and the listener stops accepting connections.
2. Open webhook requests are answered.
3. Running mirrors finish. Queued follow-up runs and new pushes are dropped and logged; the next push after the restart mirrors the latest state.
4. When server.shutdown_timeout (60 s by default) runs out, git processes are killed and the process exits.

The phases share one budget. Keep the orchestrator’s kill timeout above it: terminationGracePeriodSeconds (Kubernetes), stop_grace_period (Compose), or TimeoutStopSec (systemd).

Troubleshooting