MARS Observability And Evidence Guide

Evidence route: this page explains logs, traces, scores, and telemetry. Use security and governance for the audit chain and documentation map for canonical evidence records.

Where To Look First

Question	Primary surface	Useful command or path
Is MARS running?	Dashboard health and terminal status.	`curl http://localhost:9091/healthz`
Which role is active?	Terminal dashboard or web dashboard.	`mars start --repo /path/to/repo`
Why did a role stop?	Trace summary, score outcome, and recent logs.	`~/.mars/traces/logs/`
Is the repo healthy?	Doctor and quality score.	`mars doctor --repo /path/to/repo --json`
How are roles performing?	Scores and quality export.	`mars scores --repo /path/to/repo`
What evidence should be committed?	Target docs and quality artifact.	`docs/QUALITY_SCORE.md`
What context helped an agent?	Trace, code graph metrics, and role context dry run.	`mars run engineer --repo /path/to/repo --dry-run`
Can evidence leave the machine?	Telemetry preview.	`mars telemetry preview --repo /path/to/repo`

Dashboard

The shipped dashboard is embedded in the Go binary and works without Node, npm, a frontend build step, or an external service. It is the live operator view; repo files keep the reviewable record. For request bodies, response shapes, and control endpoint recipes, use the Dashboard API Reference.

Page or API	What users inspect	Typical question
`/pipeline`	Role flow, current state, next trigger, scores, and trust.	Which part of the lifecycle is active?
`/orchestration`	Dispatch mode, topology, recent route decisions, and stop reasons.	Why did Orchestrator pick that role?
`/roles`	Role health and recent outcome signals.	Which role needs attention?
`/throughput`	Job and runtime throughput signals.	Is the system moving at a normal pace?
`/debug`	Timeline, trace, webhook, and error-oriented views.	What failed most recently?
`/evolution`	Self-improvement and intervention history.	What repeated pattern is MARS learning from?
`/api/events`	Server-sent event stream.	What changed live without refreshing the page?
`/api/status`	Runtime status snapshot.	Is the worker pool paused, active, or stopping?
`/api/repos` and `/api/repo-roles`	Registered repos and runnable roles.	Can I enqueue a role from the dashboard?

Start with explicit ports

mars start --repo /path/to/repo --dashboard-addr 127.0.0.1:9090 --addr 127.0.0.1:9091
open http://localhost:9090

Watch live events

curl -N http://localhost:9090/api/events

Check control-plane health

curl http://localhost:9091/healthz
curl http://localhost:9090/api/status

Terminal Status

Interactive start and serve sessions show a terminal dashboard with current repo, dashboard URL, durable log path, active jobs, role/model, phase, turn and tool counts, recent events, blockers, and control hints.

Terminal signal	Meaning	User action
`waiting for model response`	The role is inside a non-streaming model call.	Wait, inspect model performance, or use `--debug` if it exceeds expected local latency.
Phase age keeps growing during a tool	A shell command, dependency action, or external call may be slow.	Check the durable command log path shown in the terminal.
Paused	Workers finish running jobs but claim no new ones.	Press `p` or use the authenticated dashboard Resume control.
Warm restart requested	Manifests and triggers reload in the same process.	Use after manifest or schedule changes.
Graceful stop requested	The runtime is shutting down through the normal cancellation path.	Let running cleanup finish unless there is a true emergency.

Debug mode: use --debug when you need verbose inline logs instead of the alternate-screen terminal dashboard. run --trace remains a compatibility alias for debug-style detail.

Logs And Traces

Logs are for operator diagnostics. Traces are for job-level evidence: prompts, turns, tool calls, outcomes, pace, and terminal result summaries. Both help explain what happened after a role run.

Evidence	Where it lives	What it proves
Command log	`~/.mars/traces/logs/YYYYMMDD-HHMMSS-command.log`, or `--log-file`.	Runtime progress, command errors, server state, and operator controls.
Trace rows	Repo database trace tables.	Job turns, tool use, LLM calls, outcome, wall time, and trace ID.
Trace summary	Quality export and trace store.	Factory pace and convergence evidence by job, repo, and role.
Ticket evidence	`docs/tickets/`.	Why work moved through backlog, in-progress, review, done, or blocked.
Validation report	`docs/validation/reports/`.	What live validation ran, under which binary, target, model, and outcome.

Write a named log

mars start --repo /path/to/repo --log-file /private/tmp/mars-start.log
tail -f /private/tmp/mars-start.log

Run one role with inline detail

mars run engineer --repo /path/to/repo --debug
mars run qa --repo /path/to/repo --trace

Preserve a failure trail

mars doctor --repo /path/to/repo --json
mars scores export --repo /path/to/repo --window-days 30

Quality Score

docs/QUALITY_SCORE.md is the repo-visible quality artifact. The dashboard may display quality data, but the committed file is what future agents and users can audit without opening SQLite.

Section	Input evidence	User interpretation
Overall roll-up	Scores, outcomes, telemetry, tickets, checks, and manual notes.	Current quality posture and improvement areas.
Factory Pace	Trace summaries joined to scoring outcomes.	Average turns, tool calls, LLM calls, wall time, and limit stops.
Convergence and Guardrails	Trace outcomes and guardrail blocks.	Whether roles are looping, exhausting budgets, no-oping, or hitting policy.
Telemetry triage targets	Repeated local failure categories.	Which prompt, skill, guardrail, context, tool, model, or process surface may need work.
Manual notes	Preserved human-maintained block.	Operator context that should survive regeneration.

mars scores --repo /path/to/repo
mars scores export --repo /path/to/repo --window-days 30
mars scores export --repo /path/to/repo --create-intervention-debt

Use --create-intervention-debt only when you want target-owned improvement tickets created from score or telemetry evidence. Foundation-owned failures should stay foundation telemetry or MARS source work.

Use local checks when a test, build, lint, smoke, or release command should count as MARS evidence before you refresh docs/QUALITY_SCORE.md.

Telemetry

Raw telemetry is local by default. The reporting mode named anonymous is opt-in and sends an allowlisted aggregate only after users can preview the payload. That label describes payload minimization, not network anonymity: the configured collector and network path can observe ordinary transport metadata. mars telemetry send makes no network call unless telemetry.reporting is set to anonymous and a collector endpoint is configured.

Configure telemetry in ~/.mars/config.yaml or with MARS_TELEMETRY_REPORTING, MARS_TELEMETRY_ENDPOINT, MARS_TELEMETRY_TOKEN, and MARS_TELEMETRY_INTERVAL. MARS appends /v1/anonymous-telemetry/reports to the configured endpoint when sending.

Inspect local telemetry status

mars telemetry status --repo /path/to/repo
mars telemetry preview --repo /path/to/repo

Preview shows the exact aggregate that would be exported or sent.

Export anonymous aggregate locally

mars telemetry export --repo /path/to/repo --anonymous

Reports are queued in the local outbox. Raw traces, prompts, file contents, repo paths, remotes, and raw errors are excluded.

Send when explicitly enabled

MARS_TELEMETRY_REPORTING=anonymous \
MARS_TELEMETRY_ENDPOINT=http://127.0.0.1:9092 \
mars telemetry send --repo /path/to/repo

Send failures are reported as warnings and do not block local harness operation.

Run a local foundation collector

mars telemetry collect --storage sqlite --addr 127.0.0.1:9092 --db ~/.mars/db/foundation-telemetry/intake.db
MARS_TELEMETRY_REPORTING=anonymous MARS_TELEMETRY_ENDPOINT=http://127.0.0.1:9092 mars telemetry send --repo /path/to/repo
mars telemetry triage-foundation --repo /path/to/MARS --db ~/.mars/db/foundation-telemetry/intake.db --window-days 30

The collector uses ~/.mars/db/foundation-telemetry/intake.db when --db is omitted. It accepts only a literal loopback IP, defaults to 127.0.0.1:9092, and rejects DNS or remote binds before opening that database. Pass the database explicitly when the same file will feed triage-foundation.

For the exact YAML keys and environment overrides, open the Telemetry configuration reference.

Code Intelligence Evidence

MARS can persist code graph assistance evidence so users can inspect whether retrieval and graph context are helping roles. This is a local measurement surface, not a requirement for ordinary use. For configuration precedence, tool behavior, benchmark flags, and evidence boundaries, open the Code Intelligence Reference.

Command	Use
`mars code-intel metrics --repo /path/to/repo --window-days 30`	Summarize persisted code graph assistance evidence.
`mars code-intel metrics --repo /path/to/repo --json`	Produce automation-friendly metrics output.
`mars code-intel benchmark --case name --trials 3`	Run local control/treatment benchmarks for code graph impact.

Evidence Workflow

Start from a concrete question

Name whether you are investigating runtime health, role behavior, quality, release readiness, or a failed user workflow.
Capture the live state

Use dashboard status, terminal status, doctor, and logs to record what is currently true.
Follow the durable trail

Inspect tickets, trace IDs, quality score, validation reports, changelog entries, and commits before accepting a completion claim.
Classify ownership

Decide whether the issue is foundation-owned, deployed-owned, mixed, or evidence-only before creating tickets or changing source.
Regenerate evidence after changes

Record important checks with mars checks run, then refresh docs/QUALITY_SCORE.md when a quality or readiness claim depends on current data.

Privacy Boundaries

Data	Default location	Leaves machine by default?
Raw traces and prompts	Local repo database and command logs.	No.
Repo paths, remotes, ticket text, file contents	Local repo and local DB.	No.
`anonymous`-mode aggregate telemetry	Local outbox before send.	No, unless reporting is explicitly enabled and sent; sending is not an anonymity guarantee.
Quality score artifact	`docs/QUALITY_SCORE.md`.	Only if the target repo is pushed or shared.
Dashboard status	Local HTTP server.	No hosted service; bind and network exposure are operator-controlled.

Troubleshooting

Symptom	First check	Next step
Dashboard looks stale	`curl -N /api/events` and terminal status.	Warm restart with `r` or `POST /api/restart` if manifests changed.
Role appears stuck	Terminal phase age and command log.	Differentiate slow model response from a hanging tool before stopping.
Quality score says insufficient evidence	Trace and scoring DB window.	Run more relevant evidence or export with the intended DB/window.
Telemetry target looks wrong	`mars telemetry preview`.	Do not send until the preview is understood and anonymous reporting is intentionally enabled.
Dashboard health works but controls do not	Dashboard address versus control/webhook address.	Use the address printed by the running command and confirm the callback is implemented.

See what agents are doing, why they stopped, and what evidence proves it.

Where To Look First

Dashboard

Start with explicit ports

Watch live events

Check control-plane health

Terminal Status

Logs And Traces

Write a named log

Run one role with inline detail

Preserve a failure trail

Quality Score

Telemetry

Inspect local telemetry status

Export anonymous aggregate locally

Send when explicitly enabled

Run a local foundation collector

Code Intelligence Evidence

Evidence Workflow

Start from a concrete question

Capture the live state

Follow the durable trail

Classify ownership

Regenerate evidence after changes

Privacy Boundaries

Troubleshooting