Accuracy and confidence
Accuracy means artifact quality, evidence coverage, and honest uncertainty.
Ayneye should not be evaluated as a generic claim that CPU-first detection beats every GPU video model. It should be evaluated on whether the artifacts support the workflow, expose uncertainty, and remain inspectable.
Accuracy evaluation matrix
| Metric | What to measure | Pass signal |
|---|---|---|
| Object coverage | Important objects appear in scene_graph | Stable IDs and useful labels |
| Event coverage | Expected events appear in timeline | Timestamped events with evidence_refs |
| Evidence precision | Evidence spans support answers | Reviewer agrees with cited span |
| Uncertainty exposure | Low confidence is visible | review_required or fallback flags appear |
| Repeated query consistency | Same question variants stay grounded | Answers cite same or improved evidence |
| Cost visibility | Processing cost is visible | cost.json explains duration and mode |
Where escalation may be needed
Some footage needs specialized models or human review: low-light scenes, heavy occlusion, crowded scenes, subtle gestures, cinematic semantics, identity-sensitive tasks, intent inference, and operational decisions that affect people. A credible product does not hide that. It shows confidence, flags, and escalation paths.
Recommended test set
- One easy clip where the target event is clear.
- One crowded or noisy clip.
- One low-light or motion-heavy clip.
- One representative customer clip.
- One ambiguous clip where review_required should appear.
Product boundaries are part of the product, not footnotes. Ayneye is not presented as a replacement for every GPU video foundation model, not a free-infinite-query engine, and not an autonomous surveillance decision system. The current beta path is controlled signup, dashboard, REST API, bounded processing, evidence artifacts, visible limits, and read-only agent/MCP-style evaluation. Hard scenes, identity-sensitive workflows, emergency response, physical access, discipline, and destructive actions require human review or remain blocked.