← CivicAttest

CivicAttest Detection Accuracy Report: 5.1 Million Records Under Test

WilWell Technologies · Independent ALPR Usage Audit Tool Original 40-log program on engine r3 (2026-08-27) · large-file blind campaign and defect remediation, builds r4.2 through r5.5 (September 2026)

The published test program now spans 5.1 million synthetic records across three distinct measurements, reported separately rather than blended into one number, because blending them once is exactly what let an inflated figure sit on this page until we caught it ourselves. They are: the original forty-log development program described below; a twenty-log, 100,000-record calibration suite re-run on every engine change as a regression gate, which most recently scored 100 of 140 planted patterns after we corrected an inflated figure of our own (below); and an eight-file, 4.8-million-record blind campaign run through September 2026, each file delivered alone, its run locked to disk, and its answer key opened only afterward. Across those eight files, 114 planted behaviors were scored and 65 were detected, twenty-six legitimate documented police incidents were presented as traps, and not one has ever been escalated. Every miss is disclosed below, and so is every one of the six defects we found in our own engine during this campaign, each found and fixed the same day it was found. The method behind these numbers is published separately as an open standard, how to test whether an ALPR audit tool actually works, which anyone is welcome to apply to us.

Here is how the original program was structured.

CivicAttest analyzes an agency's own exported license plate reader search logs and raises statistical patterns for human review. Its detection engine was first measured against forty synthetic Flock-format audit logs of 5,000 records each, roughly 200,000 records in all, authored by an independent model (Perplexity) that never saw the tool's thresholds. Each log contained seven planted misuse patterns and deliberately planted "trap" scenarios: legitimate multi-officer incidents engineered to look statistically suspicious.

The program ran in three phases, each disclosed rather than blended:

Phase 1 covered development and confirmation across twenty logs. Early exploratory tests exposed and fixed four detection defects. Ten development logs were then run blind and scored; one rule improvement followed. Six confirmation logs ran on the frozen engine: 32 of 42 patterns caught at row level (76%), 89% excluding one pattern whose generator contradicted its own answer key in every dataset examined.

Phase 2 was an adversarial red team across ten logs. The data author was asked to design evasion variants: gradual ramps under statistical gates, microbursts under the burst floor, low-volume external concentration, cross-agency paired searching, and vocabulary-masked off-hours activity. Three of seven evasion classes were still caught 10/10 (repeat-vehicle, masked off-hours, thin documentation at subject level). Four evaded as designed, and each became a named, specified fix.

Phase 3 was a standard persona-consistent series across ten logs. Fresh overt-pattern logs with a hard persona-consistency requirement, run on the frozen engine with results locked to disk before any answer key was opened.

These are the Phase 3 headline results, with the engine frozen across ten logs.

59 of 70 patterns caught at pattern level (84%); 1,101 of 1,692 planted rows identified (65%). Every one of the eleven misses is attributable to one of two specified causes with a validated fix (below).

No legitimate documented response has ever been escalated.

This section states the trap record for the original forty-log program. The one class of false positive we have found in the engine is disclosed separately in the September 2026 campaign section below, and it did not involve any trap row.

Across all forty logs, containing roughly 2,400 deliberately planted trap rows representing legitimate stolen-vehicle and missing-person responses:

The accusatory rules, meaning burst, off-hours, prohibited-purpose, and repeated-vehicle detection, touched zero trap rows across all forty tests.

Twenty-one trap rows across four logs were listed inside medium-severity volume-question flags, in every case because the officer genuinely worked the planted incident and their search rate genuinely rose during it, and the flag's own "innocent explanation" field names exactly that cause. On fully clean random data, the engine produces zero findings about any individual.

The r4 release was validated before shipping.

Every miss in the program maps to one of six specified fixes. One is already implemented and was validated live during Phase 3:

Network-scope rule (Pattern 7), now shipped as build r4.0. Built in a parallel branch after test 3 of Phase 3, then required to predict Pattern 7's account on each remaining log before the answer key was opened. Ten for ten, including seven live pre-registered predictions, with zero false fires on clean data or any prior log. Following the series close it was promoted into both engines with unit tests, and two-engine parity was re-verified across all ten Phase 3 logs plus clean-data controls. With r4.0, Phase 3 pattern detection is 67 of 70 (96%).

Outside-account concentration rule and full-group case-rate evaluation, shipped as build r4.1. The concentration rule's thresholds came from measured separation, not tuning: organic outside accounts in the test data queried hundreds of distinct plates with a 0% repeat share, while every planted concentration sat at 80 to 100 percent repeat share on 4 to 6 vehicles, which is a margin any reasonable threshold clears. Retrospectively, r4.1 catches all 70 of 70 Phase 3 patterns (100%*) with no new trap contact anywhere in the program's forty logs and two-engine parity re-verified across 28 datasets.

* The retrospective 100% is marked as such because r4.1 shipped after Phase 3 answer keys were opened. A fresh frozen-engine confirmation batch is the standing protocol before this figure is quoted without the asterisk.

Remaining roadmap (r5), from the adversarial phase: microburst aggregation across dates, gradual-trend detection, and cross-agency pair detection.

The September 2026 campaign ran eight large-file blind tests.

Each file is an independently generated synthetic export covering seventy to ninety accounts across a full year, which is the shape of a real mid-sized department on a busy regional network rather than the small-agency profile of the original forty logs. For every file: the log was delivered on its own, the engine ran and its results were locked to disk, and only then was the sealed answer key opened and the run scored. Nothing was ever tuned to a file after seeing what it contained, and no run was ever repeated after its key was open and reported as if it were still blind. Where a defect required a second run to verify a fix, that run is labeled a re-test in our engineering log, never folded into the blind numbers below.

Read the Detected and Traps escalated columns together, not apart. The 57% detection figure below comes entirely from these eight adversarial files, built specifically to include disguised and evasive misuse patterns, not routine ones, and a single hard file, BLIND08, accounts for 18 of the 49 misses on its own because the engine refused to guess at a column it could not verify rather than risk a wrong answer. The number that states whether the tool falsely accuses real police work is the last column: zero of twenty-six legitimate, documented incidents were ever escalated, across every file in the campaign.

Every flag ever attributed to a planted behavior across all eight files contained only planted rows, with one narrow exception on BLIND04 and one on BLIND07 where a rule describing an account's whole elevated period cited some of that account's ordinary rows alongside the planted ones, which we disclose below rather than round away. Completeness held on every file that carried deliberate gaps: our scan found the exact gap count and the exact missing-record count the sealed key specified, every time, on five sealed keys running.

The three early files behaved identically to each other: five of eight behaviors detected, no trap ever touched. The missed classes were a gradual late-period volume rise soft enough to sit under the statistical gates, and two timing-dependent patterns, recurring small-hours activity and a solo overnight burst, both suppressed by the same mechanism described below. A deterministic engine missing the same things the same way on three independent files is an engine behaving as designed, with a known and then-unfixed boundary.

BLIND03 was built harder and found two real defects in our engine.

BLIND03 carried eighteen planted behaviors across six families at three graded strengths rather than eight at one strength, and a different export schema than the engine had seen before. It detected nine of eighteen: every behavior in three families complete, none in the other three, the same timing and volume boundary as the three early files, now confirmed across nine behaviors instead of three.

A false positive. The rule that looks for prohibited search purposes raised a department-wide item covering two hundred searches whose stated reason was "civil standby," an ordinary lawful service call, because the rule matched the bare word "civil," which also appears in the phrase naming a prohibited immigration purpose. No officer was named and no trap was touched, but it is a flag on legitimate police work and we count it as one. Re-running the twenty-log calibration suite after the repair showed the identical defect had been firing once in every one of those twenty logs and had never been counted, because our own scorer had only ever asked whether planted behaviors were found and whether traps were escaped, never whether the engine had raised anything it could not attribute. That is twenty-one false positives of one kind, found the same week they were introduced, all fixed.

A silent degradation. BLIND03's network column held the names of the networks reached rather than a count of them. The column mapped by name, so the rule reading it was allowed to run, read nothing usable, and reported that it had found nothing, indistinguishable in the delivered report from a genuine clean result. Every rule now verifies that each column it consumes holds at least one readable value before it is allowed to run, and a column that fails is disclosed as not measured rather than counted as clean. This is precisely the failure test eight of our own published standard exists to catch, and it caught it in our own engine first.

BLIND03's gaps were also the first exact match against a sealed completeness key: eight gaps, every missing record accounted for.

The corroboration test was rewritten, and BLIND07 measured the result.

Off-hours activity and isolated search bursts had produced nothing on any large file. The cause was structural, not a threshold: the test asking whether an officer's activity was corroborated by colleagues asked only whether other officers were active nearby, and in a department running around nineteen hundred searches a day that is true at every hour of every day, so those rules could never fire. One file made the cost concrete: a single account ran 110 plate searches in sixteen minutes with no case numbers at all, and the engine passed over it because twenty-three other accounts were awake somewhere in the surrounding hour.

The rewritten test asks whether an account dominates its window rather than merely shares it, and for off-hours the presence question is dropped entirely, because a department that works around the clock has no quiet hours, only accounts do. An account must also stand apart from its own peers, and only undocumented activity drives a finding. The first version of this failed our own regression gate by flagging two thirds of one department and was discarded rather than shipped. A second defect surfaced after shipping: the burst rule was marking a window's rows as reviewed before deciding whether that window was actually suspicious, so a burst that opened gradually could be waved through on its thin opening minutes and never re-examined once it grew dense. Fixed the same day it was found, rows are now marked reviewed only once a burst is accepted as a genuine finding.

BLIND07, the first file scored under the rewritten engine, was run on the full protocol: the log alone, results locked before the key was opened, and the key itself stating the log's SHA-256 so the pair could be verified as matching. It detected 17 of 18 behaviors across all six families, escalated none of its four traps, and matched the completeness key exactly , the best result in the program. The single miss was three searches on each of eight nights, 24 rows in 700,000. Off-hours and burst, silent on every earlier file, produced nine of the seventeen detections. Four of its 39 flags could not be attributed to a planted behavior; each is a review-severity lead with its own stated innocent explanation, and no trap was among them.

BLIND08 defeated our column detector, and we are publishing the result at zero.

BLIND08 used a fifth export schema, engineered to defeat name-based column matching. Five of ten rules could not run and the engine disclosed that honestly. A sixth column was mapped wrong with full confidence: a field literally named "query mode," holding only the phrase "Exact Plate" on every one of 700,000 rows, matched our alias list for license plate and was accepted. The real plate column sat unused beside it. The locked, blind score on this file is 0 of 18 planted behaviors detected, 0 of 5 traps escalated, confirmed against the sealed key by matching flagged rows one at a time, not by account name, after an earlier internal check of ours used the weaker test and had to be corrected before it ever reached this page.

We measured what that wrong mapping would have cost on real data rather than guessing. With the department's actual case-number column restored and its documentation rate held at this file's own 86%, the bad plate mapping produced no false findings. At 50% documentation it named two officers on 158 rows they had nothing to do with. At 30% documentation, a rate some departments genuinely run, it would have named all ninety officers across 699,917 rows. A column name is a claim; we now test the values behind it before trusting it. Every candidate column is checked against the shape its kind of data actually has, since plates are high-cardinality and mostly alphanumeric, case numbers vary and contain digits, a timestamp column parses as a timestamp, and a column that fails is refused and disclosed by name, with the next-best candidate tried in its place. Reports now carry a "columns refused on content" section stating exactly what was rejected and why.

Our first attempt at that safeguard also had to look at what value shapes actually looked like for a required field with no name match at all, and the first version of that rescue nearly mapped a plain row-counter column in as the search timestamp, because a bare small number like "3" silently parses as a valid date under a very old, very lenient JavaScript behavior. Caught before it ran against any real file, by the same discipline: verify the fix against every file this program has ever held before trusting it.

BLIND09 found the sixth defect, and it was in our own scoring, not only our engine.

BLIND09 was requested at a lower, more realistic case-documentation rate, about 60% rather than the 80 to 90% of earlier files. Under the corrected column detector it parsed cleanly and scored 13 of 18 behaviors detected, 0 of 5 traps escalated, completeness exact, the first large file where a volume-shift rule fired at all. But 18 of its 21 unattributed flags were one rule, chronic missing case numbers, naming one in five officers in the department for a documentation rate that was, for this department, ordinary. The rule used one fixed rate with no comparison to the department's own norm, a line low enough that an ordinary department could cross it simply by documenting somewhat less than a higher-documentation department does.

Checking that line against every file we hold turned up the same defect sitting in a number we had already published: the three suite departments used for engine regression run a natural 43 to 45% missing-case rate, so the old rule had been flagging roughly half of every regression roster for weeks, inflating the calibration-suite score we cite below by sixteen patterns. We are correcting that figure on this page rather than letting a stale number stand once we found it wrong ourselves. The rule now names an individual only when their rate clears both the fixed floor and their own department's median by a real margin, and when the department's median itself runs high, one department-level finding states the norm instead of naming a fifth of the roster for sharing it.

Every fix on this page was verified the same way: reproduce the defect, measure its cost on real data where possible, write the fix, confirm the twenty-log regression suite and every large file this program holds still score as expected or explain exactly why a score changed, and only then ship.

Running large-file campaign totals: zero of twenty-six legitimate, documented police incidents were ever escalated, across 4,800,314 records and 114 planted behaviors scored against sealed answer keys, of which 65 were detected (57%). Six engine defects were found across the campaign, all disclosed above, all fixed the same day they were found. Combined with the twenty-log regression suite (140 patterns, 100 detected after correction) and the original forty-log development program below, the full published test program spans 5.1 million synthetic records. A separate capacity test processed 2.4 million records of clean data with zero findings.

These design properties were verified during the program.

• Determinism held. Identical input produces identical findings and an identical verification hash, every run.
• The two engines agreed. Two independently built implementations of the engine produced exactly matching findings, with the same rules, severities, subjects, and row counts, on all forty logs.
• Privacy held throughout. Plates are one-way hashed at ingest, audits complete with network access disabled, and public reports contain no personnel identifier from the source log, which automated substring testing verified, while carrying the same verification hash as the sealed version.
• The review posture stayed cautious. The engine raised a median of two to four additional medium or low question-class items per audit on organically generated activity, each printed with an explicit innocent explanation, because the tool produces review leads rather than findings of policy violation.

These are the honest limitations.

• The engine has no detection thresholds fixed to a department's size. Most rules compare an account to its own department's own statistics, computed fresh from the file being reviewed, and a handful of hard floors (a burst is still measured in searches per minute, because that is a statement about how fast a person can work, not about department size) hold constant on purpose. The tested range now runs from 21 accounts at 28 searches a day to 90 accounts at nearly 1,900, verified against sealed keys at both ends, and every report states the account count and volume actually measured. Below eight active accounts, peer comparison is skipped as statistically meaningless, and the report says so.
• Synthetic testing measures detection mechanics, not the base rate of real-world misuse. Row-level recall (65%) is lower than pattern-level recall (84%) by design on the original forty-log program: rules flag enough of a pattern to trigger human review, not every constituent row.
• A fixed ruleset can in principle be evaded by someone who knows exactly where its lines sit. This engine's lines are not fixed: they are recomputed from each department's own data on every run and never published, so there is no stable boundary to learn from outside. Six real defects were found in this engine during the tests on this page, and every one was found by the process itself, disclosed, and fixed the same day, which is the countermeasure this report documents: adversarial retesting on every release, published including the runs where we were wrong.

Test data authored independently without access to detection thresholds. All results locked to disk before answer keys were opened. The development/confirmation split, every inter-phase fix, and every miss are disclosed above. Reproduction: identical files through either engine yield identical findings.