Camera vendors and independent practices alike now offer tools that claim to detect misuse of license plate reader systems, which is a genuine improvement over having nothing at all. This page sets out how anyone can check whether such a tool does what it claims, and we hold our own engine to every test on it.
A detection tool is a claim about the future, since it asserts that if misuse occurs, the tool will surface it. Claims like that are only worth what the evidence behind them is worth, and the evidence has to exist before anyone relies on the tool, rather than arriving after a case goes wrong. What follows is not a criticism of any product, because we would rather every tool in this category get better, and a shared standard is how that usually happens.
Auditing a plate reader system is unusual work, because the volume defeats the ordinary method. A department can generate hundreds of thousands of searches in a year, which makes reading every one impossible, so oversight has historically meant a manual spot check of a small random sample or a reaction to a complaint that someone already filed. Automated detection is a real answer to that problem, and the reason to test it carefully is precisely because so much now rests on it.
The difficulty is that a detection tool almost always looks like it is working. It produces some findings, and the findings look plausible, and nobody can tell from the output alone whether the tool caught most of what was there or a small fraction of it. The only way to know is to run the tool against activity where the right answer is already known.
Someone other than the tool's author writes synthetic search logs containing specific misuse behaviors, records exactly which records constitute each behavior, and seals that key. The tool then runs and its results are locked before the key is opened. Without a key written in advance, every result can be rationalized after the fact, and a tool that finds four things out of eight looks identical to a tool that found everything there was.
The same logs carry legitimate police work engineered to look statistically suspicious, meaning several officers converging on one plate during a stolen vehicle call, or a cluster of overnight searches during a missing person response. A tool that flags these is worse than useless, because it teaches a department that the alerts are noise, and the alert everyone ignores is the one that mattered.
The trap rows make this measurable rather than rhetorical, so a tool should be able to say how many times it pointed at legitimate work across its whole test program. The honest form of this figure is a count and a denominator rather than an adjective, and a tool whose publisher cannot produce it has not measured the thing that matters most to the officer on the receiving end.
A test program that reports only what a tool found is marketing rather than evidence. Publishing the misses tells a reader what the tool does not see, which is the single most useful thing an oversight body can know before relying on it, and it is also the part that no vendor enjoys writing.
An audit that returns different results on different runs cannot be checked by anyone, and it cannot be defended in a hearing where somebody asks why the number changed. Deterministic arithmetic over the records makes a finding reproducible by a third party, which is what turns a report into evidence.
A flag that says an account behaved unusually, without naming which searches, gives a reviewer nothing to work with and gives the accused nothing to answer. Each finding should list its underlying records so a supervisor can read them, and so the person flagged can show the duty roster or the case file that explains them.
Nearly every pattern that indicates misuse also occurs for ordinary reasons, since overtime, a call-out, a long investigation, and a repeat offender all produce activity that looks irregular in a log. A tool built for accountability rather than accusation states the innocent explanation alongside the finding, because the purpose is to raise a question a human can answer rather than to deliver a verdict nobody reviewed.
A tool validated only on small samples can fail quietly on a full year from a busy department, either by breaking outright or, far worse, by silently disabling part of its own logic at scale. Testing should happen at the record counts real agencies actually produce, and whatever limit exists should be published rather than discovered by a client.
This one is structural rather than technical, and it is the test a vendor built tool cannot pass on its own. When detection reports only to an administrator inside the department under review, the department decides what happens next, and the public is asked to accept that the process worked. That may well be true, and the point is that nobody outside can confirm it. A tool intended to reassure a community needs a path by which findings reach someone the community can see, which usually means a published version and a way to verify the document has not been altered.
We publish our record because a standard nobody applies to themselves is not worth reading. Our detection engine has been measured across 5.1 million synthetic records, comprising an original forty log program, a twenty-log regression suite, and eight separate large-file tests that were run blind, with results locked to disk before any sealed answer key was opened. Across the large-file campaign, 114 planted behaviors were scored and 65 were detected, and twenty-six legitimate documented incidents were presented as traps with none escalated. On the strongest test the engine detected 17 of 18 planted behaviors on a file it had never seen, under a protocol where the answer key stated the log's SHA-256 so the pair could be verified as matching. On the weakest, a file engineered specifically to defeat column matching, it scored zero, and we publish that result at zero rather than only publishing the tests we pass.
The campaign failed us six separate times, and every failure is written up in full in the accuracy report rather than repaired quietly. A rule raised a finding against two hundred lawful service calls because it matched a single ordinary word. A second rule read a column it could not parse and reported that it had found nothing, which is exactly the silent degradation test eight above exists to catch. A column named for one purpose was matched to a field holding something else entirely with full confidence, which we measured would have accused an entire department under the right conditions had we not caught it. A rule used a fixed threshold that flagged a fifth of an ordinary department for sharing a documentation habit, and checking that threshold against our own regression suite found it had been inflating a number we had already published. Each was fixed the same day it was found, each was regression tested against every file this program holds before shipping, and every one is counted against us in our published totals. A standard is only worth anything if the people who wrote it report the times they fail it.
The misses are published in the same place as the results, in a table covering all eight large-file tests with nothing omitted. The complete record, including every failure, is in our accuracy report.
Publishing the tests is not the same as publishing the trigger points. A tool whose exact numeric thresholds are public tells anyone who wants to avoid it how far to stay under them, so our thresholds are disclosed under confidentiality to a client's attorney or a reviewer of the client's choosing rather than posted here. Every test above can be run and verified without knowing a single threshold value, which is the point of writing them this way.
Passing these tests says a tool detects what it was built to detect, and it says nothing about whether the underlying surveillance program should exist, which is a decision for a community and its elected officials rather than for an auditor. It also does not address retention schedules, destruction records, or whether data sharing matched a signed agreement, since those questions live outside a search log and need a broader data practices review. We would rather state those limits plainly than let a clean audit be read as a broader endorsement than it is.
Any vendor, agency, oversight board, or auditor is welcome to apply these tests, including to us, and we will answer questions about our methodology from anyone evaluating it seriously. If you build a tool in this category and you can beat our numbers, publish them the way we have published ours, because a community deciding whether to trust a plate reader program deserves more than competing assurances.
You can read the full accuracy record, see what a complete export makes possible, or check any report we issue on the verification page.