False positives: why they happen and how to triage them

Any scanner that works from patterns will sometimes be wrong in the cautious direction: it flags something that turns out to be harmless. MyVaultMate is rule-based, and occasional false alarms are part of how it works. That is a reasonable price for catching more real exposures, but only if you have a quick way to sort the real from the harmless. This article is that routine.

Why false positives happen

A pattern knows the shape of a value, not its meaning. Several ordinary things share the shapes of sensitive data:

• Nine-digit identifiers. An order number or a case reference written as 123-45-6789 has the same shape as an SSN.
• Long numbers that pass the Luhn check. Card detection uses a pattern plus a checksum, which removes most look-alikes. But a long tracking or reference number will pass the checksum on occasion by chance. The details are in our article on card detection.
• Test and sample data. A developer's fixture file, a training spreadsheet or a software vendor's demo database often contains made-up SSNs and card numbers built to look real.
• Documentation and templates. A help guide that shows "123-45-6789" as an example is a pattern match with no person behind it.
• Health terms in non-records. A training handout or a blank form can trip the health signal without containing anyone's information.

A fast triage routine

The report gives you what you need for most decisions without opening the file: the path, the file type, the kind of data, the severity, a short redacted sample. Work through each finding in this order:

1. Path. Where is it? A client folder, a payroll export or a desktop scan is plausibly real. A folder named for software, a vendor, test or sample is plausibly not. Cached and AppData locations are skipped by default, which removes a lot of noise already.
2. Count. One distinct value in a large file points toward a coincidence. Hundreds of distinct values in a column point toward a real list. A low count can be partial until the overnight full count reads the whole file.
3. Redacted sample. The sample, such as ***-**-1234, lets you confirm a hit without seeing the whole value. Combine it with the file name: a sample that ends in the same four digits as the client the file is named for is a very different thing from one that ends in 6789.
4. Open the file only if the first three do not settle it. Do this on the machine where it lives. Do not copy it somewhere else to look.
5. Decide. There are three outcomes: real and needs action, real but acceptable where it is, or not sensitive.

Use severity to set the order

Findings are ranked from Critical to Low. Do not spend your first hour on a Low finding in a test folder while a Critical one sits on a shared desktop. A sensible order is: Critical and High in uncontrolled places first, then Critical and High elsewhere, then everything else. If you find a pattern, such as one vendor export that triggers dozens of findings, deal with the pattern once instead of file by file.

Recording the decision

Whatever you decide, write it down. A simple log works fine, with one line per decision: the date, the file path, who reviewed it, the outcome and the reason. For example, "Not sensitive: vendor test file, made-up numbers", or "Real: moved to the tax software, original retired per retention policy." Findings can be worked through in the dashboard, and re-scans show progress, so the log and the dashboard together become your evidence that you looked.

That written trail matters more than it seems. If you operate under the FTC Safeguards Rule or a similar rule, you may be expected to document your risk assessment and how you respond to what it finds; a plain record of triage decisions helps document that you did the work. It does not, by itself, make you compliant, and your attorney or advisor can tell you what documentation your situation calls for.

Keep the false alarms from coming back

• Keep test and training data out of the folders you scan, or in a clearly labeled place.
• Put real example numbers in templates in a form that cannot pass for real, such as masked digits.
• Choose scan paths deliberately. You decide which drives and folders are scanned and can remove any at any time.

A false alarm costs you a few seconds. A missed exposure costs far more, so the right goal is a fast, consistent triage habit rather than fewer detections.