Microsoft Purview Custom Sensitive Information Types: Building Regex and Supporting Evidence That Actually Work
Discover how to build high-accuracy custom sensitive information types in Microsoft Purview by combining regular expressions, proximity settings, and supporting evidence keywords. Stop wrestling with false positives and learn how to design precision data detectors that protect proprietary company data without drowning your IT team in noisy alerts.
Key Takeaways
- Custom sensitive information types rely on regular expressions (regex) to capture proprietary patterns like employee IDs, internal project codes, and specialized contract numbers.
- Raw patterns without context fail; combining regex with supporting evidence keywords and strict proximity rules dramatically reduces false positives.
- Keyword dictionaries streamline management by letting you group hundreds of related terms into a single reusable object.
- Rigorous testing with real-world sample documents before deployment ensures your rules catch actual leaks without blocking routine business operations.
- Choosing the right confidence levels prevents your dashboard from filling up with noise from generic numeric strings like order IDs or phone extensions.
The Limits of Built-In Microsoft Purview Detectors
Microsoft ships hundreds of out-of-the-box sensitive information types, covering everything from credit card numbers and US Social Security numbers to European IBANs and medical identifiers. These built-in detectors are essential for general compliance checkboxes. However, they only cover data that looks identical across every company on the planet. Your business relies on unique internal data structures that no software vendor has ever seen.
When you need to protect proprietary information—such as internal employee serial numbers, restricted project codes, specialized customer account numbers, or legacy system ticket formats—built-in detectors fall completely short. This is where you must move beyond generic templates and design custom sensitive information types. Building these from scratch requires careful tuning so that your rules accurately recognize your organization's unique digital footprint without treating every random string of digits as a high-risk security breach.
Writing Effective Regular Expressions for Data Detection
The core engine of any custom sensitive information type is a regular expression, or regex. A regex is a compact code that describes the strict structural shape of a string rather than its literal content. For instance, if your internal contract numbers always start with the letters "CN" followed by a dash, a four-digit year, another dash, and six concluding digits, a properly structured regex will capture every string matching that exact formatting.
However, pattern matching alone introduces severe risks if not handled correctly. Writing a regex that is too loose—such as matching any seven-digit number because it seemed fast and convenient during initial testing—will instantly backfire in production. Once deployed live, a loose pattern will flag phone numbers, standard zip codes, invoice numbers, and routine financial spreadsheets. A regex with no structural discipline acts merely as a random number detector, flooding your Purview audit logs with noise.
The Power of Supporting Evidence and Proximity
To transform a basic character pattern into an intelligent data detector, you must incorporate supporting evidence and proximity rules. Shape alone does not equal meaning. An internal employee ID structured as the letter "E" followed by seven numbers might easily collide with supplier product SKUs or standard shipping references.
By adding supporting evidence keywords—such as "staff number," "HR record," or "employee profile"—you give the detection engine semantic context. Furthermore, you must define proximity rules to dictate how close those keywords must sit relative to the matched pattern. If a keyword appears in a document footer three pages away from a numeric string, it shouldn't trigger a match. Tightening your proximity window ensures that context is genuinely tied to the data value.
Leveraging Keyword Dictionaries for Scale
Managing extensive lists of terms directly inside individual rules quickly becomes an administrative nightmare. Instead of typing dozens or hundreds of individual words into every custom rule, organizations should leverage keyword dictionaries. A keyword dictionary consolidates a large list of related terms—such as internal system code names, confidential project tags, or customer entities under active NDA—into a single, easily manageable object.
When building dictionaries, quality far outweighs volume. Dumping two thousand generic words like "plan," "summary," and "review" into a dictionary will ruin your detection accuracy because those words appear in normal corporate writing all day long. Every term added to a dictionary must be specific enough to meaningfully indicate sensitivity when paired with your underlying pattern.
Testing and Validation Before Production Deployment
The single biggest mistake compliance teams make is deploying a newly crafted custom sensitive information type directly into an enforcement policy without empirical testing. Never rely solely on made-up test examples that you wrote yourself to prove the rule works. Instead, collect actual documents and emails from the business environment.
Microsoft Purview lets you upload sample files directly into the portal to test what your custom type catches and misses. Build a balanced test set containing documents that should match, documents that should explicitly not match, and borderline edge cases. If your custom detector flags something it shouldn't, analyze why—usually, the issue stems from an overly broad regex pattern, a faulty proximity window, or a keyword that appears too frequently in standard business communications. Refine your rule, retest, and iterate until the signal-to-noise ratio is razor-sharp. For a broader exploration of how these custom elements fit into comprehensive deployment strategies, check out Listen to the full episode and discover how to design policies that truly protect your cloud environment without alienating your workforce.
Frequently Asked Questions
What is the primary purpose of a custom sensitive information type in Microsoft Purview?
Custom sensitive information types allow organizations to detect and protect proprietary data formats unique to their business—such as internal employee IDs, custom project codes, or specialized account numbers—that are not covered by Microsoft's built-in detection library.
How do proximity settings prevent false positives in regex-based rules?
Proximity settings restrict how close supporting evidence keywords must be to the detected regex pattern. By requiring keywords to appear within a specific character distance, the engine avoids triggering matches on unrelated words scattered across different sections of a document.
Why should I avoid using generic words in a keyword dictionary?
Generic words appear frequently in normal daily communications. Including common terms in a keyword dictionary causes the detection engine to trigger false positives constantly, eroding user trust and generating unnecessary administrative noise.
How can I test a custom sensitive information type before deploying it?
Microsoft Purview allows you to upload real-world sample documents and emails into the portal to evaluate performance. You can test your custom rules against a mixture of positive, negative, and borderline files to verify accuracy before enabling the policy for end-users.
