Automating Security Framework Mapping

The problem, plainly

Every quarter, auditors across dozens of entities logged security findings into a central tracker. Every finding, in principle, corresponded to a specific control in the Security Assurance Framework, a catalog of the security requirements the organization had committed to.

In principle.

In practice, the findings arrived as unstructured free text written by many different people in many different languages, with no classification attached. The Security Assurance Framework, meanwhile, spoke in formal, standardized language. A finding like "passwords still stored in plain text on the HR portal" had to be mapped, by hand, to a control that said something closer to "cryptographic protection of stored authentication credentials." The mapping was possible for a human to do. It was not possible for a human to do quickly, consistently, or at scale.

That gap is what I was asked to close.

Why this is more interesting than it looks

The obvious framing is "text similarity, throw a transformer at it." That framing is not wrong, but it hides the parts of the problem that actually make it hard.

Three things made this a real modeling problem, not a demo:

The vocabulary gap is real. Findings use operational language. Controls use policy language. A model that leans on surface overlap will confidently produce wrong matches, because the wrong controls often share more surface words with a finding than the right control does. Semantic similarity is not a nice-to-have. It is the whole point.

Roughly 38% of the input was not in English. The Security Assurance Framework is monolingual English. Any solution had to make peace with that mismatch before any modeling could begin.

Compliance domains do not tolerate opacity. A model that outputs a control ID and nothing else is useless here, no matter how accurate it is. Every mapping decision must be defensible to an auditor. "The model said so" is not a defense.

Architecture at a glance

The system is retrieval-then-rank, with a human-in-the-loop step between the unsupervised proposal stage and the supervised learning stage.

Raw findings (quarterly extract)
       |
       v
Translation (internal enterprise LLM)
       |
       v
Preprocessing (cleaning, lemmatization)
       |
       v
Stage 1: BM25 / TF-IDF candidate retrieval
       |   -> top 20 candidate controls per finding
       v
Stage 2: Sentence-BERT semantic reranking
       |   -> reranked candidates with semantic scores
       v
Security Assurance team validation
       |   -> labeled (finding, correct control) pairs
       v
Stage 3: LightGBM learning-to-rank
       |   -> final ranked suggestions + explanations
       v
Auditor UI

This is the same shape that production search and recommendation systems converge on. It is not novel. It is right.

The decisions that mattered

Anyone can wire up SBERT to a similarity score. The interesting question is which of the five or six plausible design choices you make, and why. Here are the four that shaped this system.

1. Translate up front. Do not go multilingual.

The obvious alternative to translating findings into English was to use a multilingual embedding model. That would have saved a preprocessing step and kept the pipeline monolithic.

I did not do it, and the reason is one that generalizes: the target catalog is monolingual. When one side of a similarity comparison is fixed in a single language, translating the other side into that same language keeps both sides in the same embedding manifold. A multilingual model introduces an extra source of variance in exchange for solving a problem I do not actually have.

Translation happened once, upstream, and translated text was cached alongside the original.

2. Retrieval before semantics, not instead of it.

Skipping BM25 and going straight to dense semantic comparison across the full control catalog would have worked. It would also have wasted most of the compute.

BM25 is cheap and it is right most of the time. For every finding, it narrowed the full catalog to the top 20 candidates in milliseconds. Sentence-BERT then reranked those 20 candidates by genuine semantic similarity. This is not a compromise between accuracy and speed. It is a division of labor: lexical retrieval handles the easy cases, semantic reranking handles the hard ones. Both stages produce features the downstream model can use.

The threshold that mattered here was recall at N. If the true match is not in the top 20 candidates, no amount of reranking can save it. I tuned candidate count against recall on validation samples until the retrieval stage was rarely the bottleneck.

3. Frame the supervised stage as ranking, not classification.

This was the highest-leverage architectural call in the project.

The natural first instinct is to build a multi-class classifier: given a finding, predict its Security Assurance Framework control ID. I built that. It struggled, and for reasons that generalize:

• The class space is wide and the labeled examples per class are thin. A flat classifier over hundreds of controls asks a small labeled dataset to do too much.

• It throws away the structure the retrieval stage already produced. The classifier considers every control from scratch when we already know 19 of the top 20 candidates are semantically nearby.

• Every new control added to the framework requires retraining the classifier's output head.

Reframing the problem as learning-to-rank fixed all three. Instead of "given this finding, pick one of hundreds of classes," the model answers "given this finding and these 20 candidates, order them." The label space per training example collapses from N-way to 20-way. The retrieval structure becomes a feature, not an obstacle. New controls can enter the framework without touching the model, as long as the feature set is stable.

LightGBM with a lambdarank objective, using SBERT similarity, BM25 score, token overlap, severity, risk category, and domain match as features, beat both the logistic regression baseline and the feed-forward neural network I trained as alternatives.

4. Do not fine-tune SBERT.

Contrastive fine-tuning of SBERT on the labeled (finding, control) pairs was on the table. I chose not to do it in the final production pipeline.

Two reasons. First, with a modestly sized labeled dataset, fine-tuning a large model risks overfitting to the label distribution of a single quarter. Second, and more importantly, keeping SBERT as a stable pretrained component and concentrating all learning-from-labels behavior in one downstream model (LightGBM) makes the system easier to monitor, retrain, and reason about. When something drifts, I know where the drift lives. When something needs to be retrained, only one component moves.

This is a design principle worth stating on its own: push the learned parts to where they can be cheaply monitored and cheaply replaced. Fine-tuning a transformer to save five points of Top-1 accuracy is not obviously worth entangling the semantic layer with your organization's data.

The human loop

The Security Assurance team, the same people who would ultimately use the system, were the labelers. Not an external annotation service. Not the modeling team wearing another hat.

They received the ranked proposals from Stages 1 and 2, marked correct mappings, corrected wrong ones, and flagged findings where none of the top candidates were right. Confirmed matches became positive training pairs. Corrections became additional positives. Flagged findings became a signal that the framework itself might need a new control.

This choice was not primarily a data quality decision. It was an organizational one. The labels are the same judgments the labelers will apply in production. That tight loop is what made the model's outputs defensible when the auditors started using them.

Results

The correct control is the top suggestion 93.5% of the time. It is among the top three suggestions 99.2% of the time. In the remaining ~0.8% of cases, the auditor searches or escalates, which is the correct behavior when the model is genuinely uncertain.

I want to be honest about what these numbers mean. Top-1 accuracy on a ranking task with a strong retrieval stage is going to look high. The number that actually matters to the user is Top-3, because that is what the UI shows and that is what determines whether the auditor has to search further. The number that actually matters to me as a builder is the tail: the 0.8% where nothing works, because that is where the framework itself is telling me it has gaps.

Explainability, on purpose

Every prediction ships with:

1. A similarity score as a confidence signal.

2. Matched key terms between the finding and the suggested control, drawn from the lemmatized token intersection.

3. Feature attributions from the ranker, showing which signals (semantic similarity, lexical overlap, metadata match) drove the ranking.

   
   

Example output:

In compliance, this is not a nice-to-have. It is the difference between a tool the auditors will use and a tool they will refuse.

What I would change

Three things, in decreasing order of leverage.

Frame the problem as ranking from day one. I went through the classification framing first because it was the intuitive first move. It cost me time. The pattern I would carry into the next project of this shape: if the label space is wide and the labels per class are thin, start with ranking.

Build the explainability surface into the pipeline earlier. I added it later, in response to the compliance requirement. Adding it earlier would have improved the modeling too, because feature attributions surface data quality issues faster than accuracy numbers do.

Invest earlier in the retraining loop. Every quarter produced new labeled data from the human loop. The value of that data compounds over time only if the retraining path is smooth. I would have automated more of it, sooner.

Closing thought

The interesting question in an applied ML project is rarely "which model." It is "how do the pieces fit together, and where do you put the parts that learn?" This system works because BM25 does what BM25 is good at, SBERT does what SBERT is good at, LightGBM does what LightGBM is good at, and the humans do what only the humans can do. None of the individual components are novel. The composition is what earned the auditors' trust and what made the results hold up in production.

If any of this sparked a design idea or a critique, I would love to hear it. The best conversations I have had about this project have been with people who disagree with one of the four choices above.

Same pattern, different problem

The Security Assurance Framework mapping described above was built inside a specific organizational context, on data that cannot be shared. The architectural pattern is not context-specific. To make the approach concrete and inspectable, I built a public reference implementation applying the same pipeline to a different problem: mapping CVE vulnerability descriptions to CWE weakness categories, over fully public NVD and MITRE data.

Same retrieval-then-rank shape. Same LightGBM ranker on top. Same SHAP-based explanations. The domain is different, the data is public, the code is open. If any part of the design above is worth a closer look, that is where to look.