Teaching a model to read threat reports
When APT29 slipped a backdoor into SolarWinds Orion updates, the technical detail, the forged authentication certificates, the abused application-layer protocols, the whole SUNBURST kill chain, eventually made it into public threat reports. It just took analysts a long, manual read to turn those paragraphs into something a SIEM could act on. Multiply that by every APT report published every week, by every research team too small to have an analyst free to sit and read, and you get a backlog that never clears. That gap between a report landing and its contents becoming a detection rule is what I built TIEF, the Threat Intelligence Extraction Framework, to close.
Two kinds of intelligence, and one of them expires
A threat report usually mixes two different things. Indicators of Compromise are the concrete artifacts, a file hash, an IP, a domain, and they are useful precisely because they are specific. They are also brittle. An adversary changes a hash or rotates infrastructure and the indicator is dead, so IOCs only ever tell you about a compromise after it happened. Tactics, Techniques, and Procedures are the other half: how the group actually operates, independent of which server they used this week. TTPs are what let a defender hunt proactively instead of just matching a blocklist. TIEF's job is to read a report and pull out both, but treat them differently, IOCs get extracted and tagged, TTPs get classified against a framework a SOC already understands: MITRE ATT&CK.
Five stages between a PDF and a STIX file
The pipeline breaks into stages that hand off to each other. Preprocessing strips a raw report down to clean sentences. Those sentences get grouped by topic using BERTopic, transformer embeddings reduced with UMAP and clustered with HDBSCAN, so that sentences about "exfiltrating the information" or "enumerating directories and files" end up bucketed together even when they never share a keyword. In parallel, a regex-plus-gazetteer pass extracts the thirteen IOC types I cared about (IPs, hashes, CVEs, registry keys, and so on) and soft-tags them: an actual value like 192.168.0.1 gets swapped for the generic token IPv4 in the text, with the real value kept aside in a lookup table. That one design choice matters more than it looks, it stops the classifier from memorizing specific IPs instead of learning the behavior around them. Everything then feeds a fine-tuned DistilBERT model that predicts MITRE ATT&CK technique and sub-technique IDs, and the final stage packages sentence, IOCs, and TTPs into STIX 2.1 objects i.e. Attack Pattern, Indicator, Report, Relationship. So the output plugs into whatever a SOC is already running instead of sitting in a PDF.
Teaching it the techniques nobody writes about much
MITRE ATT&CK has 560 techniques and sub-techniques once you count Enterprise, ICS, and Mobile together, and I wanted TIEF to recognize all of them, not just the popular dozen. The data didn't cooperate. I scraped roughly 11,000 procedure descriptions straight from MITRE, and the class distribution was brutal, some techniques had 300 example sentences, others had one. Trained on that as-is, the classifier's F1 score was 35%, which is close to useless. So I used an LLM, Claude Sonnet, to generate around 65,000 additional synthetic procedure descriptions for the underrepresented classes, anchored to real technique IDs and real examples so the synthetic data stayed on-topic, then had graduate and PhD reviewers manually check them for technical accuracy before they went anywhere near training. That single fix took the F1 score from 35% to 93%, for a barely-there cost of one percentage point off the model's overall discriminative power. There's something fitting about a language model generating the training data that taught another language model to read security reports.
Where it holds up, and where it still trips
On held-out MITRE data, TIEF lands an F1 of 0.933, and it can tell the difference between a generic technique and a specific sub-technique underneath it, flagging T1218.010, Regsvr32 abuse, instead of just the broad "System Binary Proxy Execution" bucket it sits in. That granularity is the whole point. But when I ran it against CTI-Bench, an independent benchmark of real-world vulnerability write-ups, accuracy dropped to 55.4%. Digging into the misclassifications, most of it wasn't the model being wrong so much as being asked the wrong question: CTI-Bench entries describe a vulnerability at a high level, while TIEF was trained to recognize MITRE's granular, step-by-step "procedures." Roughly 58% of the errors traced back to genuinely similar techniques sharing vocabulary, the model confusing a file-upload flaw with system discovery because both mention "upload" and another quarter came from input text that simply didn't have enough procedural detail to classify correctly. That's a benchmark mismatch as much as a model limitation, and it's the kind of result that's more useful to publish than to bury.
DistilBERT is also a general-purpose model, not one pretrained on security text the way SecureBERT or CyberBERT are, which is the next thing I'd swap out given more time. But the core idea held: turning a report into structured, queryable intelligence doesn't require a human in the loop for the first pass, and that's enough to let a SOC react to new reporting at the speed it actually arrives instead of the speed an analyst can read.