URL: https://orion.hefaistos.org
Parent: HEFAISTOS Detection Engineering Platform: https://detect.hefaistos.org
Core Module Expansion: KEDALION Project https://kedalion.hefaistos.org
Introduction: The Fragility of Modern Detections
Detection engineering is currently engaged in an asymmetric technological arms race. For years, practitioners have focused on collecting vast quantities of data, believing telemetry acquisition to be the final milestone. We were wrong. Adversaries have demonstrated that collecting logs is insufficient if those logs are not paired with resilient, continuously validated query logic.
We have built a house of cards. Many modern detections are fragile; they are precisely tuned to detect specific, static threat artifacts rather than the underlying Tactic, Technique, or Procedure (TTP). An adversary shifting their obfuscation method—for instance, changing from standard Base64 encoding to a slightly varied custom alphabet—is often enough to bypass a significant percentage of production detections that hunt for Obfuscated Files or Information (T1027) or Deobfuscate/Decode Files or Information (T1140).
The consequences of this fragility are severe: False Negatives (FNs), where legitimate threats slip by, and False Positives (FPs), which generate alert fatigue for SOC analysts and lead to Security Controls Evasion (SAC0019). A major factor contributing to this fragility is that detection engineers do not have an efficient, localized method for high-fidelity testing before deployment. We cannot test a new rule for Security Account Manager (T1003.002) on a production Domain Controller.
This is why we built ORION Detlab.
ORION Detlab is a decentralized, high-speed detection validation factory. It exists to remove the abstraction from rule validation. It allows you to take raw telemetry from real adversary datasets (such as Mordor or OTRF), programmatically mutate that telemetry, and test your KQL logic with absolute certainty. We are moving from static detections to validated defenses.
Why ORION? Motives, Design, and the Call for Resilience
The motivation behind ORION Detlab stems from a singular, unified goal: resilience. A detection is only as good as the conditions we simulate when designing it. Static testing is dead. Adversaries are dynamic, and our detections must match that adaptability.
The Resilience Problem
Traditional detections hunt for IOCs—IP addresses, file hashes, domains. These are the bottom level of the Pyramid of Pain; they are “trivial” for the adversary to change. When we hunt for TTPs (e.g., Execution (TA0002)), we are aiming higher, but we still often fail.
Consider a simple hunt for Process Injection (T1055). If we write a rule that specifically looks for CreateRemoteThread in Sysmon Event ID 8, we have defined a good procedural hunt. But we are only resilient against adversaries using CreateRemoteThread. We are not resilient against an adversary who changes their payload to use NtCreateThreadEx or QueueUserAPC.
Resilient detections must focus on the invariant behavior—the semantic core of the technique—rather than the implementation artifact. ORION provides the localized forge to test how easily your logic fractures when subjected to slight implementation variances.
Localized Pre-SIEM Validation
We must stop using production SIEMs as test environments. When you execute a query for Event Log Analysis (D3-ELA) inside a live Microsoft Sentinel workspace, you are burning expensive compute and risking accidental degradation of real-time monitoring.
ORION Detlab is your forge. It is designed to be hosted anywhere—your workstation, a lab server, or a cloud instance—completely decoupled from your production stack. This localized validation engine provides extreme speed. You upload your telemetry (e.g., a multi-stage attack simulation of Initial Access (TA0001) or Persistence (TA0003)), write your logic, and hit ‘test.’ A full simulation across 2,000 JSON events takes less than 150ms. This loop is critical for the rapid iteration of KQL logic.
Platform Architecture and the Pivotal Shift to Native KQL
ORION Detlab is a Twelve-Factor App. It is modern, containerized, and secure.
Technology Stack
- Frontend (Port 3003): React UI featuring a custom CRT-styled cyber-terminal aesthetic, optimized for high-resolution SOC environments.
- Backend (Port 8888): FastAPI (Python) routing engine handling data aggregation, AI interaction, MSTICPy integration, and raw KQL logic checks.
- Relational Database (PostgreSQL): Stores user credentials (Role-Based Access Control, RBAC), encrypted API keys, and multi-tenant configurations.
- NoSQL Document Store (MongoDB): High-speed, localized datastore where massive adversary datasets (Telemetry Ingestion) are serialized and prepared for high-fidelity replay simulation.
The Case for Native KQL Ingestion
ORION Detlab initially incorporated a complex Sigma YAML compilation engine. This was a technical mistake. We have since pivoted to a native-KQL ecosystem. We are making KQL the absolute source of truth.
Why? Sigma is a useful abstraction, but it is not a real-time query. While helpful for translating a rule for Command and Scripting Interpreter (T1059), forcing an engineer to debug both the input YAML and the output KQL simultaneously is inefficient and error-prone. We found the internal Sigma translator (via MSTICPy) was finicky with complex nested logical operators, leading to broken output.
We have replaced the compiler with a sleek [2] KQL VALIDATOR & AI EXPLAINER. The engineer pastes their finalized KQL. The platform validates the syntax and immediately feeds it to the AI Heuristic Engine (Gemini Pro/Flash LLM). The AI analyzes the rule and generates a human-readable threat intelligence summary, mapping the exact technique (like T1059) directly to D3FEND Process Analysis (D3-PA) and Engage SAC0019 mappings before the rule is even tested.
We assume that senior detection engineers know how to write KQL. We provide tools that help verify that knowledge, rather than obscuring it behind a middle abstraction.
Deep Dive into Advanced Features for Juniors and Seniors
ORION Detlab provides a gradient of tools, moving from basic syntax validation to advanced mutation and AI debugging.
For Juniors: The AI Root Cause Analyzer and Heuristics
A significant hurdle for new detection engineers is False Negatives. When your simulator sweep across the chronological dataset returns 0 HITS, it is incredibly frustrating. The mistake is often not logical, but structural: mismatched JSON keys (e.g., EventID vs. event_id) or incorrect time formatting.
We solve this with the [AI DEBUG] function in the Telemetry Replay Simulator. This is a closed-loop system:
- When a simulation fails, the AI requests your current KQL rule and a random 5-log sample from the active MongoDB dataset.
- The model acts as a senior engineer, performing a schema comparison and root cause analysis (RCA).
- The AI outputs a precise diagnostic: “Your KQL is looking for Sysmon 1 (
EventID == 1), but the uploaded dataset uses the keyevent.event_id. Your query logic will fail.”
This turns every False Negative into a learning opportunity, accelerating the maturity of junior engineers.
For Seniors: The Mutation Engine and D3FEND Validation
Senior engineers understand that validation is not binary; it is resilient. They use ORION not just to see if a rule works, but to see how badly it breaks. This is where the Mutation Engine becomes critical.
If you are validating a rule for Obfuscated Files or Information (T1027) or Command and Control (TA0011), you can programmatically mutate the telemetry payloads. Senior engineers can test against subtle variances like Base64 encoding/decoding, character mixed-casing, or reverse-string mutations, simulating Security Controls Evasion (SAC0019).
By stress-testing the KQL against these programmatic variations, ORION Detlab helps seniors move beyond naive string matches (e.g., CommandLine contains "powershell") to resilient hunts (e.g., using CommandLine length variances and rare process parent/child relationships). This moves beyond basic validation to active countermeasure testing, directly mapping the KQL to MITRE D3FEND countermeasures like Process Analysis (D3-PA) and Authentication Event Analysis (D3-AEA).
The Two-Way CI/CD GitHub Pipeline: The Full Detection Lifecycle
ORION Detlab is not a tool; it is a workflow. It completes the entire detection engineering lifecycle by providing a Two-Way Sync directly with your security orchestration repository.
[1] SYNC PULL: The Pre-Built Library
Detection engineering does not start in a vacuum. Under the UserProfile tab, you configure a GITHUB_PULL_REPO and a GITHUB_PUSH_REPO (mapped in your PostgreSQL multi-tenant state). The platform securely queries the GitHub REST API and generates a localized dropdown menu of your entire KQL library, which simultaneously populates the “ACTIVE KQL RULES” counter in the Global Statistics Bar.
You select an existing production rule for Hijack Execution Flow (T1574). ORION downloads the content and populates the KQL Validator. It is immediately ready to be tested against a newly ingested dataset.
[2] SYNC PUSH: Validated Detection-as-Code
The ultimate Purple Team workflow is simple:
- Ingest adversary telemetry for TA0011 or other behaviors.
- Pull and validate an existing rule (SYNC PULL).
- Discover that the rule fails to detect the new threat variance.
- Stress-test the rule in the Telemetry Replay Simulator, using the Mutation Engine to find the invariant behavior.
- AI-Explain the rule, automatically tagging the new commit with the proper ATT&CK, D3FEND, and Engage mappings.
- Trigger the SYNC PUSH pipeline to commit the tuned, validated, high-fidelity rule directly back to the GitHub repository.
The resulting commit, automated and version-controlled, ensures that the production SIEM only accepts rules that have been localized, stress-tested, and validated against the invariant adversary behavior, mitigating risks of Impair Defenses (TA0005, T1562).
Conclusion: We Are Ready. Let’s Build Resilient Defenses.
Detection fragility is a technology problem that requires a methodological solution. We have focused on log collection for too long. We must now focus on validated defense validation.
By making native KQL the source of truth, leveraging AI for heuristics and debugging, and enabling a seamless two-way CI/CD pipeline, ORION Detlab provides the essential localized forge for the modern Purple Team workspace.
We invite junior and senior engineers to the platform at orion.hefaistos.org. Ingest your data, pull your rules, stress-test your logic, and commit your resilient defenses. The asymmetrical arms race is not over, but with ORION Detlab, the defensive side just deployed a massive tactical upgrade.
Lets do it. We are ready to rumble.