ThreatHunt/update.md

ThreatHunt Update Log

2026-02-20: Host-Centric Network Map & Analysis Platform

Network Map Overhaul

• Problem: Network Map showed 409 misclassified "domain" nodes (mostly process names like svchost.exe) and 0 hosts. No deduplication same host counted once per dataset.
• Root Cause: IOC column detection misclassified Fqdn as "domain" instead of "hostname"; Name column (process names) wrongly tagged as "domain" IOC; ClientId was in normalized_columns as "hostname" but not in ioc_columns.
• Solution: Created a new host-centric inventory system that scans all datasets, groups by Fqdn/ClientId, and extracts IPs, users, OS, and network connections.

New Backend Files

• backend/app/services/host_inventory.py Deduplicated host inventory builder. Scans all datasets in a hunt, identifies unique hosts via regex-based column detection (ClientId, Fqdn, User/Username, Laddr.IP/Raddr.IP), groups rows, extracts metadata. Filters system accounts (DWM-, UMFD-, LOCAL SERVICE, NETWORK SERVICE). Infers OS from hostname patterns (W10-* Windows 10). Builds network connection graph from netstat remote IPs.
• backend/app/api/routes/network.py GET /api/network/host-inventory?hunt_id=X endpoint returning {hosts, connections, stats}.
• backend/app/services/ioc_extractor.py IOC extraction service (IP, domain, hash, email, URL patterns).
• backend/app/services/anomaly_detector.py Statistical anomaly detection across datasets.
• backend/app/services/data_query.py Natural language to structured query translation.
• backend/app/services/load_balancer.py Round-robin load balancer for Ollama LLM nodes.
• backend/app/services/job_queue.py Async job queue for long-running analysis tasks.
• backend/app/api/routes/analysis.py 16 analysis endpoints (IOC extraction, anomaly detection, host profiling, triage, reports, job management).

Modified Backend Files

• backend/app/main.py Added network_router and analysis_router includes.
• backend/app/db/models.py Added 4 AI/analysis ORM models (ProcessingJob, AnalysisResult, HostProfile, IOCEntry).
• backend/app/db/engine.py Connection pool tuning for SQLite async.

Frontend Changes

• frontend/src/components/NetworkMap.tsx Complete rewrite: host-centric force-directed graph using Canvas 2D. Two node types (Host / External IP). Shows hostname, IP, OS in labels. Click popover shows FQDN, IPs, OS, logged-in users, datasets, connections. Search across hostname/IP/user/OS. Stats cards showing host counts.
• frontend/src/components/AnalysisDashboard.tsx New 6-tab analysis dashboard (IOC Extraction, Anomaly Detection, Host Profiling, Query, Triage, Reports).
• frontend/src/api/client.ts Added network.hostInventory() method + InventoryHost, InventoryConnection, InventoryStats types. Added analysis API namespace with 16 endpoint methods.
• frontend/src/App.tsx Added Analysis Dashboard route and navigation.

Results (Radio Hunt 20 Velociraptor datasets, 394K rows)

Metric	Before	After
Nodes shown	409 misclassified "domains"	163 unique hosts
Hosts identified	0	163
With IP addresses	N/A	48 (172.17.x.x LAN)
With logged-in users	N/A	43 (real names only)
OS detected	None	Windows 10 (inferred from hostnames)
Deduplication	None (same host 20 datasets)	Full (by FQDN/ClientId)
System account filtering	None	DWM-, UMFD-, LOCAL/NETWORK SERVICE removed

2026-02-23: Agent Execution Controls, Learning Mode, and Dev Startup Hardening

Agent Assist: Explicit Execution + Learning Controls

• Problem: Agent behavior was partly implicit (intent-triggered execution only), with no analyst override to force/disable execution and no explicit "learning mode" explainability toggle.
• Solution:
  • Added execution_preference to assist requests (auto | force | off).
  • Added learning_mode flag for analyst-friendly explanations and rationale.
  • Preserved deterministic execution path for policy-domain scans while allowing explicit override.

Backend Updates

• backend/app/api/routes/agent_v2.py
  • Extended AssistRequest with execution_preference and learning_mode.
  • Added _should_execute_policy_scan(request) helper:
    • off: advisory-only (never execute scan)
    • force: execute scan regardless of query phrasing
    • auto: existing intent-based policy execution behavior
  • Wired learning_mode into agent context calls.
• backend/app/agents/core_v2.py
  • Extended AgentContext with learning_mode: bool.
  • Prompt construction now adds analyst-teaching/explainability guidance when enabled.

Frontend Updates

• frontend/src/api/client.ts
  • Extended AssistRequest with execution_preference and learning_mode.
  • Extended AssistResponse with optional execution payload.
• frontend/src/components/AgentPanel.tsx
  • Added Execution selector (Auto, Force execute, Advisory only).
  • Added Learning mode switch.
  • Added execution results accordion (scope, datasets, top domains, hit count, elapsed).
  • Cleaned stream update logic to avoid loop-closure lint warnings.

Tests and Validation

• backend/tests/test_agent_policy_execution.py
  • Added regression tests for:
    • execution_preference=off (stays advisory)
    • execution_preference=force (executes scanner)
• Validation:
  • Backend tests: test_agent_policy_execution.py passed.
  • Frontend build: clean compile after warning cleanup.

Frontend Warning Cleanup

• frontend/src/components/AnalysisDashboard.tsx
  • Removed unused DeleteIcon import.
• frontend/src/components/MitreMatrix.tsx
  • Fixed useCallback dependency warning by including huntList.

Dev Reliability: Docker Compose Startup on PowerShell

• Problem: Intermittent docker compose up -d 2>&1 exit code 1 despite healthy/running containers.
• Root Cause: PowerShell 2>&1 handling can surface NativeCommandError for compose stderr/progress output (false failure signal).
• Solution:
  • Added scripts/dev-up.ps1 startup helper to:
    • run compose with stable output handling,
    • show container status,
    • verify backend/frontend readiness,
    • return actionable exit codes.
  • Updated backend liveness probe to http://localhost:8000/openapi.json (current app does not expose /health).