mirror of
https://github.com/mblanke/ThreatHunt.git
synced 2026-03-01 14:00:20 -05:00
version 0.4.0
This commit is contained in:
416
docs/update.md
416
docs/update.md
@@ -1,8 +1,10 @@
|
||||
# ThreatHunt — Session Update (February 19, 2026)
|
||||
# ThreatHunt — Session Update (February 19–20, 2026)
|
||||
|
||||
**Version: 0.4.0**
|
||||
|
||||
## Summary
|
||||
|
||||
This session covered 6 feature improvements and bug fixes across the ThreatHunt platform, all deployed via Docker Compose.
|
||||
This document covers feature improvements and bug fixes across the ThreatHunt platform.
|
||||
|
||||
---
|
||||
|
||||
@@ -134,3 +136,413 @@ Committed and pushed to GitHub (`main` branch):
|
||||
92 files changed, 13050 insertions(+), 1097 deletions(-)
|
||||
d0c9f88..9b98ab9 main -> main
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. Network Picture — Deduplicated Host Inventory (February 20, 2026)
|
||||
|
||||
**Request:** "While files are being dropped in, give me a clean clear network picture — like netstat. No 100 iterations of the same computer. Clean pic: hostname, IP, and any users logged into that machine."
|
||||
|
||||
### What It Does
|
||||
|
||||
New **Net Picture** page that scans all datasets in a hunt and produces a **one-row-per-host** inventory table. If a host appears in 900 netstat rows, it shows once — with all unique IPs, usernames, OS versions, MACs, and ports aggregated via server-side set deduplication. Supports networks with up to 1000 hosts; no artificial caps on unique values per host.
|
||||
|
||||
### Backend
|
||||
|
||||
**New service** — `backend/app/services/network_inventory.py`:
|
||||
- `build_network_picture(db, hunt_id)` streams all dataset rows in batches of 1000
|
||||
- Groups rows by `hostname` (case-insensitive), falls back to `src_ip`/`ip_address` if no hostname column
|
||||
- Per host, aggregates into Python `set()`s: IPs, users, OS, MAC addresses, protocols, open ports, remote targets, dataset sources
|
||||
- Tracks `connection_count` (total rows), `first_seen`/`last_seen` from timestamp columns
|
||||
- Returns sorted by connection count descending
|
||||
|
||||
**New API route** — `backend/app/api/routes/network.py`:
|
||||
- `GET /api/network/picture?hunt_id={id}` → `NetworkPictureResponse`
|
||||
- Response: `hosts[]` (each with hostname, ips, users, os, mac_addresses, protocols, open_ports, remote_targets, datasets, connection_count, first_seen, last_seen) + `summary` (total_hosts, total_connections, total_unique_ips, datasets_scanned)
|
||||
|
||||
**Normalizer additions** — `backend/app/services/normalizer.py`:
|
||||
- Added `mac_address` canonical mapping: `mac`, `mac_address`, `physical_address`, `mac_addr`, `hw_addr`, `ethernet`
|
||||
- Added `connection_state` canonical mapping: `state`, `status`, `tcp_state`, `conn_state`
|
||||
|
||||
### Frontend
|
||||
|
||||
**New component** — `frontend/src/components/NetworkPicture.tsx`:
|
||||
- Hunt selector dropdown → loads network picture from backend
|
||||
- MUI Table with sortable columns: Hostname, IPs, Users, OS, MAC, Connections, Ports
|
||||
- **ChipList** sub-component: shows first 5 values inline as coloured Chips; "+N more" badge expands to show all (no data hidden, just visual overflow control)
|
||||
- Click any row → expand panel showing: remote targets, all open ports, protocols, MAC addresses, dataset sources, time range
|
||||
- Search bar filters by hostname, IP, user, OS, or MAC
|
||||
- Summary stat cards: total hosts, unique IPs, connections, datasets scanned
|
||||
- Colour palette matches Network Map: IP blue, User green, OS amber, MAC purple, Port rose
|
||||
|
||||
**App integration** — `frontend/src/App.tsx`:
|
||||
- New nav item: **Net Picture** with `DevicesIcon`, route `/netpicture`
|
||||
- Import and route for `NetworkPicture` component
|
||||
|
||||
**API client** — `frontend/src/api/client.ts`:
|
||||
- Added `HostEntry`, `PictureSummary`, `NetworkPictureResponse` interfaces
|
||||
- Added `network.picture(huntId)` method
|
||||
|
||||
### Files Modified
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `backend/app/services/normalizer.py` | Added `mac_address` + `connection_state` mappings |
|
||||
| `backend/app/services/network_inventory.py` | **New** — host aggregation service |
|
||||
| `backend/app/api/routes/network.py` | **New** — `/api/network/picture` endpoint |
|
||||
| `backend/app/main.py` | Registered network router |
|
||||
| `frontend/src/api/client.ts` | Added network picture types + API method |
|
||||
| `frontend/src/components/NetworkPicture.tsx` | **New** — host inventory table component |
|
||||
| `frontend/src/App.tsx` | Added Nav item + route for Net Picture |
|
||||
|
||||
---
|
||||
|
||||
## 10. Test Data — Velociraptor Mock Network CSVs (February 20, 2026)
|
||||
|
||||
**Request:** "Create 10-15 different CSV files you'd expect from Velociraptor for a mock network with 50-100 hosts and random network traffic with a few keywords that would trigger the AUP."
|
||||
|
||||
### What Was Created
|
||||
|
||||
12 realistic Velociraptor-style CSV files in `backend/tests/test_csvs/`, generated by a Python script (`generate_test_csvs.py`) with deterministic seed for reproducibility.
|
||||
|
||||
**Mock Network:**
|
||||
- **82 hosts**: 75 workstations (IT-WS, HR-WS, FIN-WS, SLS-WS, ENG-WS, LEG-WS, MKT-WS, EXEC-WS) + 7 servers (DC-01, DC-02, FILE-01, EXCH-01, WEB-01, SQL-01, PROXY-01)
|
||||
- **14 users**: 10 named users (jsmith, agarcia, bwilson, etc.) + 4 service accounts
|
||||
- **3 subnets**: `10.10.1.x`, `10.10.2.x`, `10.10.3.x`
|
||||
- **Domain**: `acme.local`
|
||||
- **Time range**: Feb 10–20, 2026
|
||||
|
||||
### Files
|
||||
|
||||
| # | File | Rows | Velociraptor Artifact |
|
||||
|---|------|------|----------------------|
|
||||
| 1 | `01_netstat_connections.csv` | 2,012 | `Windows.Network.Netstat` |
|
||||
| 2 | `02_dns_queries.csv` | 2,964 | `Windows.Network.DNS` |
|
||||
| 3 | `03_process_listing.csv` | 1,896 | `Windows.System.Pslist` |
|
||||
| 4 | `04_network_interfaces.csv` | 94 | `Windows.Network.Interfaces` |
|
||||
| 5 | `05_logged_in_users.csv` | 171 | `Windows.Sys.Users` |
|
||||
| 6 | `06_scheduled_tasks.csv` | 410 | `Windows.System.TaskScheduler` |
|
||||
| 7 | `07_browser_history.csv` | 1,586 | `Windows.Application.Chrome.History` |
|
||||
| 8 | `08_sysmon_network.csv` | 2,517 | Sysmon Event ID 3 |
|
||||
| 9 | `09_autoruns.csv` | 342 | `Windows.Sys.AutoRuns` |
|
||||
| 10 | `10_logon_events.csv` | 1,604 | Windows Security (4624/4625) |
|
||||
| 11 | `11_proxy_logs.csv` | 2,605 | Web proxy / filter |
|
||||
| 12 | `12_file_listing.csv` | 421 | `Windows.Search.FileFinder` |
|
||||
|
||||
### AUP Triggers Embedded
|
||||
|
||||
| Category | Examples |
|
||||
|----------|----------|
|
||||
| Gambling | bet365.com, pokerstars.com, draftkings.com |
|
||||
| Gaming | steam.exe, discord.exe, steamcommunity.com, epicgameslauncher.exe |
|
||||
| Streaming | netflix.com, hulu.com, spotify.exe, open.spotify.com |
|
||||
| Downloads / Piracy | thepiratebay.org, 1337x.to, utorrent.exe, qbittorrent.exe, free_movie_2026.torrent |
|
||||
| Adult Content | pornhub.com, onlyfans.com, xvideos.com |
|
||||
| Social Media | facebook.com, tiktok.com, reddit.com |
|
||||
| Job Search | indeed.com, glassdoor.com, linkedin.com/jobs |
|
||||
| Shopping | amazon.com, ebay.com, shein.com |
|
||||
|
||||
AUP keywords appear in: DNS queries (~12%), browser history (~15%), proxy logs (~10%), process listing (~15% of hosts), autoruns (~20% of workstations), sysmon network (~10%), file listing (crack_photoshop.exe, keygen_v2.exe, free_movie_2026.torrent).
|
||||
|
||||
### Files Added
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `backend/tests/generate_test_csvs.py` | **New** — deterministic CSV generator script |
|
||||
| `backend/tests/test_csvs/*.csv` (×12) | **New** — mock Velociraptor test data |
|
||||
|
||||
---
|
||||
|
||||
## 11. Phase 8 — Analyzer Framework & Alerts (February 20, 2026)
|
||||
|
||||
### What It Does
|
||||
|
||||
Automated alerting engine with 6 built-in analyzers that scan dataset rows for suspicious activity and produce scored, MITRE-tagged alerts. Full alert lifecycle (new → acknowledged → in-progress → resolved / false-positive), bulk operations, and configurable alert rules.
|
||||
|
||||
### Backend
|
||||
|
||||
**New service** — `backend/app/services/analyzers.py` (~350 lines):
|
||||
- `BaseAnalyzer` ABC with `name`, `description`, and `async analyze(rows)` method
|
||||
- 6 built-in analyzers:
|
||||
- **EntropyAnalyzer** — Shannon entropy on command_line/url/path fields, threshold 4.5, maps to T1027 (Obfuscated Files or Information)
|
||||
- **SuspiciousCommandAnalyzer** — 19 regex patterns (mimikatz, encoded PowerShell, schtasks, psexec, vssadmin, etc.) with per-pattern MITRE mappings
|
||||
- **NetworkAnomalyAnalyzer** — Beaconing detection (dst IP frequency), suspicious ports (4444, 5555, etc.), large transfers (>10 MB), maps to T1071/T1048/T1571
|
||||
- **FrequencyAnomalyAnalyzer** — Flags values <1% occurrence in process_name/username/event_type fields
|
||||
- **AuthAnomalyAnalyzer** — Brute force (>5 failed logins per user), unusual logon types (3=network, 10=RDP), maps to T1110/T1021
|
||||
- **PersistenceAnalyzer** — Registry Run keys, services, Winlogon, IFEO patterns, maps to T1547/T1543/T1546
|
||||
- `run_all_analyzers(rows, analyzers?)` — runs all or selected analyzers, returns sorted `AlertCandidate` list
|
||||
|
||||
**New API route** — `backend/app/api/routes/alerts.py` (~300 lines):
|
||||
- `GET /api/alerts` — list with filters (status, severity, analyzer, hunt_id, dataset_id)
|
||||
- `GET /api/alerts/stats` — severity/status/analyzer/MITRE breakdowns
|
||||
- `GET /api/alerts/{id}`, `PUT /api/alerts/{id}`, `DELETE /api/alerts/{id}`
|
||||
- `POST /api/alerts/bulk-update` — bulk status changes on selected alert IDs
|
||||
- `GET /api/alerts/analyzers/list` — available analyzer metadata
|
||||
- `POST /api/alerts/analyze` — run analyzers on a dataset/hunt, auto-creates Alert records
|
||||
- Alert Rules CRUD: `GET /api/alerts/rules/list`, `POST /api/alerts/rules`, `PUT /api/alerts/rules/{id}`, `DELETE /api/alerts/rules/{id}`
|
||||
|
||||
**New ORM models** — `backend/app/db/models.py`:
|
||||
- `Alert` — 18 fields (id, title, description, severity, status, analyzer, score, evidence JSON, mitre_technique, tags JSON, hunt_id FK, dataset_id FK, case_id FK, assignee, acknowledged_at, resolved_at, timestamps; indexes on severity/status/hunt/dataset)
|
||||
- `AlertRule` — 10 fields (id, name, description, analyzer, config JSON, severity_override, enabled, hunt_id FK, timestamps; index on analyzer)
|
||||
|
||||
**Alembic migration** — `backend/alembic/versions/b4c2d3e5f6a7_add_alerts_and_alert_rules.py`
|
||||
|
||||
### Frontend
|
||||
|
||||
**New component** — `frontend/src/components/AlertPanel.tsx` (~350 lines):
|
||||
- **Alerts tab**: MUI DataGrid with severity/status chips, score, MITRE technique; checkbox selection for bulk actions (acknowledge / resolve / false-positive); click → detail dialog with evidence JSON viewer
|
||||
- **Stats tab**: Cards for total, per-severity counts, status/analyzer breakdowns, top MITRE techniques
|
||||
- **Rules tab**: Rule cards with enable/disable switch, delete; create rule dialog with analyzer selector
|
||||
- Selector bar: Hunt + Dataset pickers, "Run Analyzers" button, severity/status filters
|
||||
|
||||
**API client** — `frontend/src/api/client.ts`:
|
||||
- Added `AlertData`, `AlertStats`, `AlertRuleData`, `AnalyzerInfo`, `AnalyzeResult` interfaces
|
||||
- Added `alerts` namespace with full CRUD + analyze + rules methods
|
||||
|
||||
**App integration** — `frontend/src/App.tsx`:
|
||||
- New nav item: **Alerts** with `NotificationsActiveIcon`, route `/alerts`
|
||||
|
||||
### Files
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `backend/app/services/analyzers.py` | **New** — 6 analyzers + runner |
|
||||
| `backend/app/api/routes/alerts.py` | **New** — alerts API (CRUD, stats, bulk, rules) |
|
||||
| `backend/app/db/models.py` | Added `Alert` + `AlertRule` models |
|
||||
| `backend/alembic/versions/b4c2d3e5f6a7_...py` | **New** — alerts migration |
|
||||
| `frontend/src/components/AlertPanel.tsx` | **New** — alerts dashboard |
|
||||
| `frontend/src/api/client.ts` | Added alert types + `alerts` namespace |
|
||||
| `frontend/src/App.tsx` | Added Alerts nav + route |
|
||||
|
||||
---
|
||||
|
||||
## 12. Phase 9 — Investigation Notebooks & Playbooks (February 20, 2026)
|
||||
|
||||
### What It Does
|
||||
|
||||
Cell-based investigation notebooks (markdown / query / code) for documenting hunts, plus a playbook engine with 4 built-in response templates and step-by-step execution tracking.
|
||||
|
||||
### Backend
|
||||
|
||||
**New service** — `backend/app/services/playbook.py` (~250 lines):
|
||||
- `NotebookCell` dataclass + `validate_notebook_cells()` helper
|
||||
- 4 built-in playbook templates:
|
||||
- **Suspicious Process Investigation** (6 steps): identify → process tree → network → analyzers → MITRE → document
|
||||
- **Lateral Movement Hunt** (5 steps): remote tools → auth anomaly → network anomaly → knowledge graph → escalate
|
||||
- **Data Exfiltration Check** (5 steps): large transfers → DNS → timeline → correlate → MITRE + document
|
||||
- **Ransomware Triage** (5 steps): indicators search → all analyzers → persistence → LLM deep → critical case
|
||||
- Each step: order, title, description, action, action_config, expected_outcome
|
||||
|
||||
**New API route** — `backend/app/api/routes/notebooks.py` (~280 lines):
|
||||
- Notebook CRUD: `GET /api/notebooks`, `GET /api/notebooks/{id}`, `POST /api/notebooks`, `PUT /api/notebooks/{id}`, `DELETE /api/notebooks/{id}`
|
||||
- Cell operations: `POST /api/notebooks/{id}/cells/upsert`, `DELETE /api/notebooks/{id}/cells/{cell_id}`
|
||||
- Playbook endpoints: `GET /api/notebooks/playbooks/templates`, `GET /api/notebooks/playbooks/templates/{name}`, `POST /api/notebooks/playbooks/start`, `GET /api/notebooks/playbooks/runs`, `GET /api/notebooks/playbooks/runs/{id}`, `POST /api/notebooks/playbooks/runs/{id}/complete-step`, `POST /api/notebooks/playbooks/runs/{id}/abort`
|
||||
|
||||
**New ORM models** — `backend/app/db/models.py`:
|
||||
- `Notebook` — 10 fields (id, title, description, cells JSON, hunt_id FK, case_id FK, owner_id FK, tags JSON, timestamps; index on hunt_id)
|
||||
- `PlaybookRun` — 12 fields (id, playbook_name, status, current_step, total_steps, step_results JSON, hunt_id FK, case_id FK, started_by, timestamps, completed_at; indexes on hunt_id/status)
|
||||
|
||||
**Alembic migration** — `backend/alembic/versions/c5d3e4f6a7b8_add_notebooks_and_playbook_runs.py`
|
||||
|
||||
### Frontend
|
||||
|
||||
**New component** — `frontend/src/components/InvestigationNotebook.tsx` (~280 lines):
|
||||
- List view: Grid of notebook cards with title, description, cell count, tags, open/delete
|
||||
- Detail view: Cell-based editor with markdown/query/code cell types
|
||||
- Markdown cells render via ReactMarkdown + remarkGfm; code/query cells as monospace pre blocks
|
||||
- Edit mode: multiline TextField with Ctrl+S save shortcut
|
||||
- Cell operations: add (markdown/query/code), edit, save, delete, move up/down
|
||||
|
||||
**New component** — `frontend/src/components/PlaybookManager.tsx` (~280 lines):
|
||||
- **Templates tab**: Grid of template cards with category chips, tags, step count; view detail (MUI Stepper showing all steps) or start new run
|
||||
- **Runs tab**: List of active/completed runs with progress bars
|
||||
- Active run view: Vertical MUI Stepper with step completion, notes field, skip/abort, LinearProgress
|
||||
|
||||
**API client** — `frontend/src/api/client.ts`:
|
||||
- Added `NotebookCell`, `NotebookData`, `PlaybookTemplate`, `PlaybookStep`, `PlaybookTemplateDetail`, `PlaybookRunData` interfaces
|
||||
- Added `notebooks` namespace (CRUD + cell ops) and `playbooks` namespace (templates, runs, step-complete, abort)
|
||||
|
||||
**App integration** — `frontend/src/App.tsx`:
|
||||
- New nav items: **Notebooks** (`MenuBookIcon`, `/notebooks`) and **Playbooks** (`PlaylistPlayIcon`, `/playbooks`)
|
||||
|
||||
### Files
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `backend/app/services/playbook.py` | **New** — 4 playbook templates + cell validation |
|
||||
| `backend/app/api/routes/notebooks.py` | **New** — notebooks + playbooks API |
|
||||
| `backend/app/db/models.py` | Added `Notebook` + `PlaybookRun` models |
|
||||
| `backend/alembic/versions/c5d3e4f6a7b8_...py` | **New** — notebooks migration |
|
||||
| `frontend/src/components/InvestigationNotebook.tsx` | **New** — cell-based notebook editor |
|
||||
| `frontend/src/components/PlaybookManager.tsx` | **New** — playbook template browser + run stepper |
|
||||
| `frontend/src/api/client.ts` | Added notebook + playbook types + namespaces |
|
||||
| `frontend/src/App.tsx` | Added Notebooks + Playbooks nav + routes |
|
||||
|
||||
---
|
||||
|
||||
## 13. Docker Build Fixes (February 20, 2026)
|
||||
|
||||
Several issues were resolved to get the full 22-page frontend and updated backend deployed in Docker Compose.
|
||||
|
||||
### npm / TypeScript Fixes
|
||||
|
||||
| Issue | Fix |
|
||||
|-------|-----|
|
||||
| `npm ci` failing: "Missing: yaml@2.8.2 from lock file" | Installed `yaml@2` — `postcss-load-config` (from tailwindcss) required `^2.4.2` but only `1.10.2` was present |
|
||||
| `GridRowSelectionModel` cast to `string[]` | MUI X DataGrid v8 changed to `{ type, ids: Set }` — fixed with `Array.from(model.ids)` |
|
||||
| Recharts `formatter` type mismatch | Changed explicit `number`/`string` params to `any` in `Dashboard.tsx` |
|
||||
| Missing declarations for `cytoscape-dagre` / `cytoscape-cola` | Created `frontend/src/declarations.d.ts` |
|
||||
| Missing `HuntOut` type export | Added `export type HuntOut = Hunt` alias in `client.ts` |
|
||||
| `cytoscape.Stylesheet` no longer exists | Renamed to `cytoscape.StylesheetStyle` in ProcessTree, StorylineGraph, KnowledgeGraph |
|
||||
|
||||
### Backend Startup Fixes
|
||||
|
||||
| Issue | Fix |
|
||||
|-------|-----|
|
||||
| `init_db()` crash: "table cases already exists" | Rewrote to inspect existing tables and only create missing ones |
|
||||
| Alembic migration replay on startup | DB had all tables (from `init_db()`) but Alembic was stamped at `98ab619418bc` — ran `alembic stamp head` to sync to `c5d3e4f6a7b8` |
|
||||
|
||||
### Files Modified
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `frontend/package.json` / `package-lock.json` | Added `yaml@2` dependency |
|
||||
| `frontend/src/declarations.d.ts` | **New** — ambient module declarations |
|
||||
| `frontend/src/api/client.ts` | Added `HuntOut` type alias |
|
||||
| `frontend/src/components/AlertPanel.tsx` | Fixed `GridRowSelectionModel` handling |
|
||||
| `frontend/src/components/Dashboard.tsx` | Fixed Recharts formatter types |
|
||||
| `frontend/src/components/ProcessTree.tsx` | `Stylesheet` → `StylesheetStyle` |
|
||||
| `frontend/src/components/StorylineGraph.tsx` | `Stylesheet` → `StylesheetStyle` |
|
||||
| `frontend/src/components/KnowledgeGraph.tsx` | `Stylesheet` → `StylesheetStyle` |
|
||||
| `backend/app/db/engine.py` | Safe `init_db()` with inspector |
|
||||
|
||||
---
|
||||
|
||||
## Current Navigation (22 pages)
|
||||
|
||||
Dashboard · Hunts · Datasets · Upload · Agent · Analysis · Annotations · Hypotheses · Correlation · Network Map · Net Picture · Proc Tree · Storyline · Timeline · Search · MITRE Map · Knowledge · Cases · **Alerts** · **Notebooks** · **Playbooks** · AUP Scanner
|
||||
|
||||
## Deployment
|
||||
|
||||
```
|
||||
docker compose build
|
||||
docker compose up -d
|
||||
# Backend: http://localhost:8000 (healthy)
|
||||
# Frontend: http://localhost:3000 (200 OK)
|
||||
```
|
||||
|
||||
## Alembic Migration Chain
|
||||
|
||||
```
|
||||
9790f482da06 (initial schema)
|
||||
↓
|
||||
98ab619418bc (keyword themes + keywords)
|
||||
↓
|
||||
a3b1c2d4e5f6 (cases + activity logs)
|
||||
↓
|
||||
b4c2d3e5f6a7 (alerts + alert rules)
|
||||
↓
|
||||
c5d3e4f6a7b8 (notebooks + playbook runs) ← HEAD
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 14. Process Tree — HTTP 500 Fix (February 20, 2026)
|
||||
|
||||
**Request:** "process tree: HTTP 500"
|
||||
|
||||
**Root Cause:** `MissingGreenlet` error in `_fetch_rows` — async SQLAlchemy lazy-loading the `dataset` relationship on `DatasetRow` outside a greenlet context.
|
||||
|
||||
**Fix (`backend/app/services/process_tree.py`):**
|
||||
- Added `selectinload(DatasetRow.dataset)` to the `_fetch_rows` query so the relationship is eagerly loaded within the async session
|
||||
- Endpoint now returns data correctly (verified: 3,908 processes for full hunt)
|
||||
|
||||
---
|
||||
|
||||
## 15. Process Tree — UX Rewrite (February 20, 2026)
|
||||
|
||||
**Request:** "proctree view is horrible… there are 3908 processes and I can't see / zoom in to see and when I do click on one the resolution changes to show the data on the right. Ideally I'd like another dropdown to pick a host."
|
||||
|
||||
**Complete rewrite of `frontend/src/components/ProcessTree.tsx` (~520 lines):**
|
||||
|
||||
- **Host dropdown**: Autocomplete populated by extracting unique hostnames from tree data (82 hosts)
|
||||
- **Server-side hostname filtering**: API `?hostname=` param reduces returned data per view
|
||||
- **Grid layout fallback**: When edges < 10% of nodes (e.g. test data with no parent-child relationships), uses grid layout instead of dagre
|
||||
- **Overlay detail panel**: Absolute-positioned panel on click — no longer reflows the graph
|
||||
- **ResizeObserver**: Keeps Cytoscape canvas in sync with container size changes
|
||||
- **Search/highlight**: Filter processes by name or PID
|
||||
- **Zoom controls**: Fit, zoom-in, zoom-out buttons
|
||||
|
||||
---
|
||||
|
||||
## 16. Process Tree — White Screen Crash Fix (February 20, 2026)
|
||||
|
||||
**Request:** "I picked a different host and the screen went white"
|
||||
|
||||
**Root Cause:** Cytoscape's `destroy()` removed `<canvas>` elements that were children of the same DOM node React was managing, causing `NotFoundError: Failed to execute 'removeChild' on 'Node'` when React tried to reconcile.
|
||||
|
||||
**Fix (`frontend/src/components/ProcessTree.tsx`):**
|
||||
- Separated Cytoscape container into a plain `<div>` with zero React children
|
||||
- Loading spinner and placeholder text are now sibling overlays (not children of the Cytoscape div)
|
||||
- Added explicit `cyRef.current = null` after `destroy()`
|
||||
- Cleanup on unmount prevents stale references
|
||||
- Verified: switching between hosts (DC-01 → EXEC-WS-039 → ENG-WS-052) works with 0 console errors
|
||||
|
||||
---
|
||||
|
||||
## 17. LLM Analysis — Response Parsing Fix (February 20, 2026)
|
||||
|
||||
**Request:** "llm analysis: HTTP 500 — searching for evidence of adult site access"
|
||||
|
||||
**Root Cause Chain:**
|
||||
|
||||
1. **`AttributeError: 'dict' object has no attribute 'strip'`** — `OllamaProvider.generate()` returns a dict (`{"response": "<llm text>", "model": ..., ...}`), but `_parse_analysis` expected a string.
|
||||
|
||||
2. **Wrong dict key extraction** — Initial dict-handling fix looked for `raw.get("analysis")` but Ollama's `/api/generate` endpoint puts the LLM text in the `"response"` key.
|
||||
|
||||
3. **JSON parsing failures** — The 70B model sometimes produces slightly malformed JSON (trailing commas, etc.) that `json.loads` rejects.
|
||||
|
||||
**Fixes (`backend/app/services/llm_analysis.py`):**
|
||||
|
||||
| Change | Detail |
|
||||
|--------|--------|
|
||||
| Dict response extraction | `raw.get("response")` instead of `raw.get("analysis")` — correctly extracts LLM text from Ollama API dict |
|
||||
| Robust JSON parsing | New `_extract_json_candidates()` generator: tries full text, then outermost `{…}` block, then trailing-comma-fixed variant |
|
||||
| Reduced prompt size | `max_sample` 50 → 20 rows, `max_chars` 8000 → 6000 |
|
||||
| Reduced row limit | `/llm-analyze` endpoint loads 2000 rows (was 5000) |
|
||||
| Timeout | `asyncio.wait_for(..., timeout=300)` wraps the LLM call — returns graceful "timed out" message instead of hanging |
|
||||
| Debug logging | `_parse_analysis` logs extraction source, text length, parse success/failure with error details |
|
||||
|
||||
**Results:**
|
||||
- Quick mode (llama3.1:latest on roadrunner): 18s, confidence 0.85, risk score 65, 3 key findings — all structured fields populated
|
||||
- Deep mode (llama3.1:70b on wile): 128–185s, full analysis returned — JSON parsing succeeds when model produces valid JSON, falls back to plain text otherwise
|
||||
|
||||
---
|
||||
|
||||
## 18. Version Bump — 0.3.0 → 0.4.0 (February 20, 2026)
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `backend/app/config.py` | `APP_VERSION` 0.3.0 → 0.4.0 |
|
||||
| `frontend/package.json` | `version` 0.1.0 → 0.4.0 |
|
||||
|
||||
---
|
||||
|
||||
## Files Modified This Session (Sections 14–18)
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `backend/app/services/process_tree.py` | Added `selectinload(DatasetRow.dataset)` to `_fetch_rows` |
|
||||
| `frontend/src/components/ProcessTree.tsx` | Complete rewrite: host dropdown, grid layout, overlay panel, Cytoscape DOM separation |
|
||||
| `backend/app/services/llm_analysis.py` | Fixed dict response extraction, robust JSON parsing, reduced prompt, added timeout |
|
||||
| `backend/app/api/routes/analysis.py` | Row limit 5000 → 2000 |
|
||||
| `backend/app/config.py` | Version 0.3.0 → 0.4.0 |
|
||||
| `frontend/package.json` | Version 0.1.0 → 0.4.0 |
|
||||
|
||||
## Deployment
|
||||
|
||||
```
|
||||
docker compose up -d --build backend
|
||||
# Backend: http://localhost:8000 (healthy, v0.4.0)
|
||||
# Frontend: http://localhost:3000 (200 OK)
|
||||
```
|
||||
Reference in New Issue
Block a user