feat: interactive network map, IOC highlighting, AUP hunt selector, type filters

- NetworkMap: hunt-scoped force-directed graph with click-to-inspect popover - NetworkMap: zoom/pan (wheel, drag, buttons), viewport transform - NetworkMap: clickable IP/Host/Domain/URL legend chips to filter node types - NetworkMap: brighter colors, 20% smaller nodes - DatasetViewer: IOC columns highlighted with colored headers + cell tinting - AUPScanner: hunt dropdown replacing dataset checkboxes, auto-select all - Rename 'Social Media (Personal)' theme to 'Social Media' with DB migration - Fix /api/hunts timeout: Dataset.rows lazy='noload' (was selectin cascade) - Add OS column mapping to normalizer - Full backend services, DB models, alembic migrations, new routes - New components: Dashboard, HuntManager, FileUpload, NetworkMap, etc. - Docker Compose deployment with nginx reverse proxy
2026-03-01 14:00:20 -05:00 · 2026-02-19 15:41:15 -05:00
parent d0c9f88268
commit 9b98ab9614
92 changed files with 13042 additions and 1089 deletions
--- a/docs/INTEGRATION_GUIDE.md
+++ b/docs/INTEGRATION_GUIDE.md
@@ -0,0 +1,414 @@
+# ThreatHunt Analyst-Assist Agent - Integration Guide
+
+## Quick Reference
+
+### Files Created
+
+**Backend (10 files)**
+- `backend/app/agents/core.py` - ThreatHuntAgent class
+- `backend/app/agents/providers.py` - LLM provider interface
+- `backend/app/agents/config.py` - Agent configuration
+- `backend/app/agents/__init__.py` - Module initialization
+- `backend/app/api/routes/agent.py` - /api/agent/* endpoints
+- `backend/app/api/__init__.py` - API module init
+- `backend/app/main.py` - FastAPI application
+- `backend/app/__init__.py` - App module init
+- `backend/requirements.txt` - Python dependencies
+- `backend/run.py` - Development server entry point
+
+**Frontend (7 files)**
+- `frontend/src/components/AgentPanel.tsx` - React chat component
+- `frontend/src/components/AgentPanel.css` - Component styles
+- `frontend/src/utils/agentApi.ts` - API communication
+- `frontend/src/App.tsx` - Main application with agent
+- `frontend/src/App.css` - Application styles
+- `frontend/src/index.tsx` - React entry point
+- `frontend/public/index.html` - HTML template
+- `frontend/package.json` - npm dependencies
+- `frontend/tsconfig.json` - TypeScript config
+
+**Docker & Config (5 files)**
+- `Dockerfile.backend` - Backend container
+- `Dockerfile.frontend` - Frontend container
+- `docker-compose.yml` - Full stack orchestration
+- `.env.example` - Configuration template
+- `.gitignore` - Version control exclusions
+
+**Documentation (3 files)**
+- `AGENT_IMPLEMENTATION.md` - Detailed technical guide
+- `IMPLEMENTATION_SUMMARY.md` - High-level overview
+- `INTEGRATION_GUIDE.md` - This file
+
+### Provider Configuration Quick Start
+
+**Option 1: Online (OpenAI) - Easiest**
+```bash
+cp .env.example .env
+# Edit .env:
+THREAT_HUNT_AGENT_PROVIDER=online
+THREAT_HUNT_ONLINE_API_KEY=sk-your-openai-key
+THREAT_HUNT_ONLINE_MODEL=gpt-3.5-turbo
+
+docker-compose up -d
+# Access at http://localhost:3000
+```
+
+**Option 2: Local Model (Ollama) - Best for Privacy**
+```bash
+# Install Ollama and pull model
+ollama pull mistral  # or llama2, neural-chat, etc.
+
+cp .env.example .env
+# Edit .env:
+THREAT_HUNT_AGENT_PROVIDER=local
+THREAT_HUNT_LOCAL_MODEL_PATH=/path/to/model
+
+# Update docker-compose.yml to connect to Ollama
+# Add to backend service:
+# extra_hosts:
+#   - "host.docker.internal:host-gateway"
+# THREAT_HUNT_AGENT_PROVIDER=local
+# THREAT_HUNT_LOCAL_MODEL_PATH=~/.ollama/models/
+
+docker-compose up -d
+```
+
+**Option 3: Internal Service - Enterprise**
+```bash
+cp .env.example .env
+# Edit .env:
+THREAT_HUNT_AGENT_PROVIDER=networked
+THREAT_HUNT_NETWORKED_ENDPOINT=http://your-inference-service:5000
+THREAT_HUNT_NETWORKED_KEY=your-api-key
+
+docker-compose up -d
+```
+
+## Installation Steps
+
+### Prerequisites
+- Docker & Docker Compose (recommended)
+- OR Python 3.11 + Node.js 18 (local development)
+
+### Method 1: Docker (Recommended)
+
+```bash
+cd /path/to/ThreatHunt
+
+# 1. Configure provider
+cp .env.example .env
+# Edit .env and set your LLM provider
+
+# 2. Build and start
+docker-compose up -d
+
+# 3. Verify
+curl http://localhost:8000/api/agent/health
+curl http://localhost:3000
+
+# 4. Access UI
+open http://localhost:3000
+```
+
+### Method 2: Local Development
+
+**Backend**:
+```bash
+cd backend
+
+# Create virtual environment
+python -m venv venv
+source venv/bin/activate  # or venv\Scripts\activate on Windows
+
+# Install dependencies
+pip install -r requirements.txt
+
+# Set provider (choose one)
+export THREAT_HUNT_ONLINE_API_KEY=sk-your-key
+# OR
+export THREAT_HUNT_LOCAL_MODEL_PATH=/path/to/model
+# OR
+export THREAT_HUNT_NETWORKED_ENDPOINT=http://service:5000
+
+# Run server
+python run.py
+# API at http://localhost:8000/docs
+```
+
+**Frontend** (new terminal):
+```bash
+cd frontend
+
+# Install dependencies
+npm install
+
+# Start dev server
+REACT_APP_API_URL=http://localhost:8000 npm start
+# App at http://localhost:3000
+```
+
+## Testing the Agent
+
+### 1. Check Agent Health
+```bash
+curl http://localhost:8000/api/agent/health
+
+# Expected response (if configured):
+{
+  "status": "healthy",
+  "provider": "OnlineProvider",
+  "max_tokens": 1024,
+  "reasoning_enabled": true
+}
+```
+
+### 2. Test API Directly
+```bash
+curl -X POST http://localhost:8000/api/agent/assist \
+  -H "Content-Type: application/json" \
+  -d '{
+    "query": "What file modifications are suspicious?",
+    "dataset_name": "FileList",
+    "artifact_type": "FileList",
+    "host_identifier": "DESKTOP-TEST",
+    "data_summary": "System file listing from scan"
+  }'
+```
+
+### 3. Test UI
+1. Open http://localhost:3000
+2. See sample data table
+3. Click "Ask" button at bottom right
+4. Type a question in the agent panel
+5. Verify response appears with suggestions
+
+## Deployment Checklist
+
+- [ ] Configure LLM provider (env vars)
+- [ ] Test agent health endpoint
+- [ ] Test API with sample request
+- [ ] Test frontend UI
+- [ ] Configure CORS if frontend on different domain
+- [ ] Add authentication (JWT/OAuth) for production
+- [ ] Set up logging/monitoring
+- [ ] Create backups of configuration
+- [ ] Document provider credentials management
+- [ ] Set up auto-scaling (if needed)
+
+## Monitoring & Troubleshooting
+
+### Check Logs
+```bash
+# Backend logs
+docker-compose logs -f backend
+
+# Frontend logs
+docker-compose logs -f frontend
+
+# Specific error
+docker-compose logs backend | grep -i error
+```
+
+### Common Issues
+
+**503 - Agent Unavailable**
+```
+Cause: No LLM provider configured
+Fix: Set THREAT_HUNT_ONLINE_API_KEY or other provider env var
+```
+
+**CORS Error in Browser Console**
+```
+Cause: Frontend and backend on different origins
+Fix: Update REACT_APP_API_URL or add frontend domain to CORS
+```
+
+**Slow Responses**
+```
+Cause: LLM provider latency (especially online)
+Options:
+  1. Use local provider instead
+  2. Reduce MAX_TOKENS
+  3. Check network connectivity
+```
+
+**Provider Not Found**
+```
+Cause: Model path or endpoint doesn't exist
+Fix: Verify path/endpoint in .env
+     docker-compose exec backend python -c "from app.agents import get_provider; get_provider()"
+```
+
+## API Reference
+
+### POST /api/agent/assist
+
+Request guidance on artifact data.
+
+**Request Body**:
+```typescript
+{
+  query: string;                    // Analyst question
+  dataset_name?: string;            // CSV dataset name
+  artifact_type?: string;           // Artifact type
+  host_identifier?: string;         // Host/IP identifier
+  data_summary?: string;            // Context description
+  conversation_history?: Array<{    // Previous messages
+    role: string;
+    content: string;
+  }>;
+}
+```
+
+**Response**:
+```typescript
+{
+  guidance: string;                     // Advisory text
+  confidence: number;                   // 0.0 to 1.0
+  suggested_pivots: string[];           // Analysis directions
+  suggested_filters: string[];          // Data filters
+  caveats?: string;                     // Limitations
+  reasoning?: string;                   // Explanation
+}
+```
+
+**Status Codes**:
+- `200` - Success
+- `400` - Bad request
+- `503` - Service unavailable
+
+### GET /api/agent/health
+
+Check agent availability and configuration.
+
+**Response**:
+```typescript
+{
+  status: "healthy" | "unavailable" | "error";
+  provider?: string;                    // Provider class name
+  max_tokens?: number;                  // Max response length
+  reasoning_enabled?: boolean;
+  configured_providers?: {              // If unavailable
+    local: boolean;
+    networked: boolean;
+    online: boolean;
+  };
+}
+```
+
+## Security Notes
+
+### For Production
+
+1. **Authentication**: Add JWT token validation to endpoints
+   ```python
+   from fastapi.security import HTTPBearer
+   security = HTTPBearer()
+   
+   @router.post("/assist")
+   async def assist(request: AssistRequest, credentials: HTTPAuthorizationCredentials = Depends(security)):
+       # Verify token
+   ```
+
+2. **Rate Limiting**: Install and use `slowapi`
+   ```python
+   from slowapi import Limiter
+   limiter = Limiter(key_func=get_remote_address)
+   
+   @limiter.limit("10/minute")
+   async def assist(request: AssistRequest):
+   ```
+
+3. **HTTPS**: Use reverse proxy (nginx) with TLS
+
+4. **Data Filtering**: Filter sensitive data before LLM
+   ```python
+   # Remove IPs, usernames, hashes
+   filtered = filter_sensitive(request.data_summary)
+   ```
+
+5. **Audit Logging**: Log all agent requests
+   ```python
+   logger.info(f"Agent: user={user_id} query={query} host={host}")
+   ```
+
+## Configuration Reference
+
+**Agent Settings**:
+```bash
+THREAT_HUNT_AGENT_PROVIDER          # auto, local, networked, online
+THREAT_HUNT_AGENT_MAX_TOKENS        # Default: 1024
+THREAT_HUNT_AGENT_REASONING         # Default: true
+THREAT_HUNT_AGENT_HISTORY_LENGTH    # Default: 10
+THREAT_HUNT_AGENT_FILTER_SENSITIVE  # Default: true
+```
+
+**Provider: Local**:
+```bash
+THREAT_HUNT_LOCAL_MODEL_PATH        # Path to .gguf or other model
+```
+
+**Provider: Networked**:
+```bash
+THREAT_HUNT_NETWORKED_ENDPOINT      # http://service:5000
+THREAT_HUNT_NETWORKED_KEY           # API key for service
+```
+
+**Provider: Online**:
+```bash
+THREAT_HUNT_ONLINE_API_KEY          # Provider API key
+THREAT_HUNT_ONLINE_PROVIDER         # openai, anthropic, google, etc
+THREAT_HUNT_ONLINE_MODEL            # Model name (gpt-3.5-turbo, etc)
+```
+
+## Architecture Decisions
+
+### Why Pluggable Providers?
+- Deployment flexibility (cloud, on-prem, hybrid)
+- Privacy control (local vs online)
+- Cost optimization
+- Vendor lock-in prevention
+
+### Why Conversation History?
+- Better context for follow-up questions
+- Maintains thread of investigation
+- Reduces redundant explanations
+
+### Why Read-Only?
+- Safety: Agent cannot accidentally modify data
+- Compliance: Adheres to governance requirements
+- Trust: Humans retain control
+
+### Why Config-Based?
+- No code changes for provider switching
+- Easy environment-specific configuration
+- CI/CD friendly
+
+## Next Steps
+
+1. **Configure Provider**: Set env vars for your chosen LLM
+2. **Deploy**: Use docker-compose or local development
+3. **Test**: Verify health endpoint and sample request
+4. **Integrate**: Add to your threat hunting workflow
+5. **Monitor**: Track agent usage and quality
+6. **Iterate**: Gather analyst feedback and improve
+
+## Support & Troubleshooting
+
+See `AGENT_IMPLEMENTATION.md` for detailed troubleshooting.
+
+Key support files:
+- Backend logs: `docker-compose logs backend`
+- Frontend console: Browser DevTools
+- Health check: `curl http://localhost:8000/api/agent/health`
+- API docs: http://localhost:8000/docs (when running)
+
+## References
+
+- **Governance**: See `goose-core/governance/AGENT_POLICY.md`
+- **Intent**: See `THREATHUNT_INTENT.md`
+- **Technical**: See `AGENT_IMPLEMENTATION.md`
+- **FastAPI**: https://fastapi.tiangolo.com
+- **React**: https://react.dev
+- **Docker**: https://docs.docker.com
+