diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000..ee1004e --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,472 @@ +# Architecture Documentation + +This document describes the architecture and design decisions for VelociCompanion. + +## System Overview + +VelociCompanion is a multi-tenant, cloud-native threat hunting companion designed to work with Velociraptor. It provides secure authentication, data isolation, and role-based access control. + +``` +┌─────────────┐ ┌─────────────┐ ┌──────────────┐ +│ │ │ │ │ │ +│ Frontend │────▶│ Backend │────▶│ PostgreSQL │ +│ (React) │ │ (FastAPI) │ │ Database │ +│ │ │ │ │ │ +└─────────────┘ └─────────────┘ └──────────────┘ + │ + ▼ + ┌─────────────┐ + │ │ + │ Velociraptor│ + │ Servers │ + │ │ + └─────────────┘ +``` + +## Technology Stack + +### Backend +- **FastAPI**: Modern, fast web framework for building APIs +- **SQLAlchemy**: SQL toolkit and ORM +- **PostgreSQL**: Relational database +- **Alembic**: Database migration tool +- **Python-Jose**: JWT token handling +- **Passlib**: Password hashing with bcrypt + +### Frontend +- **React**: UI library +- **TypeScript**: Type-safe JavaScript +- **Axios**: HTTP client +- **React Router**: Client-side routing + +### Infrastructure +- **Docker**: Containerization +- **Docker Compose**: Multi-container orchestration + +## Core Components + +### 1. Authentication System + +#### JWT Token Flow +``` +1. User submits credentials (username/password) +2. Backend verifies credentials +3. Backend generates JWT token with: + - user_id (sub) + - tenant_id + - role + - expiration time +4. Frontend stores token in localStorage +5. All subsequent requests include token in Authorization header +6. Backend validates token and extracts user context +``` + +#### Password Security +- Passwords are hashed using bcrypt with automatic salt generation +- Password hashes are never exposed in API responses +- Plaintext passwords are never logged or stored + +#### Token Security +- Tokens expire after 30 minutes (configurable) +- Tokens are signed with HS256 algorithm +- Secret key must be at least 32 characters + +### 2. Multi-Tenancy + +#### Data Isolation +Every database query is automatically scoped to the user's tenant: + +```python +# Example: Listing hosts +hosts = db.query(Host).filter(Host.tenant_id == current_user.tenant_id).all() +``` + +#### Tenant Creation +- Default tenant is created automatically on first user registration +- Admin users can create additional tenants +- Users are assigned to exactly one tenant + +#### Cross-Tenant Access +- Regular users: Can only access data in their tenant +- Admin users: Can access all data in their tenant +- Super-admin (future): Could access multiple tenants + +### 3. Role-Based Access Control (RBAC) + +#### Roles +- **user**: Standard user with read/write access to their tenant's data +- **admin**: Elevated privileges within their tenant + - Can manage users in their tenant + - Can create/modify/delete resources + - Can view all data in their tenant + +#### Permission Enforcement +```python +# Endpoint requiring admin role +@router.get("/users") +async def list_users( + current_user: User = Depends(require_role(["admin"])) +): + # Only admins can access this + pass +``` + +### 4. Database Schema + +#### Core Tables + +**tenants** +- id (PK) +- name (unique) +- description +- created_at + +**users** +- id (PK) +- username (unique) +- password_hash +- role +- tenant_id (FK → tenants) +- is_active +- created_at + +**hosts** +- id (PK) +- hostname +- ip_address +- os +- tenant_id (FK → tenants) +- host_metadata (JSON) +- created_at +- last_seen + +**cases** +- id (PK) +- title +- description +- status (open, closed, investigating) +- severity (low, medium, high, critical) +- tenant_id (FK → tenants) +- created_at +- updated_at + +**artifacts** +- id (PK) +- artifact_type (hash, ip, domain, email, etc.) +- value +- description +- case_id (FK → cases) +- artifact_metadata (JSON) +- created_at + +#### Relationships +``` +tenants (1) ──< (N) users +tenants (1) ──< (N) hosts +tenants (1) ──< (N) cases +cases (1) ──< (N) artifacts +``` + +### 5. API Design + +#### RESTful Principles +- Resources are nouns (users, hosts, cases) +- HTTP methods represent actions (GET, POST, PUT, DELETE) +- Proper status codes (200, 201, 401, 403, 404) + +#### Authentication +All endpoints except `/auth/register` and `/auth/login` require authentication. + +``` +Authorization: Bearer +``` + +#### Response Format +Success: +```json +{ + "id": 1, + "username": "john", + "role": "user", + "tenant_id": 1 +} +``` + +Error: +```json +{ + "detail": "User not found" +} +``` + +### 6. Frontend Architecture + +#### Component Structure +``` +src/ +├── components/ # Reusable UI components +│ └── PrivateRoute.tsx +├── context/ # React Context providers +│ └── AuthContext.tsx +├── pages/ # Page components +│ ├── Login.tsx +│ └── Dashboard.tsx +├── utils/ # Utilities +│ └── api.ts # API client +├── App.tsx # Main app component +└── index.tsx # Entry point +``` + +#### State Management +- **AuthContext**: Global authentication state + - Current user + - Login/logout functions + - Loading state + - Authentication status + +#### Routing +``` +/login → Login page (public) +/ → Dashboard (protected) +/* → Redirect to / (protected) +``` + +### 7. Security Architecture + +#### Authentication Flow +1. Frontend sends credentials to `/api/auth/login` +2. Backend validates and returns JWT token +3. Frontend stores token in localStorage +4. Token included in all API requests +5. Backend validates token on each request + +#### Authorization Flow +1. Extract JWT from Authorization header +2. Verify token signature and expiration +3. Extract user_id from token payload +4. Load user from database +5. Check user's role for endpoint access +6. Apply tenant scoping to queries + +#### Security Headers +```python +# CORS configuration +allow_origins=["http://localhost:3000"] +allow_credentials=True +allow_methods=["*"] +allow_headers=["*"] +``` + +## Data Flow Examples + +### User Registration +``` +1. POST /api/auth/register + {username: "john", password: "pass123"} +2. Backend hashes password +3. Create default tenant if needed +4. Create user record +5. Return user object (without password_hash) +``` + +### Host Ingestion +``` +1. Velociraptor sends data to POST /api/ingestion/ingest + - Must include valid JWT token +2. Extract tenant_id from current user +3. Find or create host with hostname +4. Update host metadata +5. Return success response +``` + +### Listing Resources +``` +1. GET /api/hosts with Authorization header +2. Validate JWT token +3. Extract tenant_id from user +4. Query: SELECT * FROM hosts WHERE tenant_id = ? +5. Return filtered results +``` + +## Deployment Architecture + +### Development +``` +┌──────────────────────────────────────┐ +│ Docker Compose │ +├──────────────────────────────────────┤ +│ Frontend:3000 Backend:8000 DB:5432│ +└──────────────────────────────────────┘ +``` + +### Production (Recommended) +``` +┌─────────────┐ ┌─────────────┐ +│ Nginx/ │ │ Frontend │ +│ Traefik │────▶│ (Static) │ +│ (HTTPS) │ └─────────────┘ +└──────┬──────┘ + │ + ▼ +┌─────────────┐ ┌──────────────┐ +│ Backend │ │ PostgreSQL │ +│ (Multiple │────▶│ (Managed) │ +│ instances) │ └──────────────┘ +└─────────────┘ +``` + +## Performance Considerations + +### Database Indexing +- Primary keys on all tables +- Unique index on usernames +- Index on tenant_id columns for fast filtering +- Index on hostname for host lookups + +### Query Optimization +- Always filter by tenant_id early in queries +- Use pagination for large result sets (skip/limit) +- Lazy load relationships when not needed + +### Caching (Future) +- Cache tenant information +- Cache user profiles +- Cache frequently accessed hosts + +## Monitoring & Logging + +### Health Checks +``` +GET /health → {"status": "healthy"} +``` + +### Logging +- Request logging via Uvicorn +- Error tracking in application logs +- Database query logging (development only) + +### Metrics (Future) +- Request count per endpoint +- Authentication success/failure rate +- Database query performance +- Active user count + +## Migration Strategy + +### Database Migrations +```bash +# Create migration +alembic revision --autogenerate -m "Description" + +# Apply migration +alembic upgrade head + +# Rollback +alembic downgrade -1 +``` + +### Schema Evolution +1. Create migration for schema changes +2. Test migration in development +3. Apply to staging environment +4. Verify data integrity +5. Apply to production during maintenance window + +## Testing Strategy + +### Unit Tests (Future) +- Test individual functions +- Mock database connections +- Test password hashing +- Test JWT token creation/verification + +### Integration Tests (Future) +- Test API endpoints +- Test authentication flow +- Test multi-tenancy isolation +- Test RBAC enforcement + +### Manual Testing +- Use test_api.sh script +- Use FastAPI's /docs interface +- Test frontend authentication flow + +## Future Enhancements + +### Phase 2 +- Refresh tokens for longer sessions +- Password reset functionality +- Email verification +- Two-factor authentication (2FA) + +### Phase 3 +- Audit logging +- Advanced search and filtering +- Real-time notifications +- Velociraptor direct integration + +### Phase 4 +- Machine learning for threat detection +- Automated playbooks +- Integration with SIEM systems +- Advanced reporting and analytics + +## Troubleshooting Guide + +### Common Issues + +**Token Expired** +- Tokens expire after 30 minutes +- User must login again +- Consider implementing refresh tokens + +**Permission Denied** +- User lacks required role +- Check user's role in database +- Verify endpoint requires correct role + +**Data Not Visible** +- Check tenant_id of user +- Verify data belongs to correct tenant +- Ensure tenant_id is being applied to queries + +**Database Connection Failed** +- Check DATABASE_URL environment variable +- Verify PostgreSQL is running +- Check network connectivity + +## Development Guidelines + +### Adding New Endpoints + +1. Create route in `app/api/routes/` +2. Add authentication dependency +3. Apply tenant scoping to queries +4. Add role check if needed +5. Create Pydantic schemas +6. Update router registration in main.py +7. Test with /docs interface + +### Adding New Models + +1. Create model in `app/models/` +2. Add tenant_id foreign key +3. Create migration +4. Create Pydantic schemas +5. Create CRUD routes +6. Apply tenant scoping + +### Code Style + +- Follow PEP 8 for Python +- Use type hints +- Write docstrings for functions +- Keep functions small and focused +- Use meaningful variable names + +## References + +- [FastAPI Documentation](https://fastapi.tiangolo.com/) +- [SQLAlchemy Documentation](https://docs.sqlalchemy.org/) +- [JWT RFC](https://tools.ietf.org/html/rfc7519) +- [OAuth 2.0 RFC](https://tools.ietf.org/html/rfc6749) diff --git a/QUICKSTART.md b/QUICKSTART.md new file mode 100644 index 0000000..e8ec881 --- /dev/null +++ b/QUICKSTART.md @@ -0,0 +1,263 @@ +# Quick Start Guide + +This guide will help you get VelociCompanion up and running in minutes. + +## Prerequisites + +- Docker and Docker Compose installed +- 8GB RAM minimum +- Ports 3000, 5432, and 8000 available + +## Step 1: Start the Application + +```bash +# Clone the repository +git clone https://github.com/mblanke/ThreatHunt.git +cd ThreatHunt + +# Start all services +docker-compose up -d + +# Check service status +docker-compose ps +``` + +Expected output: +``` +NAME COMMAND SERVICE STATUS PORTS +threathunt-backend-1 "sh -c 'alembic upgr…" backend running 0.0.0.0:8000->8000/tcp +threathunt-db-1 "docker-entrypoint.s…" db running 0.0.0.0:5432->5432/tcp +threathunt-frontend-1 "docker-entrypoint.s…" frontend running 0.0.0.0:3000->3000/tcp +``` + +## Step 2: Verify Backend is Running + +```bash +# Check backend health +curl http://localhost:8000/health + +# Expected response: +# {"status":"healthy"} + +# View API documentation +open http://localhost:8000/docs +``` + +## Step 3: Access the Frontend + +Open your browser and navigate to: +``` +http://localhost:3000 +``` + +You should see the VelociCompanion login page. + +## Step 4: Create Your First User + +### Option A: Via API (using curl) + +```bash +# Register a new user +curl -X POST http://localhost:8000/api/auth/register \ + -H "Content-Type: application/json" \ + -d '{ + "username": "admin", + "password": "admin123", + "role": "admin" + }' + +# Login to get a token +curl -X POST http://localhost:8000/api/auth/login \ + -H "Content-Type: application/x-www-form-urlencoded" \ + -d "username=admin&password=admin123" +``` + +### Option B: Via Frontend + +1. The first time you access the app, you'll need to register via API first (as shown above) +2. Then login through the frontend at http://localhost:3000/login + +## Step 5: Explore the API + +Use the interactive API documentation at: +``` +http://localhost:8000/docs +``` + +Click "Authorize" and enter your token in the format: +``` +Bearer YOUR_TOKEN_HERE +``` + +## Step 6: Test the API + +Run the test script to verify all endpoints: + +```bash +./test_api.sh +``` + +Expected output: +``` +=================================== +VelociCompanion API Test Script +=================================== + +1. Testing health endpoint... +✓ Health check passed + +2. Registering a new user... +✓ User registration successful + +3. Logging in... +✓ Login successful + +4. Getting current user profile... +✓ Profile retrieval successful + +5. Listing tenants... +✓ Tenants list retrieved + +6. Listing hosts... +Hosts: [] + +7. Testing authentication protection... +✓ Authentication protection working + +=================================== +API Testing Complete! +=================================== +``` + +## Common Operations + +### Create a Host + +```bash +# Get your token from login +TOKEN="your_token_here" + +# Create a host +curl -X POST http://localhost:8000/api/hosts \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "hostname": "workstation-01", + "ip_address": "192.168.1.100", + "os": "Windows 10" + }' +``` + +### List Hosts + +```bash +curl -X GET http://localhost:8000/api/hosts \ + -H "Authorization: Bearer $TOKEN" +``` + +### Ingest Data + +```bash +curl -X POST http://localhost:8000/api/ingestion/ingest \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "hostname": "server-01", + "data": { + "artifact": "Windows.System.TaskScheduler", + "results": [...] + } + }' +``` + +## Troubleshooting + +### Database Connection Issues + +```bash +# Check if database is running +docker-compose logs db + +# Restart database +docker-compose restart db +``` + +### Backend Not Starting + +```bash +# Check backend logs +docker-compose logs backend + +# Common issues: +# - Database not ready: Wait a few seconds and check logs +# - Port 8000 in use: Stop other services using that port +``` + +### Frontend Not Loading + +```bash +# Check frontend logs +docker-compose logs frontend + +# Rebuild frontend if needed +docker-compose build frontend +docker-compose up -d frontend +``` + +### Reset Everything + +```bash +# Stop and remove all containers and volumes +docker-compose down -v + +# Start fresh +docker-compose up -d +``` + +## Next Steps + +1. **Create Additional Users**: Use the `/api/auth/register` endpoint +2. **Set Up Tenants**: Create tenants via `/api/tenants` (admin only) +3. **Integrate with Velociraptor**: Configure Velociraptor to send data to `/api/ingestion/ingest` +4. **Explore Cases**: Create and manage threat hunting cases +5. **Configure VirusTotal**: Set up VirusTotal API integration for hash lookups + +## Security Considerations + +⚠️ **Before deploying to production:** + +1. Change the `SECRET_KEY` in docker-compose.yml or .env file + - Must be at least 32 characters + - Use a cryptographically random string + +2. Use strong passwords for the database + +3. Enable HTTPS/TLS for API and frontend + +4. Configure proper firewall rules + +5. Review and update CORS settings in `backend/app/main.py` + +## Development Mode + +To run in development mode with hot reload: + +```bash +# Backend +cd backend +python -m venv venv +source venv/bin/activate +pip install -r requirements.txt +uvicorn app.main:app --reload + +# Frontend (in another terminal) +cd frontend +npm install +npm start +``` + +## Support + +- Documentation: See [README.md](README.md) +- API Docs: http://localhost:8000/docs +- Issues: GitHub Issues diff --git a/test_api.sh b/test_api.sh new file mode 100755 index 0000000..4a667ec --- /dev/null +++ b/test_api.sh @@ -0,0 +1,116 @@ +#!/bin/bash + +# Test script for VelociCompanion API +# This script demonstrates the authentication flow and basic API usage + +API_URL="http://localhost:8000" + +echo "===================================" +echo "VelociCompanion API Test Script" +echo "===================================" +echo "" + +# Colors for output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +NC='\033[0m' # No Color + +# Test 1: Health Check +echo -e "${YELLOW}1. Testing health endpoint...${NC}" +response=$(curl -s -o /dev/null -w "%{http_code}" "$API_URL/health") +if [ "$response" = "200" ]; then + echo -e "${GREEN}✓ Health check passed${NC}" +else + echo -e "${RED}✗ Health check failed (HTTP $response)${NC}" + exit 1 +fi +echo "" + +# Test 2: Register a new user +echo -e "${YELLOW}2. Registering a new user...${NC}" +register_response=$(curl -s -X POST "$API_URL/api/auth/register" \ + -H "Content-Type: application/json" \ + -d '{ + "username": "testuser", + "password": "testpass123", + "role": "user" + }') + +if echo "$register_response" | grep -q "username"; then + echo -e "${GREEN}✓ User registration successful${NC}" + echo "Response: $register_response" +else + echo -e "${YELLOW}⚠ User might already exist or registration failed${NC}" + echo "Response: $register_response" +fi +echo "" + +# Test 3: Login +echo -e "${YELLOW}3. Logging in...${NC}" +login_response=$(curl -s -X POST "$API_URL/api/auth/login" \ + -H "Content-Type: application/x-www-form-urlencoded" \ + -d "username=testuser&password=testpass123") + +TOKEN=$(echo "$login_response" | grep -o '"access_token":"[^"]*"' | cut -d'"' -f4) + +if [ -n "$TOKEN" ]; then + echo -e "${GREEN}✓ Login successful${NC}" + echo "Token: ${TOKEN:0:20}..." +else + echo -e "${RED}✗ Login failed${NC}" + echo "Response: $login_response" + exit 1 +fi +echo "" + +# Test 4: Get current user profile +echo -e "${YELLOW}4. Getting current user profile...${NC}" +profile_response=$(curl -s -X GET "$API_URL/api/auth/me" \ + -H "Authorization: Bearer $TOKEN") + +if echo "$profile_response" | grep -q "username"; then + echo -e "${GREEN}✓ Profile retrieval successful${NC}" + echo "Profile: $profile_response" +else + echo -e "${RED}✗ Profile retrieval failed${NC}" + echo "Response: $profile_response" +fi +echo "" + +# Test 5: List tenants +echo -e "${YELLOW}5. Listing tenants...${NC}" +tenants_response=$(curl -s -X GET "$API_URL/api/tenants" \ + -H "Authorization: Bearer $TOKEN") + +if echo "$tenants_response" | grep -q "name"; then + echo -e "${GREEN}✓ Tenants list retrieved${NC}" + echo "Tenants: $tenants_response" +else + echo -e "${YELLOW}⚠ No tenants found or access denied${NC}" + echo "Response: $tenants_response" +fi +echo "" + +# Test 6: List hosts +echo -e "${YELLOW}6. Listing hosts...${NC}" +hosts_response=$(curl -s -X GET "$API_URL/api/hosts" \ + -H "Authorization: Bearer $TOKEN") + +echo "Hosts: $hosts_response" +echo "" + +# Test 7: Test authentication protection (without token) +echo -e "${YELLOW}7. Testing authentication protection...${NC}" +unauth_response=$(curl -s -o /dev/null -w "%{http_code}" -X GET "$API_URL/api/hosts") + +if [ "$unauth_response" = "401" ]; then + echo -e "${GREEN}✓ Authentication protection working (got 401 as expected)${NC}" +else + echo -e "${RED}✗ Authentication not properly enforced (got HTTP $unauth_response)${NC}" +fi +echo "" + +echo "===================================" +echo -e "${GREEN}API Testing Complete!${NC}" +echo "==================================="