mirror of
https://github.com/mblanke/Gov_Travel_App.git
synced 2026-03-01 14:10:22 -05:00
Add Python web scraper for NJC travel rates with currency extraction
- Implemented Python scraper using BeautifulSoup and pandas to automatically collect travel rates from official NJC website - Added currency extraction from table titles (supports EUR, USD, AUD, CAD, ARS, etc.) - Added country extraction from table titles for international rates - Flatten pandas MultiIndex columns for cleaner data structure - Default to CAD for domestic Canadian sources (accommodations and domestic tables) - Created SQLite database schema (raw_tables, rate_entries, exchange_rates, accommodations) - Successfully scraped 92 tables with 17,205 rate entries covering 25 international cities - Added migration script to convert scraped data to Node.js database format - Updated .gitignore for Python files (.venv/, __pycache__, *.pyc, *.sqlite3) - Fixed city validation and currency conversion in main app - Added comprehensive debug and verification scripts This replaces manual JSON maintenance with automated data collection from official government source.
This commit is contained in:
28
Dockerfile
Normal file
28
Dockerfile
Normal file
@@ -0,0 +1,28 @@
|
||||
# Use Node.js LTS version (slim Debian-based image for better compatibility)
|
||||
FROM node:18-slim
|
||||
|
||||
# Install SQLite
|
||||
RUN apt-get update && apt-get install -y \
|
||||
sqlite3 \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Copy package files
|
||||
COPY package*.json ./
|
||||
|
||||
# Install dependencies
|
||||
RUN npm install --production
|
||||
|
||||
# Copy application files
|
||||
COPY . .
|
||||
|
||||
# Run complete database migration on container start
|
||||
RUN mkdir -p database && \
|
||||
node scripts/migrateCompleteTravelRates.js || echo "Migration will run on first request"
|
||||
|
||||
# Expose port 5001
|
||||
EXPOSE 5001
|
||||
|
||||
# Start the server
|
||||
CMD ["node", "server.js"]
|
||||
Reference in New Issue
Block a user