- Implemented Python scraper using BeautifulSoup and pandas to automatically collect travel rates from official NJC website - Added currency extraction from table titles (supports EUR, USD, AUD, CAD, ARS, etc.) - Added country extraction from table titles for international rates - Flatten pandas MultiIndex columns for cleaner data structure - Default to CAD for domestic Canadian sources (accommodations and domestic tables) - Created SQLite database schema (raw_tables, rate_entries, exchange_rates, accommodations) - Successfully scraped 92 tables with 17,205 rate entries covering 25 international cities - Added migration script to convert scraped data to Node.js database format - Updated .gitignore for Python files (.venv/, __pycache__, *.pyc, *.sqlite3) - Fixed city validation and currency conversion in main app - Added comprehensive debug and verification scripts This replaces manual JSON maintenance with automated data collection from official government source.
7.2 KiB
Database Update Guide
This guide explains how to update the per diem and accommodation rate databases when new rates are published by the National Joint Council or other government sources.
📁 Database Files
The application uses two JSON database files located in the data/ directory:
perDiemRates.json- Meal allowances, incidental expenses, and private accommodation ratesaccommodationRates.json- Hotel and commercial accommodation rates by city
🔄 When to Update
Update the databases when:
- NJC publishes new travel directive rates (typically annually)
- Accommodation rates change in the government directory
- New cities or regions are added to coverage
- Currency exchange rates significantly change
📋 How to Update Per Diem Rates
Step 1: Check Official Sources
Visit these official sources for current rates:
Step 2: Update perDiemRates.json
-
Open
data/perDiemRates.jsonin a text editor -
Update the metadata section:
"metadata": { "effectiveDate": "YYYY-MM-DD", "version": "X.X", "lastUpdated": "YYYY-MM-DD" } -
Update rates for each region. Example for Canada:
"canada": { "name": "Canada (Provinces)", "currency": "CAD", "meals": { "breakfast": { "rate100": 29.05, // Update this value "rate75": 21.80, // 75% of rate100 "rate50": 14.55 // 50% of rate100 }, "lunch": { "rate100": 29.60, "rate75": 22.20, "rate50": 14.80 }, "dinner": { "rate100": 60.75, "rate75": 45.55, "rate50": 30.40 } }, "incidentals": { "rate100": 17.30, "rate75": 13.00 } } -
Calculate derived rates:
rate75=rate100× 0.75rate50=rate100× 0.50- Update
totalrates as sum of breakfast + lunch + dinner
-
Repeat for all regions:
canada,yukon,nwt,nunavut,usa,alaska,international
Step 3: Validate JSON Format
Before saving, validate your JSON:
- Use an online JSON validator (e.g., jsonlint.com)
- Ensure all brackets, braces, and commas are correct
- Check that numbers don't have quotes around them
🏨 How to Update Accommodation Rates
Step 1: Check Official Sources
Visit:
- Government Accommodation Directory
- Select specific cities to see current rates
Step 2: Update accommodationRates.json
-
Open
data/accommodationRates.jsonin a text editor -
Update the metadata section with new date and version
-
Update or add city rates:
"cityname": { "name": "City Name, Province", "province": "Province/State", "region": "canada", "standardRate": 150.00, // Update this "maxRate": 185.00, // Update this "currency": "CAD", "notes": "Any special notes" } -
Key guidelines:
- City keys should be lowercase, no spaces (e.g., "newyork", "losangeles")
standardRate= typical government-approved ratemaxRate= maximum without special authorization- Include
currency(CAD or USD)
-
Update defaults section for regional averages:
"defaults": { "canada": { "standardRate": 150.00, "maxRate": 185.00, "currency": "CAD" } }
Step 3: Add New Cities
To add a new city:
"citykey": {
"name": "Full City Name, Province/State",
"province": "Province or State",
"region": "canada|usa|yukon|nwt|nunavut|alaska|international",
"standardRate": 0.00,
"maxRate": 0.00,
"currency": "CAD|USD",
"notes": "Optional notes about the city"
}
✅ Testing After Updates
- Save both JSON files
- Refresh the web application in your browser
- Test with different destinations:
- Select each destination type
- Verify meal rates are correct
- Check accommodation suggestions
- Verify the footer shows updated effective date
- Check browser console (F12) for any errors
🌍 International Rates
International rates are more complex as they vary by country. For international updates:
- Consult NJC Appendix D
- Rates may be in local currency or CAD
- Add new countries to
internationalCitiessection - Include currency conversion notes
Example:
"citykey": {
"name": "City Name, Country",
"country": "Country Name",
"region": "international",
"standardRate": 200.00,
"maxRate": 300.00,
"currency": "CAD",
"notes": "Convert from EUR/GBP/etc. High cost city."
}
📊 Rate Tiers (Extended Stay)
The application currently uses 100% rates. For extended stays (31+ days, 121+ days), the database includes reduced rates:
- Days 1-30: Use
rate100(100% of allowance) - Days 31-120: Use
rate75(75% of meal allowance) - Days 121+: Use
rate50(50% of meal allowance, 75% incidentals)
Note: Extended stay logic can be implemented in future versions.
🔒 Data Integrity Checklist
Before finalizing updates:
- All numbers are valid decimals (use .00 format)
- No missing commas between items
- All JSON brackets and braces match
- Metadata dates are updated
- Version number is incremented
- Calculations are correct (rate75 = rate100 × 0.75)
- Currency codes are uppercase (CAD, USD)
- Region codes match application options
- File saved with UTF-8 encoding
📞 Support Resources
Official Government Sources:
- NJC Travel Directive Main Page
- Appendix C - Canadian/USA Rates
- Appendix D - International Rates
- Accommodation Directory
JSON Validation Tools:
📝 Version History Template
Keep a log of database updates:
Version 1.1 - 2025-XX-XX
- Updated meal rates for Canada based on new NJC directive
- Added 3 new cities to accommodation database
- Adjusted international rates for currency changes
Version 1.0 - 2025-10-30
- Initial database creation
- Rates effective October 1, 2025
⚠️ Important Notes
- Always backup the existing JSON files before making changes
- Test thoroughly after updates to ensure the app still functions
- Document changes in the metadata section
- Validate JSON before deploying to avoid application errors
- Communicate updates to users about new effective dates
🚀 Quick Update Workflow
- 📥 Download latest rates from NJC website
- 📋 Open JSON files in text editor
- ✏️ Update rates and metadata
- ✅ Validate JSON syntax
- 💾 Save files
- 🧪 Test in browser
- 📢 Document changes
- ✨ Deploy updates
Last Updated: October 30, 2025
Database Version: 1.0