mirror of
https://github.com/mblanke/Gov_Travel_App.git
synced 2026-03-01 14:10:22 -05:00
Add Python web scraper for NJC travel rates with currency extraction
- Implemented Python scraper using BeautifulSoup and pandas to automatically collect travel rates from official NJC website - Added currency extraction from table titles (supports EUR, USD, AUD, CAD, ARS, etc.) - Added country extraction from table titles for international rates - Flatten pandas MultiIndex columns for cleaner data structure - Default to CAD for domestic Canadian sources (accommodations and domestic tables) - Created SQLite database schema (raw_tables, rate_entries, exchange_rates, accommodations) - Successfully scraped 92 tables with 17,205 rate entries covering 25 international cities - Added migration script to convert scraped data to Node.js database format - Updated .gitignore for Python files (.venv/, __pycache__, *.pyc, *.sqlite3) - Fixed city validation and currency conversion in main app - Added comprehensive debug and verification scripts This replaces manual JSON maintenance with automated data collection from official government source.
This commit is contained in:
95
improvements.json
Normal file
95
improvements.json
Normal file
@@ -0,0 +1,95 @@
|
||||
{
|
||||
"project": "Govt Travel App",
|
||||
"analyzed_at": "2025-10-31T14:59:29.610433",
|
||||
"total_files": 4,
|
||||
"average_score": 7.0,
|
||||
"files": [
|
||||
{
|
||||
"file": "flightService.js",
|
||||
"filepath": "/projects/Govt Travel App/flightService.js",
|
||||
"score": 6,
|
||||
"strengths": [
|
||||
"The code is well-structured and follows a clear logic flow.",
|
||||
"It includes error handling for Amadeus client initialization and API requests."
|
||||
],
|
||||
"issues": [],
|
||||
"improvements": [],
|
||||
"security": [
|
||||
"None"
|
||||
],
|
||||
"raw_analysis": "SCORE: 6\n\nSTRENGTHS:\n- The code is well-structured and follows a clear logic flow.\n- It includes error handling for Amadeus client initialization and API requests.\n\nISSUES:\n\n* **Potential Security Risk:** The `process.env` variables are directly used to initialize the Amadeus client. This could lead to sensitive information being exposed if not handled properly, especially when using `.env` files in a shared environment.\n* **Code Duplication**: There is a repeated check for `amadeus` initialization inside the `searchFlights` function. This should be refactored into a separate utility function or a service class to avoid duplication.\n\nIMPROVEMENTS:\n\n* **Type Checking and Documentation**: The function parameters could benefit from type annotations, and function descriptions (JSdoc-style) would improve code readability.\n* **Performance Optimization**: Instead of making multiple API calls with the same parameters, it might be more efficient to store the Amadeus client instance in a cache or singleton pattern. However, this should be balanced against the need for lazy initialization based on available credentials.\n\nSECURITY:\n- None"
|
||||
},
|
||||
{
|
||||
"file": "script.js",
|
||||
"filepath": "/projects/Govt Travel App/script.js",
|
||||
"score": 7,
|
||||
"strengths": [
|
||||
"The code is well-structured and easy to follow, with clear separation of concerns between database loading, metadata display update, and rate validation.",
|
||||
"The use of async/await for database fetching is a good practice."
|
||||
],
|
||||
"issues": [
|
||||
"The global variables `perDiemRatesDB`, `accommodationRatesDB`, and `transportationRatesDB` are not initialized with a default value. If the code is executed before the databases are loaded, these variables will be `null` or `undefined`.",
|
||||
"The database loading function `loadDatabases()` does not handle cases where some of the databases fail to load while others succeed.",
|
||||
"The `validateRatesAndShowWarnings()` function modifies the `warnings` array in place. If this function is called multiple times with the same input, the warnings will be overwritten."
|
||||
],
|
||||
"improvements": [
|
||||
"Initialize global variables with default values: Instead of letting them remain `null` or `undefined`, consider initializing them with an empty object `{}` or a default value that indicates no data is available.",
|
||||
"Handle partial database loading failures: Consider adding a check to see if all databases have loaded successfully before proceeding. If some fail, display an error message and prevent further execution.",
|
||||
"Refactor `validateRatesAndShowWarnings()` to return the warnings array instead of modifying it in place. This will make the function more predictable and easier to use."
|
||||
],
|
||||
"security": [
|
||||
"None"
|
||||
],
|
||||
"raw_analysis": "SCORE: 7\n\nSTRENGTHS:\n- The code is well-structured and easy to follow, with clear separation of concerns between database loading, metadata display update, and rate validation.\n- The use of async/await for database fetching is a good practice.\n\nISSUES:\n- The global variables `perDiemRatesDB`, `accommodationRatesDB`, and `transportationRatesDB` are not initialized with a default value. If the code is executed before the databases are loaded, these variables will be `null` or `undefined`.\n- The database loading function `loadDatabases()` does not handle cases where some of the databases fail to load while others succeed.\n- The `validateRatesAndShowWarnings()` function modifies the `warnings` array in place. If this function is called multiple times with the same input, the warnings will be overwritten.\n\nIMPROVEMENTS:\n- Initialize global variables with default values: Instead of letting them remain `null` or `undefined`, consider initializing them with an empty object `{}` or a default value that indicates no data is available.\n- Handle partial database loading failures: Consider adding a check to see if all databases have loaded successfully before proceeding. If some fail, display an error message and prevent further execution.\n- Refactor `validateRatesAndShowWarnings()` to return the warnings array instead of modifying it in place. This will make the function more predictable and easier to use.\n\nSECURITY:\n- None\n\nHere is a refactored version of the code that addresses these issues:\n\n```javascript\nlet perDiemRatesDB = {};\nlet accommodationRatesDB = {};\nlet transportationRatesDB = {};\n\nasync function loadDatabases() {\n try {\n const [perDiemResponse, accommodationResponse, transportationResponse] = await Promise.all([\n fetch('data/perDiemRates.json'),\n fetch('data/accommodationRates.json'),\n fetch('data/transportationRates.json')\n ]);\n\n if (!perDiemResponse.ok || !accommodationResponse.ok || !transportationResponse.ok) {\n throw new Error('Failed to load rate databases');\n }\n\n [perDiemRatesDB, accommodationRatesDB, transportationRatesDB] = await Promise.all([\n perDiemResponse.json(),\n accommodationResponse.json(),\n transportationResponse.json()\n ]);\n\n updateMetadataDisplay();\n validateRatesAndShowWarnings(perDiemRatesDB, accommodationRatesDB, transportationRatesDB);\n\n return true;\n } catch (error) {\n console.error('Error loading databases:', error);\n alert('Error loading rate databases. Please refresh the page.');\n return false;\n }\n}\n\nfunction updateMetadataDisplay() {\n if (perDiemRatesDB && perDiemRatesDB.metadata) {\n const footer = document.querySelector('footer p');\n if (footer) {\n footer.textContent = `Based on NJC Travel Directive effective ${perDiemRatesDB.metadata.effectiveDate} (Rates updated: ${perDiemRatesDB.metadata.lastUpdated})`;\n }\n }\n\n validateRatesAndShowWarnings(perDiemRatesDB, accommodationRatesDB, transportationRatesDB);\n}\n\nfunction validateRatesAndShowWarnings(...databases) {\n const warnings = [];\n\n databases.forEach((database) => {\n if (database && database.metadata) {\n const effectiveDate = new Date(database.metadata.effectiveDate);\n const lastUpdated = new Date(database.metadata.lastUpdated);\n const monthsSinceUpdate = (new Date() - lastUpdated) / (1000 * 60 * 60 * 24 * 30);\n\n if (monthsSinceUpdate > 12) {\n warnings.push({\n type: 'outdated',\n database: database.name,\n message: `Per diem rates were last updated ${lastUpdated.toLocaleDateString()} (${Math.floor(monthsSinceUpdate)} months ago). Please verify current rates.`,\n lastUpdated: database.metadata.lastUpdated\n });\n } else if (monthsSinceUpdate > 10) {\n warnings.push({\n type: 'warning',\n database: database.name,\n message: `Per diem rates approaching update cycle. Last updated ${lastUpdated.toLocaleDateString()}.`,\n lastUpdated: database.metadata.lastUpdated\n });\n }\n }\n });\n\n // Display warnings here...\n}\n```\n\nNote that I've added a `validateRatesAndShowWarnings()` function with a variable number of arguments to accept the loaded databases, and modified the `updateMetadataDisplay()` function to call this new version. This way, you can easily display warnings for all loaded databases at once."
|
||||
},
|
||||
{
|
||||
"file": "server.js",
|
||||
"filepath": "/projects/Govt Travel App/server.js",
|
||||
"score": 7,
|
||||
"strengths": [
|
||||
"The code is well-organized and uses a consistent naming convention.",
|
||||
"It includes environment variable loading using dotenv, which is a good practice.",
|
||||
"The API endpoints are clearly defined with route handlers for each path."
|
||||
],
|
||||
"issues": [
|
||||
"There is no error handling for missing or invalid environment variables. If AMADEUS_API_KEY or AMADEUS_API_SECRET is not set, the server will crash.",
|
||||
"The `getAirportCode` function from the `flightService` module is called without any input validation. This could lead to unexpected behavior if the function is not designed to handle null or undefined inputs.",
|
||||
"There are no checks for potential errors when reading files using `path.join(__dirname, 'index.html')`, etc."
|
||||
],
|
||||
"improvements": [
|
||||
"Consider adding a check for missing or invalid environment variables and provide a more informative error message instead of crashing the server.",
|
||||
"Validate input parameters for the `/api/flights/search` endpoint to prevent unexpected behavior. For example, you can use a library like `joi` to validate query parameters.",
|
||||
"Use a more robust way to handle errors in route handlers, such as using `res.status(500).send({ error: 'Internal Server Error' })` instead of logging the error message.",
|
||||
"Consider adding API documentation using tools like Swagger or OpenAPI."
|
||||
],
|
||||
"security": [
|
||||
"The code does not have any obvious security concerns. However, it is essential to ensure that environment variables are not committed to version control and that sensitive data (e.g., API keys) are handled securely."
|
||||
],
|
||||
"raw_analysis": "SCORE: 7\n\nSTRENGTHS:\n- The code is well-organized and uses a consistent naming convention.\n- It includes environment variable loading using dotenv, which is a good practice.\n- The API endpoints are clearly defined with route handlers for each path.\n\nISSUES:\n- There is no error handling for missing or invalid environment variables. If AMADEUS_API_KEY or AMADEUS_API_SECRET is not set, the server will crash.\n- The `getAirportCode` function from the `flightService` module is called without any input validation. This could lead to unexpected behavior if the function is not designed to handle null or undefined inputs.\n- There are no checks for potential errors when reading files using `path.join(__dirname, 'index.html')`, etc.\n\nIMPROVEMENTS:\n- Consider adding a check for missing or invalid environment variables and provide a more informative error message instead of crashing the server.\n- Validate input parameters for the `/api/flights/search` endpoint to prevent unexpected behavior. For example, you can use a library like `joi` to validate query parameters.\n- Use a more robust way to handle errors in route handlers, such as using `res.status(500).send({ error: 'Internal Server Error' })` instead of logging the error message.\n- Consider adding API documentation using tools like Swagger or OpenAPI.\n\nSECURITY:\n- The code does not have any obvious security concerns. However, it is essential to ensure that environment variables are not committed to version control and that sensitive data (e.g., API keys) are handled securely."
|
||||
},
|
||||
{
|
||||
"file": "styles.css",
|
||||
"filepath": "/projects/Govt Travel App/styles.css",
|
||||
"score": 8,
|
||||
"strengths": [
|
||||
"Well-structured CSS with clear and consistent naming conventions.",
|
||||
"Effective use of variables for color scheme and layout properties.",
|
||||
"Good practice in using `display: block` and `margin-bottom` to create vertical spacing."
|
||||
],
|
||||
"issues": [
|
||||
"The file is quite large, making it difficult to navigate and maintain. Consider breaking it down into smaller modules or partials.",
|
||||
"There are several hard-coded values throughout the code (e.g., font sizes, padding, margins). Consider introducing constants or variables to make these values more flexible and easily updateable.",
|
||||
"The `box-shadow` property is used multiple times with different values. Create a variable for this value to reduce repetition."
|
||||
],
|
||||
"improvements": [
|
||||
"Consider using a CSS preprocessor like Sass or Less to simplify the code and enable features like nesting, mixins, and variables.",
|
||||
"Use more semantic class names instead of generic ones (e.g., `.form-section` could be `.contact-form`).",
|
||||
"Add comments to explain the purpose and behavior of each section or module.",
|
||||
"Consider using CSS grid or flexbox for layout instead of relying on floats or inline-block.",
|
||||
"Update the code to follow modern CSS best practices, such as using `rem` units for font sizes and margins."
|
||||
],
|
||||
"security": [],
|
||||
"raw_analysis": "SCORE: 8\n\nSTRENGTHS:\n- Well-structured CSS with clear and consistent naming conventions.\n- Effective use of variables for color scheme and layout properties.\n- Good practice in using `display: block` and `margin-bottom` to create vertical spacing.\n\nISSUES:\n- The file is quite large, making it difficult to navigate and maintain. Consider breaking it down into smaller modules or partials.\n- There are several hard-coded values throughout the code (e.g., font sizes, padding, margins). Consider introducing constants or variables to make these values more flexible and easily updateable.\n- The `box-shadow` property is used multiple times with different values. Create a variable for this value to reduce repetition.\n\nIMPROVEMENTS:\n- Consider using a CSS preprocessor like Sass or Less to simplify the code and enable features like nesting, mixins, and variables.\n- Use more semantic class names instead of generic ones (e.g., `.form-section` could be `.contact-form`).\n- Add comments to explain the purpose and behavior of each section or module.\n- Consider using CSS grid or flexbox for layout instead of relying on floats or inline-block.\n- Update the code to follow modern CSS best practices, such as using `rem` units for font sizes and margins.\n\nSECURITY:\nNone"
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user