THE GITHUB REPO LINK: https://github.com/realmistic/economics-workshop-dec-2024The journey of building our Economic Data Dashboard showcases a modern data pipeline and development workflow, leveraging AI assistance to accelerate development while maintaining high code quality.
The EvolutionWhat started as a simple Jupyter notebook exploring financial data has evolved into a production-ready dashboard with real-time updates. The development process followed a natural progression from data collection to visualization to deployment.
Data Collection & StorageOur data pipeline begins with two primary sources:
FRED (Federal Reserve Economic Data) and
Yahoo Finance. We initially prototyped data collection in a Jupyter notebook, then modularized it into dedicated Python scripts. The fred_data_retrieval.py script handles macroeconomic indicators, while btc_minute_data.py manages cryptocurrency data collection.
Data persistence is handled through
SQLite, chosen for its simplicity and portability. This local-first approach makes the application easy to develop and deploy, while still maintaining robust data management capabilities.
Visualization & InterfaceThe frontend is built with
Streamlit, which transforms our Python scripts into a polished web application. We leverage
Plotly for interactive charts, providing users with rich data exploration capabilities. The application is organized into themed pages (economic indicators, stock market, interest rates, etc.), each focusing on specific aspects of the economic landscape.
Development WorkflowOne of the most interesting aspects of this project is the
AI-assisted development approach using
Cline. This tool accelerated our development process by:
- Generating boilerplate code
- Suggesting optimizations
- Helping with code organization
- Assisting with debugging
Automation & DeploymentThe project uses a clever combination of
GitHub Actions and
Streamlit Cloud for automated updates:
- GitHub Actions runs our daily data collection workflow:
- Executes the FRED data retrieval script
- Updates the SQLite database
- Commits the updated database back to the repository
- Streamlit Cloud handles deployment:
- Automatically detects new commits to the database file
- Rebuilds and deploys the dashboard with fresh data
- No complex CI/CD needed - just push and deploy
This simple yet effective setup ensures our dashboard always shows the latest economic data without requiring complex deployment pipelines.
The ResultThe final product is a daily-updated economic dashboard that combines multiple data sources and interactive visualizations. The modular structure (pages/ directory, separate styling in static/css/, utility functions) makes the codebase maintainable and extensible.
This architecture demonstrates how modern tools and practices can be combined to create a sophisticated financial data platform, using GitHub Actions for data automation and Streamlit Cloud for seamless deployment.