ETL pipeline for forecasting (ERP -> BigQuery -> Power BI)

#Data-Platform for automating procurement forecasting in a manufacturing company.

Complete development from scratch: architecture design, building #ETL pipeline, integration with #ERP, and setting up an analytical dashboard.

Data integration:
— connection to ERP via #REST API
— incremental data loading while preserving history
— API error handling (retry, logging)
— integration of ARIMA model (#R-script) with BigQuery

Storage and processing:
— design of #DWH in BigQuery
— calculation of product balances by warehouses
— transformation of raw data into analytical tables

Automation:
— launching #ETL processes on schedule (#GCP Cloud Scheduler)
— nightly #pipeline: data update —> model execution —> report update

Visualization:
— building new and adapting existing dashboards in Power BI for analysis of balances and forecasts

Features:
— replacement of manual accounting in Excel
— stable daily data updates without manager involvement
— minimized data discrepancies between ERP and #BI after extensive testing

Stack: #Python, #BigQuery, #GoogleCloud, #PowerBI, #RScript, #REST, #Docker