Goals achieved

• Ingest different source files into ADLS Gen2 and archive existing files to cost effective storage location.
• Ingestion, the pipeline should add 2 columns: source file name & load datetime.
• Design log framework on top of separate table with source file name & load datetime and record count
• Post staging, we will apply basic transformation (as per shared stored procedure)
• All data should be stored as parquet in Data warehouse solution
• Final result should have connectivity to PowerBI dashboards

Benefits

• Atomized Data profiling, data governance and Data archival process.
• Efficient Customer and agent portfolios.