Write out the DataFrame as a single Parquet file:ĭf.write.parquet('lta_table_path', mode='overwrite').Filter the DataFrame to include only the latest version:ĭf = df.filter('version = (SELECT max(version) from delta_table_path)').Get the latest version of the Delta table:ĭelta_table = DeltaTable.forPath(spark, delta_table_path).Load the Delta Parquet files into a Spark DataFrameĭf = ('delta').load(delta_table_path).To convert Delta Parquet files to a single Parquet file with the latest version of Delta, you can use Apache Spark and Delta Lake. (future state) - Add a step in the spark notebook that will take the latest version of Dim Activity in our delta lake or delta table and write out as a single parquet file for downstream usage.Write the updated version of Dim Activity table to a delta lake as parquet files and as a delta table in serverless sql lake database.Use sparknote book to process the raw parquet involving changes to Dim Activity.This is loaded into the raw part of our datalake daily On Prem Data Warehouse (Dim Activity) -> Get changes on Dim Activity based on last modified date in on Prem table and the watermark column in our Azure SQL Db.Take the output of (3.) above and extract a single parquet file that is the latest version of a table in the delta lake.

We store the output (2.) in a delta lake as parquet files and in a serverless sql lake database using delta.

Use spark notebook to process these changes from raw -> curated (note our on prem data does not need any changing to match business needs, that is already complete).Get changes from source dbs (on prem SQL Server) into raw as a parquet via last modified date in source dbs and watermark column in our datasource table.We are using spark notebooks with Delta via pyspark and spark.sql. I am looking for some best practices on how to implement and change to our existing incremental load DW in Azure Synapse.

Edge current version