Why Standard dbt Incremental Strategies Fall Short?
While data processing in the Bronze and Silver layers is handled via Apache Spark in Databricks, we chose dbt to manage the Gold layer transformations. To optimize processing efficiency, we enabled Change Data Feed (CDF) on our Silver Delta tables to capture row-level changes between pipeline executions.