Lakehouse best practices

Schedule OPTIMIZE and VACUUM as standing maintenance jobs. Thousands of small files are the most common cause of slow queries, and they arise on their own with frequent small writes. Whoever writes streaming or micro-batches without compaction builds a slow Lakehouse guaranteed.

In newly created workspaces, V-Order is off by default for Spark and Lakehouse; Warehouse keeps it on. Direct Lake works fine without it, so treat it as an accelerator rather than a prerequisite: it makes a noticeable difference mainly on a cold cache. Turn it on for Gold and for everything Power BI reads, and do not turn it off "temporarily for faster writes", that temporariness becomes permanent and the reports get slow. V-Order is a read-versus-write trade-off though, not a free knob: on heavy, write-intensive intermediate tables that feed no reporting, you may deliberately leave it off.

The biggest silent capacity drain is a full reload of Silver every night. Use MERGE with the Change Data Feed so you only process the delta. And treat transformation as tested, reusable code: develop in notebooks, but productionize in Spark Job Definitions or pipelines, a notebook with hidden cell state is not a production artifact. Put shared logic in a tested library instead of copied notebooks; ten copies are ten places to patch.

Spark bills per vCore-second, so shorter runs are directly cheaper. First fix small files, partitioning and joins before you scale up. Set a short session timeout (a session that keeps running while nobody works is the biggest silent cost), use high concurrency to share sessions, and consider the Native Execution Engine: one of the few knobs that is both faster and cheaper. Do verify on your real workload that the heavy operators actually run native, the fallback to standard Spark is silent.

Do not allow uncontrolled schema evolution in Silver and Gold. Sources may change; your Silver schema changes through a deliberate change with versioning, not through mergeSchema surprises on a Friday afternoon.

One wide "Gold for everything" becomes unmanageable. Make Gold models per use case (reporting, ML features, API) and accept limited duplication, you trade cheap storage for expensive confusion if you do it the other way around.

Every pipeline that only moves data is technical debt. First check whether a OneLake shortcut suffices; copy only when you need transformation, isolation or retention the source does not provide.

Monitor query duration on Gold and freshness per table, those are the two metrics users call about. Capacity consumption is your problem; these two are their problem, and therefore your first dashboards.