From Jira Ticket to Production: What Does a Mature Databricks Workflow Look Like?

On Reddit someone asked how you work with Databricks in a real project. Not in a demo, but across the chain of Jira, GitHub, Databricks, production.

The specific questions were familiar. Where do you write PySpark, in notebooks or in VS Code? Where do the tests run? How do you do code review? How do you deploy to Databricks? How do you move from development to test to production? And what role do service principals and CI/CD play in that?

It is a good question with a misleading front. The choice between notebook and IDE feels like the core, and it is not. The core is where your source of truth lives and how controlled the movement is from requirement to production. Once that is settled, the editor becomes a preference.

The most detailed reply described a fairly mature chain: a ticket becomes a feature branch, development happens in VS Code, pytest runs locally, the code goes to Databricks through bundles, integration tests run there and output is validated, and only then comes the pull request with review and linting. After merge a CI/CD pipeline takes over towards development, test and finally production, with a service principal as the identity, a dedicated workspace per environment and a manual approval before the last step.

The arguments for local development were practical: navigation, linting, formatting, Git integration, fast feedback and refactoring. The same participant noted that problems simply surface earlier locally than through the UI.

Not everyone works that way. Another participant develops in folders inside the workspace and uses DevOps and service principals to move through QA to production. That is not wrong by definition. It only becomes a problem when the workspace becomes the source of truth instead of Git.

A final reply went one step further and described working from Genie Code with connectors to Jira, GitHub and Databricks, expecting this to get more interesting once write access for those connectors becomes available.

The described workflow fits the documentation well. On four points that documentation is now more specific, or different.

Start with the name. Databricks Asset Bundles were recently renamed to Declarative Automation Bundles. The documentation still lists the old name as the former name, so you will recognise it, but new documentation, templates and conversations use the new term. They are and remain the recommended approach to CI/CD on Databricks: jobs, pipelines, dashboards, model serving endpoints and MLflow objects described as source files and deployed as a single unit.

The second point is authentication, and it is the most important correction to the discussion. A service principal remains the recommendation for all non-development automation, for exactly the reason the discussion gives: a production deployment then does not stall when someone leaves the company. But Databricks goes further. For CI/CD authentication the documentation recommends workload identity federation, because it removes the need for a Databricks secret entirely. It is called the most secure way to authenticate your automated flows, and for regulated industries it is prescribed explicitly to avoid long-lived secrets in GitHub Actions or Azure DevOps.

There is a separation that goes with it and that the discussion does not make. Use separate service principals for deployment and for runtime. The deployment identity needs minimal data access, while every production job should have its own run-as identity, scoped only to the data and resources that workload actually touches. That keeps deployments safe when you tighten or rotate data access.

The third point is the order of testing. The workflow in the discussion deploys to Databricks and runs integration tests before the pull request is opened. That works, but the documented chain places the weight differently. Databricks describes three testing layers. Unit tests on importable modules in src/, using pytest, running on every pull request so a failure blocks the merge. Then bundle validate locally, and in CI preferably a bundle deploy to a non-production workspace to catch YAML and resource-mapping issues. And finally integration tests in staging after deployment, with end-to-end runs and hard data quality assertions such as row counts and schema expectations. The gate to production is that all tests pass on the main branch and in staging.

The branching strategy belongs with that. The discussion assumes feature branches merged into main, which is right, but the documentation is more specific: use a trunk-based strategy with short-lived feature branches so main is always deployable. The chain is then develop locally or in the workspace, a short branch, merge to main, automatically to staging, tests, and then to production.

The fourth point concerns environment separation, and here the discussion misses a layer. Separate workspaces per environment are indeed the recommendation, with a concrete growth path: up to roughly five data engineers two workspaces are enough (development and production), above that it becomes three, where staging should be functionally representative of production with the same bundle configuration, schemas and critical integrations, even if it is scaled down.

But workspaces alone are not enough. The documentation also prescribes data separation: a single Unity Catalog metastore with separate catalogs for dev, staging and production, mirroring your workspaces. And then the step most often missing in practice: bind the production catalog in ISOLATED mode to the production workspace only. That makes production data unreachable from development or staging even if an identity is misconfigured. For developers, personal schemas in the non-production catalogs are the recommendation, with bundles using something like dev_${user_name} so nobody overwrites another person's tables.

On notebooks the documentation is clearer than the discussion. Do not treat them as the primary container for business logic. Put that logic in importable .py modules in src/, and SQL in .sql files, and use notebooks as a thin orchestration and visualisation layer on top. For existing projects the recommendation is to migrate incrementally, one module at a time.

Finally Genie Code. It exists and is the AI assistant for developers inside the workspace, with inline suggestions, error diagnosis and support in notebooks, the SQL editor, the Lakeflow Pipelines Editor, dashboards and MLflow. The expectation in the discussion about write access has been partly overtaken: external connectors to Atlassian (Jira and Confluence), GitHub and Microsoft 365 among others exist, and some of them can genuinely perform write actions, depending on the OAuth scopes you consent to and your permissions in the source system. With caveats though: those connectors are in Beta and must be released by a workspace admin, the GitHub connector can reach only public repositories by default, and GitHub Enterprise Server is not supported.

The "VS Code or notebooks" question disappears once you rephrase it. Where does the source of truth live?

If the answer is Git, then Databricks is your execution platform. A notebook is then a tool for exploration, debugging and visualisation, and the fact that you can also run production code in one is irrelevant, because that code comes from the repository. Anyone who develops in the workspace and then brings the change into Git can achieve the same, provided that last step always happens.

If the answer is the workspace, you get the pattern the discussion rightly rejects: manual changes, copy and paste, drift between environments, and nobody knowing which version runs in production.

"Databricks is your execution platform, not your version control. The moment those two swap, every other question gets harder."

What strikes me most in the discussion is that the identity question is treated as a detail when it is the heaviest one. The discussion correctly observes that a personal account should not perform production deployments. The documented recommendation goes further and removes the secret itself. A service principal with a long-lived token is still a secret sitting in a pipeline, one that has to be rotated and that can leak. Workload identity federation solves that by letting the pipeline fetch a short-lived token based on its own identity. That is not an optimisation but the difference between a secret you manage and a secret you do not have.

The second underestimated layer is the catalog. Many teams stand up three workspaces and consider the separation done. That holds for compute and for workspace-level permissions, but not for data. Without catalog separation, and without the production catalog in ISOLATED mode, a misconfigured identity in development can still reach production data. The workspace boundary is an organisational boundary, the catalog binding is the technical one.

There is one more choice that often goes wrong and does not come up in the discussion. Do not create a bundle per environment. The recommendation is small and focused: everything a single team owns in one bundle, and that one bundle covers dev, staging and production through targets. Separate bundles follow product boundaries, ownership or clearly different lifecycles, not environments. Creating a bundle per environment duplicates configuration and brings back exactly the drift bundles were meant to solve.

On the approval step, finally. The discussion mentions a manual approval before production, and that is sensible for regulated workloads, change management and separation of duties. But treat that approval as a decision, not as a test. If staging is functionally representative and the integration tests with data quality assertions pass, you technically know enough. The approval is about timing, communication and accountability. An approval that serves as a safety net for missing tests gives false confidence.

And Genie Code, or any AI assistant, changes none of that. It can dramatically change how fast code appears, but not the need for version control, review, tests, approvals and controlled deployments. If anything, the faster code appears, the more the gates around it matter.