Do something cool with it: a week on Databricks

“Do something cool with it.”

This was the charge I was given by our CTO when he handed me a transactional data set of CSV files from a pet project our team is working on for a client. I decided to put my data warehousing experience to work and see how quickly I could stand up a full analytics + transactional platform on Databricks including natural language queries, a front-end application, and agentic processing.

Since I’m in charge of the partnership with Databricks, first-hand experience goes a long way toward understanding the potential of the platform and the challenges that our customers are up against.

About a week of effort got me so much farther than I expected.

What Was I Given?

The CSVs held audiobook sales data from our InAudio project. The tables provided looked like this:

• bestseller_lists.csv
• categories.csv
• category_rankings.csv
• retailers.csv
• snapshots.csv (daily sales and review counts)
• title_formats.csv
• titles.csv

.. as well as a few housekeeping tables.

The data came from a nicely normalized PostgreSQL database - direct dumped to CSV without foreign keys or any metadata. Fortunately, the columns and tables were well named.

This would need organization and processing before anyone could do real reporting on it.

What Would This Normally Take?

In a typical organization, what I attempted this week would have been a lengthy enterprise project. It probably would have been over-scoped and delivered months late. Somebody would have argued to include all of InAudio’s data - and that would have led to months of data engineering just to get the data imported. A data analyst and a data engineer would get assigned to the project. BI licenses would have to be procured. The data warehouse would get stood up - then the real integration could begin. During all of that inactivity, there would probably have been a server plus licenses sitting idle just waiting to be built out.

The economics behind my week look completely different from that idle server. Databricks serverless SQL bills by the second, at roughly $0.70 per DBU in US regions with the infrastructure baked into the rate (these are published list rates.. pricing may vary depending on your region, your agreement, and the product used). The smallest warehouse burns about 4 DBUs an hour, which works out to $2.80 an hour while it’s awake. The important phrase is “while it’s awake”: the warehouse scales to zero when idle, and a small company’s workload is mostly idle time. A nightly load and a burst of morning dashboard queries might run the meter for an hour or two a day. The flip side: a meter that lets you in cheap also runs when nobody’s watching, so auto-stop and budget alerts are worth setting up on day one.

Building the Medallion

Though my long-term plan was to have a daily data feed from the InAudio sales data, that wasn’t set up on the source side (yet) and I had a single point-in-time snapshot of the data. It was a simple matter to upload the files to a Databricks volume.

Now that the data existed inside Databricks, the next step was to create the medallion architecture and process the files into the Bronze/Silver/Gold tables. Technically speaking, we already had a Bronze layer with the uploaded files - but I wanted them in table format.

I was an early adopter of Microsoft Fabric, so most of my experience around data processing was imperative in nature. This seemed like a good place to start.. because as we always said when teaching ski lessons “new hill, old skill”. I wrote a PySpark notebook that loaded each CSV file into a dataframe and wrote it into the catalog as a table.

More PySpark scripts cleaned up the column names and added timestamps to make the Silver tables. Finally, I did some summarization and created dimension / fact tables to make a reasonable Gold layer. For those curious, here are the summary tables created:

• bestseller_history
• category_movers
• daily_title_metrics
• title_current

.. and the dimensions + fact tables:

• dim_category
• dim_date
• dim_title_format
• fact_bestseller_position
• fact_category_rank
• fact_snapshot

I could have quit there - I’ve now got a proper medallion architecture that will process the daily feed idempotently once it’s flowing (to avoid duplicates). Databricks (and more recently, Fabric) has a cooler trick up its sleeve.

The modern way to implement a medallion architecture is using a declarative pipeline. Instead of step-by-step processing of the data, I just need to declare what I want the end state of the data to look like and Databricks figures out how to get it there. I can also add any rules or constraints to enforce data quality as part of the declaration. When you execute the declarative pipeline - it tests each constraint and checks what new data has arrived and only runs the parts that need it.

Total time for each version of the ETL pipeline? About two days.

Full disclosure - I used Claude to assist with the grunt work of coding - but I did need to guide it every step of the way to get the final shape I wanted. You will as well.

Time spent: 1 hour for the data import, about 2 days for the pipeline.

What About the Dashboard?

Before this project, I had absolutely no experience with Databricks’ AI/BI dashboard tool. I am certified in Microsoft’s Power BI and am very familiar with it.. so it would be an interesting contrast to create a dashboard with a new tool.

.. and I’m not sure if AI/BI would win that fight on features alone. Power BI has a many-year head start and is the gold standard for BI. The crux of this experiment is using the platform as presented. That means taking advantage of Genie Code to create my dashboard in natural language terms.

I won’t say Genie Code got it perfect - and I did need to adjust a few things by hand. The net result was a great starting point and one that’s entirely appropriate to grow into a more useful tool.

Time spent: about 4 hours.

Asking Questions in Plain English

Accessing the data is the next hurdle that a small business needs to clear. Historically, this would require a seat of a BI tool and a lot of SQL statements to retrieve the data coherently. Though all of that is still useful and relevant - it’s not mandatory any longer. Genie adds context-driven natural language queries directly to the dataset. Setting it up could be as simple as pointing directly to the data table, but you’ll get better results if you give Genie prompting and instructions on how to interpret the data. If you’re so inclined - you can even have Genie examine the data and write its own context prompts.

Another couple of hours, and I could now just ask my new data platform “What indie authors have an increased review velocity over the past month?”

Time spent: about 4 hours, setup and testing.

Adding the Transactional Layer

One other feature of Databricks that I was very excited to exercise is Lakebase. Lakebase is a Databricks-hosted version of Postgres that separates the compute and the storage. This has some great side benefits - like not having to clone the data for transactional usage, full elasticity including scale-to-zero, and instantaneous branching of the database.

I introduced the notion of a user, a list of titles to track, and an alert table. An online user can watch a title and set alerts on large changes. It syncs the lakehouse tables it needs into Postgres, and the app’s own tables sync back to the lakehouse.

I wanted to access the transactional data via a Databricks App - all built using the App builder inside of Databricks. The app, which I named Bookpulse, lets a user manage a watchlist and displays agent-generated alerts about sales trends for audiobooks.

Now that the platform had users, the next question was what someone watching a title would actually want to know. I landed on four kinds of alert rules:

• A title breaks into the top 1,000 sellers (a bestseller-rank threshold)
• A title climbs more than 200 rank spots in a single day
• The price drops below $4.. which usually means a promotion is running
• Review velocity passes five per day, the closest public proxy for a sales spike

Each rule lives as a row in a Postgres table, tied to a single watchlist item.. remove the title from your watchlist and its rules go with it. After the nightly pipeline runs and Lakebase syncs, an agent reads the active rules, evaluates them against the fresh metrics, and writes any hits to an alerts table that Bookpulse displays. A hosted LLM then turns the day’s hits into a plain-English daily brief. The morning after one watched title jumped more than 200 spots, the brief led with it.

Time spent: 4 hours on Lakebase setup, 4 hours on the Bookpulse app, and about 2 days on the agent.

What Did a Week Buy?

Here’s the final tally:

• Data import: 1 hour
• Declarative pipeline for ETL and the medallion architecture: 2 days
• Genie setup and testing: 4 hours
• Lakebase setup: 4 hours
• AI/BI dashboard: 4 hours
• App generation and testing: 4 hours
• Agent creation and modification: 2 days
• .. and a daily feed that downloads the full dump and reloads it idempotently, so new records land without duplicates? 4 hours.

(The imperative PySpark pipeline isn’t counted - it was a test run and didn’t make the final build.)

What Did It Cost to Run?

So what did all of this cost to run? I pulled yesterday’s bill, a full day with everything turned on: the import, the declarative pipeline, the processing agents, and the Lakebase sync. The total was $4.65. Stretch that over a month and it comes to about $141, and $41 of that was me personally poking at notebooks and queries.. the platform itself runs closer to $101. The biggest single line item is Lakebase, the transactional database, at roughly $67 a month. The entire ETL chain costs about $19 a month to run, and the LLM serving behind the agents about $13.

Before anyone objects: this is one snapshot-sized dataset and one user. A company with daily feeds and twenty people asking Genie questions will land higher, and your bill will grow with usage. The order of magnitude is the point. Once you counted licenses and the people to run it, platforms in this category used to open the conversation at six figures a year. This one runs at hobby-project money, and for reference, Microsoft’s smallest Fabric capacity is $262.80 a month before you’ve bought a single Power BI license.

Does a Small Organization Need a Big Platform?

Let’s talk about the elephant in the room. Does a small organization need a big platform?

The strongest voices against it have come from Jordan Tigani, a founding BigQuery engineer. He put out an essay called “Big Data is Dead” that argues that the majority of enterprise data warehouses are under a terabyte and that a single cloud machine can now comfortably handle what required a cluster a decade ago.

He’s right, of course. My stack of CSVs would run happily on a laptop, and data size was never the reason to buy a platform. The benefits that matter sit beyond query speed and performance: governed catalog, managed pipeline, natural language queries, transactional data, apps, and the list goes on. That’s the stuff I bought for $4.65 a day.

I’m not advocating that an office manager with Excel take this on (well, in most cases.. I don’t want to overgeneralize). What I built is deliberately PoC-level. A production version would need the unglamorous parts: cost controls, governance, access controls, and the DataOps discipline around all of it. That work still requires trained experts in Databricks, data warehousing, and reporting.

The argument is about scale. An enterprise data program used to mean a dedicated team, a multi-year roadmap, and a budget to match. A small business can now hire a small team and get results in months instead of years, for a fraction of what those programs used to cost. Tarmac is a Databricks partner and this is exactly the type of problem that we solve all the time: fix it once, and leave the customer with a platform where the data is easy to reach and easy to extend when the next need comes along.

Now or Later?

This argument lands close to home. My lovely wife is in marketing at Andersen Windows and they solve the same type of problem. People live with drafty windows for years.. they get used to the drafts and don’t notice their energy bills are higher than they should be. A small team of experts can handle this in about the same time my PoC took, and the rewards run for years: the drafts are gone and the energy bill drops every month after (and the impact to the resale value doesn’t hurt either!).

Did I mention we’re getting new windows this fall?