unit testing in dbt

and how to go about it

In my previous article How to set up a unit testing framework for your dbt projects, I addressed unit testing for relatively simple models inside of dbt. dbt has the concept of materialisations, where you can choose how your model physicalizes the data once it is run. According to dbt,

When using the materialization, your model is rebuilt as a table on each run, via a statement.

This is perhaps more familiar as a truncate-and-load strategy. However, constant rebuilds can be unwieldy and slow when dealing with large amounts of event-based data.

Incremental models are tables…

unit testing in dbt

Photo by Christian Wiediger on Unsplash

dbt (data build tool) is a powerful tool in the ELT pipeline that allows you to perform transformations on data loaded in your warehouse. In this article, we’ll discuss custom testing within dbt. We’ll be using GCP services such as BigQuery, Cloud Build and Cloud Source Repositories.

So you’ve set up your dbt project and created your models. How do you test them? In other words, how do you ensure that your models are performing exactly the right kind of transformations that you intend? This is where unit testing comes in.

The idea is to provide input files that resemble…

Cloud Run is a fully managed service provided by Google Cloud that allows you to deploy containerised applications. In deploying a container image to Cloud Run, you create what’s called a service. These services have an immutable URL associated with them which can be used to access your app. While you can lock them down using IAM, there’s always additional security benefits to using external ingress controls and whitelisting IPs from trusted sources. After all, you don’t want just anybody accessing your app.

A Cloud Run service URL

How do we do it? Cloud Load Balancing and Cloud Armor! Network-based ingress controls can be implemented…

Betsy Varghese

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store