In my previous article How to set up a unit testing framework for your dbt projects, I addressed unit testing for relatively simple models inside of dbt. dbt has the concept of materialisations, where you can choose how your model physicalizes the data once it is run. According to dbt,
When using the
tablematerialization, your model is rebuilt as a table on each run, via a
create table asstatement.
This is perhaps more familiar as a truncate-and-load strategy. However, constant rebuilds can be unwieldy and slow when dealing with large amounts of event-based data.
Incremental models are tables…
dbt (data build tool) is a powerful tool in the ELT pipeline that allows you to perform transformations on data loaded in your warehouse. In this article, we’ll discuss custom testing within dbt. We’ll be using GCP services such as BigQuery, Cloud Build and Cloud Source Repositories.
So you’ve set up your dbt project and created your models. How do you test them? In other words, how do you ensure that your models are performing exactly the right kind of transformations that you intend? This is where unit testing comes in.
The idea is to provide input files that resemble…
Cloud Run is a fully managed service provided by Google Cloud that allows you to deploy containerised applications. In deploying a container image to Cloud Run, you create what’s called a service. These services have an immutable URL associated with them which can be used to access your app. While you can lock them down using IAM, there’s always additional security benefits to using external ingress controls and whitelisting IPs from trusted sources. After all, you don’t want just anybody accessing your app.