Streamline DBT Model Development With a Notebook-Style Workspace By Nguyen Tran | June, 2023

Mage, an open-source data pipeline tool for data transformation and integration tasks, is a free alternative to DBT Cloud.

Dana basically complements DBT with a range of benefits, including:

1. Integrated web based IDE: Mage provides a convenient web-based IDE where you can easily develop and explore data models within a single interface.
2. Language Flexibility: With Mage, you can combine the strengths of different tools and languages ​​along with DBT for enhanced data processing capabilities.
3. Visualizing the DBT Model Output: Mage provides a built-in visualization capability, allowing users to intuitively view the outputs generated by DBT models with just a few clicks.
4. Data extraction and loading: In addition to data transformation, Mage provides functionality for data extraction and loading, enabling a more comprehensive end-to-end data pipeline solution.
5. Pipeline scheduling and retry mechanism: Mage allows you to schedule your data pipelines and automatically retry failed components, ensuring smooth and reliable execution of your data integration processes.

Let’s delve deeper into each of these features.

Feel free to explore and experiment with the source code by cloning this GitHub repository:

install mage

You can install mage using docker, pip, or conda. This article will use Docker to install mage and initialize the project.

docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai /app/run_app.sh mage start (project_name)

For example, let’s name our project “dbt_mage”, so the command becomes:

docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai /app/run_app.sh mage start dbt_mage

Find other ways to install mage here.

create a pipeline

To view the Mage UI, open it in your browser.

Click “New” and select “Standard (Batch)” to create a new batch pipeline. Rename it as “dbt_pipeline”.

install dependencies

Since we will be using BigQuery as the data warehouse for dbt, we need to install dbt-bigquery By adding it to the “requirements.txt” file and clicking “Install Package”.

Create a DBT Project

To create a DBT project, navigate to the right panel and click on the Terminal button.

Go to “dbt” folder under your project and execute command dbt init,

cd dbt_mage/dbt 
dbt init demo -s

This command adds a “demo” folder to the dbt directory.

Right-click on the “demo” folder and create a new file called “profiles.yml”. Specify your BigQuery credentials in this file.