Most engineers in their professional life will have to deal with data integrations. In the past few years, a few companies such as Fivetran and StitchData have emerged for batch-based integrations, and Segment for event-based ones. But none of these companies…

CDC pipeline guide using Azure DataFactory with Azure DataBricks Delta Lake’s change data feed

Dbt tests are great, but sometimes might not go far enough. When and how can you level up your dbt testing capabilities?

See how to leverage the Airflow ShortCircuitOperator to create data circuit breakers to prevent bad data from reaching your data pipelines.

In this article, I will be showing you how to build a reliable data pipeline for your small business to improve your productivity and data security.

Streamline your data pipeline by enabling non-engineers to define validation logic, review data, and fix issues.

This article will share architectural schemes for building a system that collects data from Ethereum blockchain and runs a real-time analysis of Defi token data

Learn how to manage and debug data pipelines in Airflow with real-world practical examples. Use the Grid View for observability and manual debugging.

Learn how to stream data efficiently from a REST API into a Kafka topic using Kafka Connect.

A small modern data stack that ETLs data from a PostgreSQL database into a ClickHouse database.

Learn the basics of data engineering with a practical ETL pipeline project. Explore how weather, flight, city data are extracted, transformed, loaded into a DB.