How to connect to a Databricks Cluster from a local IDE
Use databricks-connect to run code on a Databricks cluster from your local IDE. Covers setup, IDE integration, dbutils, notebooks via CLI, and limitations.
Michał Milik
Blog - Page 15
How to improve Airflow 2.0 performance with Smart Sensors?
Compare Azure Data Factory and Apache Airflow: source integration, workflow management, data transformation, and scheduling flexibility for your needs.
How to simplify Airflow 2.0 setup with the DAG Versioning and Serialization
Airflow 2.0 added DAG serialization and versioning. See how these features simplify setup and deployment and keep a history of changes to your pipelines.
How to spin up to 1000 parallel Airflow 2.0 tasks in 5 minutes from nothing with the CeleryExecutor and Kubernetes
Run up to 1000 parallel tasks in Apache Airflow 2.0 within 5 minutes by combining the CeleryExecutor with Kubernetes. Setup steps from a clean start.
Improve your python code quality – a python linters overview
Python linters check code against PEP 8 and catch errors early. Compare Pylint, Flake8, and PyFlakes to find the right balance of depth and simplicity.
Introduction to data quality: terms, definitions & examples of use
Data quality basics: the six dimensions from accuracy to timeliness, plus how to assess, monitor, and improve the data your business relies on.
Powerful REST API in Airflow 2.0 — what do you need to know?
Discover the benefits of Apache Airflow 2.0, including REST API, efficient scheduling, and smart sensors. Learn how it improves workflow management.
Python „Code check” implementation in IDE + Github action
Use flake8 to catch Python errors and style issues before the code even runs, both in your IDE and in a GitHub Actions workflow that checks every push.
Michał Milik
Streaming Data Architecture: Components, Patterns & Benefits
Streaming data architecture explained: core components, Lambda vs Kappa patterns, key design trade-offs, and the business benefits of real-time data.
Marcin Boruch
Test Driven Development in Python using Pytest
Write tests before code with pytest: a Test Driven Development workflow in Python, from project structure to a worked example.
The difference between Big Data and Data Analytics
Big Data vs Data Analytics: the two sound similar but solve different problems. Learn what sets them apart and when your data team should use each one.
The new, redesigned user interface in Airflow 2.0
Airflow 2.0 ships a redesigned UI, a faster scheduler, smart sensors and a full REST API. A walkthrough of what changed for data engineering teams.