#67 Airflow (part 3): Conditional tasks by weekday

Linear Pipeline

Also known as static pipeline, it performs fixed functions and later they are linked together. However, the drawback is that each task will be executed linearly, not sequentially or in other words, one after another. There are certain situation where we need to go beyond this pipeline design, for instance some tasks are conditioned by the result of execution of other tasks. This is when we need to put dynamic pipeline into action.

Dynamic pipeline

Basically what a dynamic workflow in Airflow means is that you have a DAG with a bunch of tasks inside of it and depending on some outside parameters (that also aren’t known at the time the DAG starts) will sometimes create different amounts and/or types of tasks.

For example:

Monday: load the data -> Tuesday: process data …. -> Sunday: wrap up to generate a report for Monday meeeting. In this situation, the execution of the workflow cannot be linear as it depends on external parameters, in this case is the week day.

Let’s create such DAG then!

Writing DAG python file in action

Requirement

• VS Code latest version
• Docker Desktop (with this docker-compose.yml file). Set up your own Fernent key before making any connection. Change the database if needed. Set up own password and username for…