Python Channel

We are using Airflow as a scheduler. I want to invoke a simple bash operator in a DAG. The bash scr… Read more Store And Access Password Using Apache Airflow

I am trying to start my airflow webserver, but it says it is an unrecognised argument $ airflow web… Read more Airflow: Error: Unrecognized Arguments: Webserver

Working with airflow I try simple DAG work. I wrote custom operators and other files that I want to… Read more Airflow On Docker - Path Issue

We are using Airflow's KubernetesPodOperator for our data pipelines. What we would like to add … Read more Airflow: How To Use Trigger Parameters In Functions

I'm writing a python script that gets parameters from a json file, opens a template script spec… Read more Listing Variables From External Script Python Without The Built-in And Imported Variables

I am adding airflow to a web application that manually adds a directory containing business logic t… Read more How To Run Airflow With Celeryexecutor On A Custom Docker Image

I'm currently trying to write an airflow job that will allow me to ssh into an EC2 instance and… Read more Python Script To Ssh Into A Jumphost And Sftp From Within That Box

I have submitted a Hive job using Dataproc Workflow Template with the help of Airflow operator (Dat… Read more How To Get Jobid That Was Submitted Using Dataproc Workflow Template

I am using airflow for my data pipeline project. I have configured my project in airflow and start … Read more How Do I Restart Airflow Webserver?

I have an airflow environment locally deployed on WSL, and I am using VScode to debug and code. My … Read more Import Error In Vscode Despite Setting The Pythonpath

Trying to setup airflow using SQL Server as the backend, but get stuck during the initdb command: u… Read more Airflow - Initiation Of Db Stuck In Sql Server

I have a set of tasks in my dag builder module which uses Python operator as used worldwide in Airf… Read more Bashoperator Raising Importerror For A Lib Used In Other Pythonoperators

So my issue is that I build ETL pipelines in Airflow, but really develop and test the Extract, Tran… Read more Etl In Airflow Aided By Jupyter Notebooks And Papermill

I have created sample DAG, where I had DAG config as below. default_args = { 'owner': &… Read more "handling Signal: Ttou" Message While Running Dag In Airflow

class SFTPOperation(object): PUT = 'put' GET = 'get' operation=SFTPOpera… Read more How To Define Operations Of An Stfp Operator On Airflow?

With assistance from this answer https://stackoverflow.com/a/41730510/4200352 I am executing a pyth… Read more Airflow Python Script With Execution_date In Op_kwargs

So I have this quite nice DAG in airflow which basically runs several analysis steps (implemented a… Read more Run Airflow DAG For Each File

I have a Git repository which (among other things) holds Airflow DAGs in airflow directory. I have … Read more How To Use Non-installable Modules From DAG Code?

I am getting the following error : File ' ', line 1, in template TemplateSyntaxError: unex… Read more TemplateSyntaxError: Unexpected Char U'\\' At 205 Airflow Macros

I have a workflow where I have two parallel processes (sentinel_run and sentinel_skip) which should… Read more Airflow "none_failed" Skipping When Upstream Skips