Airflow

What is Airflow?

Airflow is an open-source platform for developing, scheduling, and monitoring batch-oriented workflows. Airflow’s extensible Python framework enables you to build workflows connecting with virtually any technology. A web interface helps manage the state of your workflows. Airflow is deployable in many ways, varying from a single process on your laptop to a distributed setup to support even the biggest workflows.

• Links to Airflow Documentation

Installation Instructions

To install airflow simply pip install "RegScale-CLI[airflow]".

RegScale-CLI Airflow Customizations

The configs and scripts held within this folder are example scripts using the SDK, and are set up to make the deployment
of Airflow for RegScale much simpler, so that your compliance can be left shifted automatically beyond belief!

DAGs

These DAGs are provided as examples for what can be done with RegScale for Airflow. For customized DAGs to meet your
specific needs, contact our sales team.

RegScale-CLI as Airflow PythonOperators

Through the convenience of structuring our CLI via click, we can now dynamically generate
Airflow's PythonOperator to execute a CLI command.

These operators and default arguments can be found in
from regscale.airflow.hierarchy's AIRFLOW_CLICK_OPERATORS.
Below is an example DAG to query CISA and then send reminders.

Python

"""An example RegScale-CLI based DAG."""
from airflow import DAG

from regscale.airflow.config import DEFAULT_ARGS, yesterday
from regscale.airflow.hierarchy import AIRFLOW_CLICK_OPERATORS as OPERATORS
from regscale.airflow.tasks.groups import setup_task_group


dag = DAG(
    'query_cisa_and_send_reminders',
    default_args=DEFAULT_ARGS,
    description="Query CISA and send reminders",
    schedule_interval='@daily',
    start_date=yesterday(),
    is_paused_upon_creation=False,
)

setup = setup_task_group(dag)

cisa_alerts = OPERATORS['cisa__ingest_cisa_alerts']['lambda'](dag=dag)
cisa_kev = OPERATORS['cisa__ingest_cisa_kev']['lambda'](dag=dag)
send_reminders = OPERATORS['admin_actions__send_reminders']['lambda'](dag=dag)

setup >> [cisa_alerts, cisa_kev] >> send_reminders

In the above example, we use double underscores __ (dunderscores) to simulate spaces between the command groups passed
to the RegScale-CLI entrypoint, so no need to use regscale about, we can just index the key about.

Under the hood, we call a python lambda that is a factory function for creating a dynamic PythonOperator object
available for use within a DAG. The pre-built setup_task_group is also a factory function for generating a setup task
group. The group will share the same first 8 characters from a UUID4, allowing the setups to avoid collision when
multiple setup tasks may be used.

Contact Information

To contact us regarding customizing your shift left compliance via Airflow, contact us.

License Information

This is limited to RegScale Enterprise on the platform side, but we provide an airflow image for use by unlicensed users.