Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is a managed service for Apache Airflow that lets you orchestrate knowledge pipelines and workflows at scale. With Amazon MWAA, you possibly can design Directed Acyclic Graphs (DAGs) that describe your workflows with out managing the operational burden of scaling the infrastructure. On this publish, we offer steering on how one can optimize efficiency and save price by following greatest practices.

Amazon MWAA environments embrace 4 Airflow elements hosted on teams of AWS compute sources: the scheduler that schedules the work, the employees that implement the work, the net server that gives the UI, and the metadata database that retains monitor of state. For intermittent or various workloads, optimizing prices whereas sustaining value and efficiency is essential. This publish outlines greatest practices to realize price optimization and environment friendly efficiency in Amazon MWAA environments, with detailed explanations and examples. It might not be mandatory to use all of those greatest practices for a given Amazon MWAA workload; you possibly can selectively select and implement related and relevant ideas to your particular workloads.

Proper-sizing your Amazon MWAA setting

Proper-sizing your Amazon MWAA setting makes certain you have got an setting that is ready to concurrently scale throughout your totally different workloads to supply the very best price-performance. The setting class you select to your Amazon MWAA setting determines the scale and the variety of concurrent duties supported by the employee nodes. In Amazon MWAA, you possibly can select from 5 totally different setting courses. On this part, we focus on the steps you possibly can observe to right-size your Amazon MWAA setting.

Monitor useful resource utilization

Step one in right-sizing your Amazon MWAA setting is to observe the useful resource utilization of your current setup. You may monitor the underlying elements of your environments utilizing Amazon CloudWatch, which collects uncooked knowledge and processes knowledge into readable, close to real-time metrics. With these setting metrics, you have got higher visibility into key efficiency indicators that can assist you appropriately dimension your environments and debug points together with your workflows. Based mostly on the concurrent duties wanted to your workload, you possibly can modify the setting dimension in addition to the utmost and minimal staff wanted. CloudWatch will present CPU and reminiscence utilization for all of the underlying AWS providers make the most of by Amazon MWAA. Consult with Container, queue, and database metrics for Amazon MWAA for added particulars on accessible metrics for Amazon MWAA. These metrics additionally embrace the variety of base staff, further staff, schedulers, and net servers.

Analyze your workload patterns

Subsequent, take a deep dive into your workflow patterns. Look at DAG schedules, process concurrency, and process runtimes. Monitor CPU/reminiscence utilization throughout peak intervals. Question CloudWatch metrics and Airflow logs. Determine long-running duties, bottlenecks, and resource-intensive operations for optimum setting sizing. Understanding the useful resource calls for of your workload will assist you to make knowledgeable choices concerning the applicable Amazon MWAA setting class to make use of.

Select the precise setting class

Match necessities to Amazon MWAA setting class specs (mw1.small to mw1.2xlarge) that may deal with your workload effectively. You may vertically scale up or scale down an current setting by means of an API, the AWS Command Line Interface (AWS CLI), or the AWS Administration Console. Bear in mind {that a} change within the setting class requires a scheduled downtime.

Wonderful tune configuration parameters

Wonderful-tuning configuration parameters in Apache Airflow is essential for optimizing workflow efficiency and price reductions. It lets you tune settings equivalent to Auto scaling, parallelism, logging, and DAG code optimizations.

Auto scaling

Amazon MWAA helps employee auto scaling, which mechanically adjusts the variety of operating employee and net server nodes based mostly in your workload calls for. You may specify the minimal and most variety of Airflow staff that run in your setting. For employee node auto scaling, Amazon MWAA makes use of RunningTasks and QueuedTasks metrics, the place (duties operating + duties queued) / (duties per employee) = (required staff). If the required variety of staff is bigger than the present variety of operating staff, Amazon MWAA will add further employee cases utilizing AWS Fargate, as much as the utmost worth specified by the utmost employee configuration.

Auto scaling in Amazon MWAA will gracefully downscale when there are extra further staff than required. For instance, let’s assume a big Amazon MWAA setting with a minimal of 1 employee and a most of 10, the place every massive Amazon MWAA employee can assist as much as 20 duties. Let’s say, every day at 8:00 AM, DAGs begin up that use 190 concurrent duties. Amazon MWAA will mechanically scale to 10 staff, as a result of the required staff = 190 requested duties (some operating, some queued) / 20 (duties per employee) = 9.5 staff, rounded as much as 10. At 10:00 AM, half of the duties full, leaving 85 operating. Amazon MWAA will then downscale to six staff (95 duties/20 duties per employee = 5.25 staff, rounded as much as 6). Any staff which can be nonetheless operating duties stay protected throughout downscaling till they’re full, and no duties will probably be interrupted. Because the queued and operating duties lower, Amazon MWAA will take away staff with out affecting operating duties, all the way down to the minimal specified employee depend.

Net server auto scaling in Amazon MWAA lets you mechanically scale the variety of net servers based mostly on CPU utilization and lively connection depend. Amazon MWAA makes certain your Airflow setting can seamlessly accommodate elevated demand, whether or not from REST API requests, AWS CLI utilization, or extra concurrent Airflow UI customers. You may specify the utmost and minimal net server depend whereas configuring your Amazon MWAA setting.

Logging and metrics

On this part, we focus on the steps to pick out and set the suitable log configurations and CloudWatch metrics.

Select the precise log ranges

If enabled, Amazon MWAA will ship Airflow logs to CloudWatch. You may view the logs to find out Airflow process delays or workflow errors with out the necessity for added third-party instruments. It’s essential to allow logging to view Airflow DAG processing, duties, scheduler, net server, and employee logs. You may allow Airflow logs on the INFO, WARNING, ERROR, or CRITICAL degree. Whenever you select a log degree, Amazon MWAA sends logs for that degree and better ranges of severity. Commonplace CloudWatch logs costs apply, so decreasing log ranges the place attainable can scale back total prices. Use essentially the most applicable log degree based mostly on setting, equivalent to INFO for dev and UAT, and ERROR for manufacturing.

Set applicable log retention coverage

By default, logs are stored indefinitely and by no means expire. To scale back CloudWatch price, you possibly can modify the retention coverage for every log group.

Select required CloudWatch metrics

You may select which Airflow metrics are despatched to CloudWatch through the use of the Amazon MWAA configuration possibility metrics.statsd_allow_list. Consult with the whole record of obtainable metrics. Some metrics equivalent to schedule_delay and duration_success are revealed per DAG, whereas others equivalent to ti.end are revealed per process per DAG.

Due to this fact, the cumulative variety of DAGs and duties instantly affect your CloudWatch metric ingestion prices. To regulate CloudWatch prices, select to publish selective metrics. For instance, the next will solely publish metrics that begin with scheduler and executor:

metrics.statsd_allow_list = scheduler,executor

We advocate utilizing metrics.statsd_allow_list with metrics.metrics_use_pattern_match.

An efficient apply is to make the most of common expression (regex) sample matching in opposition to the whole metric title as a substitute of solely matching the prefix initially of the title.

Monitor CloudWatch dashboards and arrange alarms

Create a customized dashboard in CloudWatch and add alarms for a specific metric to observe the well being standing of your Amazon MWAA setting. Configuring alarms lets you proactively monitor the well being of the setting.

Optimize AWS Secrets and techniques Supervisor invocations

Airflow has a mechanism to retailer secrets and techniques equivalent to variables and connection info. By default, these secrets and techniques are saved within the Airflow meta database. Airflow customers can optionally configure a centrally managed location for secrets and techniques, equivalent to AWS Secrets and techniques Supervisor. When specified, Airflow will first verify this alternate secrets and techniques backend when a connection or variable is requested. If the alternate backend comprises the wanted worth, it’s returned; if not, Airflow will verify the meta database for the worth and return that as a substitute. One of many elements affecting the associated fee to make use of Secrets and techniques Supervisor is the variety of API calls made to it.

On the Amazon MWAA console, you possibly can configure the backend Secrets and techniques Supervisor path for the connections and variables that will probably be utilized by Airflow. By default, Airflow searches for all connections and variables within the configured backend. To scale back the variety of API calls Amazon MWAA makes to Secrets and techniques Supervisor in your behalf, configure it to make use of a lookup sample. By specifying a sample, you slim the attainable paths that Airflow will have a look at. This may assist in decreasing your prices when utilizing Secrets and techniques Supervisor with Amazon MWAA.

To make use of a secrets and techniques cache, allow AIRFLOW_SECRETS_USE_CACHE with TTL to assist to cut back the Secrets and techniques Supervisor API calls.

For instance, if you wish to solely lookup a particular subset of connections, variables, or config in Secrets and techniques Supervisor, set the related *_lookup_pattern parameter. This parameter takes a regex as a string as worth. To lookup connections beginning with m in Secrets and techniques Supervisor, your configuration file ought to seem like the next code:

[secrets]
backend = airflow.suppliers.amazon.aws.secrets and techniques.secrets_manager.SecretsManagerBackend
backend_kwargs =

{
  "connections_prefix": "airflow/connections",
  "connections_lookup_pattern": "^m",
  "profile_name": "default"
}

DAG code optimization

Schedulers and staff are two elements which can be concerned in parsing the DAG. After the scheduler parses the DAG and locations it in a queue, the employee picks up the DAG from the queue. On the level, all of the employee is aware of is the DAG_id and the Python file, together with another data. The employee has to parse the Python file with a view to run the duty.

DAG parsing is run twice, as soon as by the scheduler after which by the employee. As a result of the employees are additionally parsing the DAG, the period of time it takes for the code to parse dictates the variety of staff wanted, which provides price of operating these staff.

For instance, for a complete of 200 DAGs having 10 duties every, taking 60 seconds per process to parse, we are able to calculate the next:

• Complete duties throughout all DAGs = 2,000
• Time per process = 60 seconds + 20 seconds (parse DAG)
• Complete time = 2000 * 80 = 160,000 seconds
• Complete time per employee = 72,000 seconds
• Variety of staff wants = Complete time/Complete time per employee = 160,000/72,000 = ~3

Now, let’s enhance the time taken to parse the DAGs to 100 seconds:

• Complete duties throughout all DAGs = 2,000
• Time per process = 60 seconds + 100 seconds
• Complete time = 2,000 *160 = 320,000 seconds
• Complete time per employee = 72,000 seconds
• Variety of staff wants = Complete time/Complete time per employee = 320,000/72,000 = ~5

As you possibly can see, when the DAG parsing time elevated from 20 seconds to 100 seconds, the variety of employee nodes wanted elevated from 3 to five, thereby including compute price.

To scale back the time it takes for parsing the code, observe the very best practices within the subsequent sections.

Take away top-level imports

Code imports will run each time the DAG is parsed. For those who don’t want the libraries being imported to create the DAG objects, transfer the import to the duty degree as a substitute of defining it on the high. After it’s outlined within the process, the import will probably be referred to as solely when the duty is run.

Keep away from a number of calls to databases just like the meta database or exterior system database. Variables are used inside the DAG which can be outlined within the meta database or a backend system like Secrets and techniques Supervisor. Use templating (Jinja) whereby calls to populate the variables are solely made at process runtime and never at process parsing time.

For instance, see the next code:

import pendulum
from airflow import DAG
from airflow.decorators import process
import numpy as np  # <-- DON'T DO THAT!

with DAG(
    dag_id="example_python_operator",
    schedule=None,
    start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
    catchup=False,
    tags=["example"],
) as dag:

    @process()
    def print_array():
        """Print Numpy array."""
        import numpy as np  # <-- INSTEAD DO THIS!
        a = np.arange(15).reshape(3, 5)
        print(a)
        return a
    print_array()

The next code is one other instance:

# Unhealthy instance
from airflow.fashions import Variable

foo_var = Variable.get("foo")  # DON'T DO THAT

bash_use_variable_bad_1 = BashOperator(
    task_id="bash_use_variable_bad_1", bash_command="echo variable foo=${foo_env}", env={"foo_env": foo_var}
)

bash_use_variable_bad_2 = BashOperator(
    task_id="bash_use_variable_bad_2",
    bash_command=f"echo variable foo=${Variable.get('foo')}",  # DON'T DO THAT
)

bash_use_variable_bad_3 = BashOperator(
    task_id="bash_use_variable_bad_3",
    bash_command="echo variable foo=${foo_env}",
    env={"foo_env": Variable.get("foo")},  # DON'T DO THAT
)

# Good instance
bash_use_variable_good = BashOperator(
    task_id="bash_use_variable_good",
    bash_command="echo variable foo=${foo_env}",
    env={"foo_env": "{{ var.worth.get('foo') }}"},
)

@process
def my_task():
    var = Variable.get("foo")  # that is wonderful, as a result of func my_task referred to as solely run process, not scan DAGs.
print(var)

Writing DAGs

Advanced DAGs with a lot of duties and dependencies between them can affect efficiency of scheduling. One option to hold your Airflow occasion performant and nicely utilized is to simplify and optimize your DAGs.

For instance, a DAG that has easy linear construction A → B → C will expertise much less delays in process scheduling than a DAG that has a deeply nested tree construction with an exponentially rising variety of dependent duties.

Dynamic DAGs

Within the following instance, a DAG is outlined with hardcoded desk names from a database. A developer has to outline N variety of DAGs for N variety of tables in a database.

# Unhealthy instance
dag_params = getData()
no_of_dags = int(dag_params["no_of_dags"]['N'])
# construct a dag for every quantity in no_of_dags
for n in vary(no_of_dags):
    dag_id = 'dynperf_t1_{}'.format(str(n))
default_args = {'proprietor': 'airflow','start_date': datetime(2022, 2, 2, 12, n)}

To scale back verbose and error-prone work, use dynamic DAGs. The next definition of the DAG is created after querying a database catalog, and creates as many DAGs dynamically as there are tables within the database. This achieves the identical goal with much less code.

def getData():
    shopper = boto3.shopper('dynamodb’)
    response = shopper.get_item(
        TableName="mwaa-dag-creation",
        Key={'key': {'S': 'mwaa’}}
    )
    return response["Item"]

Stagger DAG schedules

Operating all DAGs concurrently or inside a brief interval in your setting can lead to the next variety of employee nodes required to course of the duties, thereby rising compute prices. For enterprise situations the place the workload will not be time-sensitive, contemplate spreading the schedule of DAG runs in a approach that maximizes the utilization of obtainable employee sources.

DAG folder parsing

Less complicated DAGs are often solely in a single Python file; extra complicated DAGs could be unfold throughout a number of information and have dependencies that needs to be shipped with them. You may both do that all within the DAG_FOLDER , with a regular filesystem structure, or you possibly can bundle the DAG and all of its Python information up as a single .zip file. Airflow will look into all of the directories and information within the DAG_FOLDER. Utilizing the .airflowignore file specifies which directories or information Airflow ought to deliberately ignore. This may enhance the effectivity of discovering a DAG inside a listing, bettering parsing occasions.

Deferrable operators

You may run deferrable operators on Amazon MWAA. Deferrable operators have the flexibility to droop themselves and unlock the employee slot. No duties within the employee means fewer required employee sources, which may decrease the employee price.

For instance, let’s assume you’re utilizing a lot of sensors that look forward to one thing to happen and occupy employee node slots. By making the sensors deferrable and utilizing employee auto scaling enhancements to aggressively downscale staff, you’ll instantly see an affect the place fewer employee nodes are wanted, saving on employee node prices.

Dynamic Job Mapping

Dynamic Job Mapping permits a approach for a workflow to create quite a few duties at runtime based mostly on present knowledge, slightly than the DAG creator having to know prematurely what number of duties can be wanted. That is much like defining your duties in a for loop, however as a substitute of getting the DAG file fetch the info and try this itself, the scheduler can do that based mostly on the output of a earlier process. Proper earlier than a mapped process is run, the scheduler will create N copies of the duty, one for every enter.

Cease and begin the setting

You may cease and begin your Amazon MWAA setting based mostly in your workload necessities, which can end in price financial savings. You may carry out the motion manually or automate stopping and beginning Amazon MWAA environments. Consult with Automating stopping and beginning Amazon MWAA environments to cut back price to discover ways to automate the cease and begin of your Amazon MWAA setting retaining metadata.

Conclusion

In conclusion, implementing efficiency optimization greatest practices for Amazon MWAA can considerably scale back total prices whereas sustaining optimum efficiency and reliability. Key methods embrace right-sizing setting courses based mostly on CloudWatch metrics, managing logging and monitoring prices, utilizing lookup patterns with Secrets and techniques Supervisor, optimizing DAG code, and selectively stopping and beginning environments based mostly on workload calls for. Repeatedly monitoring and adjusting these settings as workloads evolve can maximize your cost-efficiency.

In regards to the Authors

Sriharsh Adari is a Senior Options Architect at AWS, the place he helps clients work backward from enterprise outcomes to develop revolutionary options on AWS. Over time, he has helped a number of clients on knowledge platform transformations throughout business verticals. His core space of experience contains expertise technique, knowledge analytics, and knowledge science. In his spare time, he enjoys enjoying sports activities, binge-watching TV reveals, and enjoying Tabla.

Retina Satish is a Options Architect at AWS, bringing her experience in knowledge analytics and generative AI. She collaborates with clients to grasp enterprise challenges and architect revolutionary, data-driven options utilizing cutting-edge applied sciences. She is devoted to delivering safe, scalable, and cost-effective options that drive digital transformation.

Jeetendra Vaidya is a Senior Options Architect at AWS, bringing his experience to the realms of AI/ML, serverless, and knowledge analytics domains. He’s captivated with aiding clients in architecting safe, scalable, dependable, and cost-effective options.