render_template_as_native_obj = True, but configurable task by task

[sirVir]

Hi.

I have extended the KubernetesPodOperator adding some extra properties to be templated, so I can configure them through Airflow variables:

class CustomKubernetesPodOperator(KubernetesPodOperator):
    template_fields = KubernetesPodOperator.template_fields + (
        "is_delete_operator_pod",
        "log_events_on_failure",
    )

    def __init__(
     ...
    ):
        ...
        self.is_delete_operator_pod = "{{ var.value.is_delete_operator_pod }}"
        self.log_events_on_failure = "{{ var.value.log_events_on_failure }}"
         ...
        super().__init__(
           ...
            log_events_on_failure=self.log_events_on_failure,
            is_delete_operator_pod=self.is_delete_operator_pod,
            *args,
            **kwargs,
        )

The parent class requires log_events_on_failure and is_delete_operator_pod arguments to be boolean.

The arguments have correct types, but only when DAG is initialized with render_template_as_native_obj = True on the DAG-level. Otherwise they will always be True, as "False" string evaluates to True boolean.

This is not ideal, as sometimes we would like to render some of the Tasks with dumb strings and some other with proper types. Also, it would make it easier to have the operator working as intended without the need of special configuration on the DAG level. So we would generally want to be able to configure whether the templated fields are rendered as native objects or string, task by task.

Currently, I see no way of making it work. I tried overriding the render_template_fields method on in my CustomKubernetesPodOperator, to enforce jinja_env = airflow.templates.NativeEnvironment() but during the rendering of the task instance, the value of render_template_as_native_obj from the DAG is still used and breaks:

ERROR [airflow.task.operators] Exception rendering Jinja template for task 'my_custom_operator_task', field 'is_delete_operator_pod'. Template: '{{ var.value.is_delete_operator_pod }}'
Traceback (most recent call last):
  File "/.../env/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 602, in _do_render_template_fields
    rendered_content = self.render_template(
  File "/.../env/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 663, in render_template
    return render_template_to_string(template, context)
  File "/.../env/lib/python3.9/site-packages/airflow/utils/helpers.py", line 288, in render_template_to_string
    return render_template(template, cast(MutableMapping[str, Any], context), native=False)
  File "/.../env/lib/python3.9/site-packages/airflow/utils/helpers.py", line 283, in render_template
    return "".join(nodes)
TypeError: sequence item 0: expected str instance, bool found

Is there something I am missing to achieve the behavior I would like to see? Is it reasonable to ask for more granular control over what is rendered in what way as a Feature request?

[potiuk]

No. This control is per DAG. Sure. Opening a PR fixing it makes sense, I also personally think render_template_as_native_object could be a task parameter. But not sure if others will agree Maybe you can even contribute it. proposing a PR is the best way to start a discussion on it - you can also create an issue which hopefully somoene will pick up and implement, but implementing it on your own is the most certain way it will be looked at.

[sirVir]

I started as a discussion because I could have missed something, and maybe there is a way of making it work with current setup?

I have run into a corner case, where I want to do "dumb" string templating for some fields, and "typed" for others. I tried many workarounds and cannot make it work:

• when I set the render_template_as_native_obj = True on my DAG, then it will try to interpret all the strings, meaning that I cannot render, for example "('filename.txt')" because it will be replaced by "filename.txt"
• vice-versa, when I set the render_template_as_native_obj = False, then it breaks my custom operator. I tried overriding the render_template_fields in my custom operator method like that:

from distutils.util import strtobool
...
class CustomKubernetesPodOperator(KubernetesPodOperator):
...
        self.is_delete_operator_pod = "{{ var.value.is_delete_operator_pod | bool }}"
        self.log_events_on_failure = "{{ var.value.log_events_on_failure | bool }}"
...
    def render_template_fields(
        self,
        context,
        jinja_env = None,
    ) -> None:
        if not jinja_env:
            jinja_env = self.get_template_env()
        jinja_env.filters['bool'] = strtobool
        self._do_render_template_fields(self, self.template_fields, context, jinja_env, set())

but it doesn't help, as the templating is configured globally and the setting conflicts later.

If it is indeed impossible to achieve with the way the templating works in Airflow currently, I think it would be a very useful feature.

[potiuk]

Impossible as far as I see. And yes it would be great to add it.

[dejii]

Encountered this same scenario. I've opened up a PR to move this discussion forward #60619