Jump to content

Data Platform/Systems/Airflow/Developer guide/Addendum

From Wikitech

Reflections on AWight's first Airflow integration

2026-02 (google doc)

The Airflow devenv is powerful and intuitive.

Devenv connects to the production systems but as your user so watch out that scripts support alternate configurations for both production and development.  Actually, this felt like a weak spot since I had a hard time figuring this out and doing the switching.  I would add a level of overrides in which a base config is extended by prod and dev (also test if we're ready to talk about that).

It would be nice to link to examples of common airflow command line hacks, eg.

  • airflow-devenv show
  • airflow-devenv logs -f dev-awight gitsync
  • airflow-devenv exec --cmd "bash" airflow-dev-awight-scheduler-6c4897d578-l6wm4
  • airflow dags list-import-errors

What is the command line to force "reserialization of dags"?

Reserialization of DAGs is "refreshing job definitions" in plain English ffs.  This whole "dag" jargon is a strategic misstep IMO—crowing about "λ-calculus" is not going to popularize that phrase either.  I will call these jobs in polite conversation.

Props are great, the precedent is already to push as much as possible into this constant block.  Overriding variables from the UI was helpful for smoke-testing and quickly iterating in the devenv.  For example, to supply a newer build of my external resources.

But is it possible that the developer guide is overcomplicating the issue of VariableProperties?  It looks like our DagProperties already provides variable overrides, at least the variables system worked allowed me to conveniently override job props without touching VariableProperties directly.

External command integration is straightforward, run the target and its support as a bash command.  Supply the required object files in an artifact if possible, otherwise in HDFS.

Note that separate Airflow instances each have their own dag_config.py and artifacts.

Constructor arguments to custom Wikimedia operators, upstream Airflow operators, and Bash operators and sensors all different slightly, even for parameters such as command and bash_command; env vs append_env (watch out to not kill os.environ.copy); and resources which all have an identical or similar meaning for each subclass.

Outbound web proxy still needs to be investigated: I found that Skein vs. Bash operators seem to require different proxy settings, but this issue isn't confirmed yet.  It could be convenient to include the set_proxy script in executor images, to simplify how jobs enable egress.  The BashOperator should possibly have its egress closed unless an enabling label is present.

The CI is annoying tbh, I didn't get any value out of the strict expectations which just copy over the supplied properties and environment, although I can see good reasons to eventually include those files in the repo as documentation.  My preference would be that CI on ordinary dev patches only run * basic tests ensuring valid syntax * basic unit tests which don't "overfit" during development, and then a separate optional / final job before MR merge does all the strict stuff like making sure you have a static example of expected vars.

This trick for refreshing Kerberos when I got weird permissions errors (expired KRB5_CC for the service user):

sudo -u analytics-wmde kerberos-run-command analytics-wmde klist

You will have to generate tests, in both the airflow-dags and your instance's HQL scripts repos.  An open secret is that the "_test/t1_create*" scripts can be used to create tables in your user schema ahead of running a job in the devenv:

(analytics-wmde flavored, this is in the docs)

python3 hql/gen_hql_test_scripts.py -d hql/airflow_jobs/wiki_page_cite_references

spark3-sql -f hql/airflow_jobs/wiki_page_cite_references/_test/t1_create_table_wiki_page_cite_references_raw.hql

Iceberg is the future but currently it has a buglike behavior of getting into corrupt states when its metadata files are mistreated or missing.  Upstream has several open and closed bugs about this but the problem persists.  Tricks we had to use include:

- Copy metadata into the missing location, if you have an exact match available.

- Recursively remove stale directories if they block re-creating a table.  The way to avoid this is to drop table T purge in the first place.

- To drop a table even when its metadata is complaining:

spark.sharedState.externalCatalog.dropTable("default", "wiki_page_cite_references_monthly", false, false)

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Data_Platform/Systems/Airflow/Developer_guide/Addendum&oldid=2385082"