Jump to content

Dumps/Airflow/Operations

From Wikitech

Making a change to the SQL/XML Dump DAGs

These DAGs are specific in the way they are defined: they are code-generated instead of being manually written. The reason for that is: we used to define all of the DAGs in a single python file, the parsing of which was taking about 2 minutes. Not only was it a hotspot in the DAG parsing phase, but it was also causing the DAGs to temporarily disappear while that parsing was occurring.

To take full advantage of the DAG parsing parallelism (by default, we have twice as many DAG parsing workers as available CPUs in the scheduler pod), we decided to code-generate the XML/SQL dag files, each of these files only defining a single DAG.

See https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1625 for the gory details.

This means that if you want to make a change to these DAGs, you need to make a change to the DAG template file and/or the code generation script, and regenerate the DAG files.

  • The code generation script is in charge of defining the wikis in scope for each DAG, the DAG keyword arguments, the Airflow pool, etc, and renders each DAG file by injecting these parameters into the DAG file template
  • The DAG file template contains the actual DAG, with string template plaeholders for each of the parameters injected by the code generation script

For example, let's assume we'd like to remove the wip=True DAG keyword argument from each of these DAGs. To do this, we would remove it from the code generation file.

diff --git a/scripts/generate_dumps_sql_xml_dags.py b/scripts/generate_dumps_sql_xml_dags.py
index a3c28060..a152e8b9 100755
--- a/scripts/generate_dumps_sql_xml_dags.py
+++ b/scripts/generate_dumps_sql_xml_dags.py
@@ -104,7 +104,6 @@ common_dag_kwargs = {
     "default_args": {
         "email": DUMPS_ALERTS_RECIPIENT,
     },
-    "wip": True,
 }

 RegularSqlXmlDumps = Dumps(

We would then regenerate all DAG files, which will also call black and isort on the generated DAG files.

~/wmf/airflow-dags T406874 *14 !3 ?2 ❯ make dumps/dags/sql_xml
reformatted /Users/brouberol/wmf/airflow-dags/dumps/dags/sql_xml/mediawiki_dumps_sql_xml_large_a_to_z_full.py
...
reformatted /Users/brouberol/wmf/airflow-dags/dumps/dags/sql_xml/mediawiki_dumps_sql_xml_regular_t_to_v_partial.py

All done! ✨ 🍰 ✨
20 files reformatted, 1 file left unchanged.
Fixing /Users/brouberol/wmf/airflow-dags/dumps/dags/sql_xml/mediawiki_dumps_sql_xml_regular_l_to_m_full.py
...
Fixing /Users/brouberol/wmf/airflow-dags/dumps/dags/sql_xml/mediawiki_dumps_sql_xml_regular_a_to_b_full.py

You'll see your change being reflected in all DAGs (partial diff for clarity):

diff --git a/dumps/dags/sql_xml/mediawiki_dumps_sql_xml_regular_a_to_b_partial.py b/dumps/dags/sql_xml/mediawiki_dumps_sql_xml_regular_a_to_b_partial.py
index 07170f48..ae42caf5 100644
--- a/dumps/dags/sql_xml/mediawiki_dumps_sql_xml_regular_a_to_b_partial.py
+++ b/dumps/dags/sql_xml/mediawiki_dumps_sql_xml_regular_a_to_b_partial.py
@@ -271,7 +271,6 @@ dag_kwargs = {
     "max_active_runs": 1,
     "max_active_tasks": 32,
     "default_args": {"email": "data-platform-alerts@wikimedia.org"},
-    "wip": True,
 }
 dag_kwargs["schedule"] = PARTIAL_DUMP_SCHEDULE
 dag_kwargs["user_defined_filters"] = filters
diff --git a/dumps/dags/sql_xml/mediawiki_dumps_sql_xml_regular_c_to_e_full.py b/dumps/dags/sql_xml/mediawiki_dumps_sql_xml_regula`r_c_to_e_full.py
index 68d293ee..3343974b 100644
--- a/dumps/dags/sql_xml/mediawiki_dumps_sql_xml_regular_c_to_e_full.py
+++ b/dumps/dags/sql_xml/mediawiki_dumps_sql_xml_regular_c_to_e_full.py
@@ -247,7 +247,6 @@ dag_kwargs = {
     "max_active_runs": 1,
     "max_active_tasks": 32,
     "default_args": {"email": "data-platform-alerts@wikimedia.org"},
-    "wip": True,
 }
 dag_kwargs["schedule"] = FULL_DUMP_SCHEDULE
 dag_kwargs["user_defined_filters"] = filters

Once you're done, open a merge request with the change in the code generation script / template and the generated DAG files, which will then get automatically deployed when the MR is merged.

Re-running a failed dump command

TODO

Re-running a failed sync command

TODO

I'm getting paged

The dumps CephFS volume is filling up


You can dynamically resize the CephFS volume by adjusting the dumps.persistence.size value in deployment-charts/helmfile.d/dse-k8s-services/mediawiki-dumps-legacy/values.yaml. Send a patch, get it reviewed, and then ssh to the deployment server to run

% ssh deployment.eqiad.wmnet
brouberol@deploy1003:~$ cd /srv/deployment-charts/helmfile.d/dse-k8s-services/mediawiki-dumps-legacy
brouberol@deploy1003:~$ helmfile -e dse-k8s-services -i apply
Do not live edit the PVC size, as it would get reverted at the next helmfile apply (or at least the helmfile apply would fail as the operator might refuse to downsize the volume).

The fetch_wiki_list_from_noc dag is failing

If this dag is failing, it means that https://noc.wikimedia.org exposes different list of wikis than the one we have in airflow-dags.

If that is the case, it means that either the next dump dag run would be dumping a wiki that has been removed, or wouldn't be dumping a wiki that was recently added. To fix this issue, run the following script in your local airflow-dags checkout:

make dumps/dags/sql_xml

Send a merge request with the changes. The new wiki lists will be automatically deployed to airflow once the MR is merged.

I need to clear a dump wiki lock

Exec into the mediawiki-dumps-legacy-deploy toolbox (extra permission needed here beyond mediawiki-dumps-legacy in the fashion described at Kubernetes/Deployments#Advanced use cases: using kubectl) to remove the lock file.

% ssh deployment.eqiad.wmnet
brouberol@deploy1003:~$ kube_env mediawiki-dumps-legacy-deploy dse-k8s-eqiad
brouberol@deploy1003:~$ kubectl exec -it $(kubectl get pod -l component=toolbox --no-headers -o custom-columns=":metadata.name") -- bash
www-data@mediawiki-dumps-legacy-toolbox-66b8dc5599-x29r2:/$ rm /mnt/dumpsdata/xmldatadumps/private/<wiki>/<date>/lock_*

Sometimes after clearing a dump wiki lock and attempting to resume by clearing the specific failed task in Airflow (which you find by clicking the wiki group chevron to see all tasks in the wiki group to identify the specific failed task), another failure will nevertheless result. Be sure to examine this repeated failure in the Airflow log, and note that you may need to additionally do the following.

  • (ceph) delete anything in /mnt/dumpsdata/xmldatadumps/temp/<first_letter_of_wiki>/<wiki>
  • (ceph) delete everything under /mnt/dumpsdata/xmldatadumps/public/<wiki>/<yyyymmdd_of_dump>
  • (ceph) delete everything under /mnt/dumpsdata/xmldatadumps/private/wiki/<yyyymmdd_of_dump>
  • (on each clouddumps host) delete everything under /srv/dumps/xmldatadumps/public/<wiki>/<yyyymmdd_of_dump>/
  • clear out the whole task group (i.e., don't just clear the specific failed task, but rather the parent group under which the specific failed task exists)

On 6-October-2026, for example, the following was needed due to an Airflow alert with subject:"[data platform alerts] Airflow alert: <TaskInstance: mediawiki_dumps_sql_xml_large_a_to_z_full.enwiki.dump_remaining_full scheduled__2026-09-01T08:00:00+00:00 [failed]>" , with the DAG running on airflow-dumps.wikimedia.org. Clearing of the lockfile and clearing of the specific failed task (which in this case was the dump_remaining_full task for the enwiki "group") had resulted in another failure, and so it was necessary to do the fuller set of file deletions.

Note: Although k8s access will provide deletion privileges for some files, in order to delete files on dump servers (clouddumps1001.wikimedia.org and clouddumps1002.wikimedia.org) as in the following example's last two commands, it is necessary to be a member of a highly privileged group such as ops or a specific member of dumps-roots in data.yaml.

btullis@deploy1003:~$ kube-env mediawiki-dumps-legacy-deploy dse-k8s-eqiad
btullis@deploy1003:~$ kubectl exec -it $(kubectl get pod -l component=toolbox --no-headers -o custom-columns=":metadata.name") -- bash
Defaulted container "dump-toolbox" out of: dump-toolbox, mediawiki-dumps-legacy-resources-tls-proxy
www-data@mediawiki-dumps-legacy-dump-toolbox-59cc97c9bf-mnmfn:/$ cd /mnt/dumpsdata/xmldatadumps/public/enwiki/20261001/

www-data@mediawiki-dumps-legacy-dump-toolbox-59cc97c9bf-mnmfn:/mnt/dumpsdata/xmldatadumps/public/enwiki/20261001$ rm *
www-data@mediawiki-dumps-legacy-dump-toolbox-59cc97c9bf-mnmfn:/mnt/dumpsdata/xmldatadumps/public/enwiki/20261001$ cd ../../../temp/e/enwiki

www-data@mediawiki-dumps-legacy-dump-toolbox-59cc97c9bf-mnmfn:/mnt/dumpsdata/xmldatadumps/temp/e/enwiki$ rm *
www-data@mediawiki-dumps-legacy-dump-toolbox-59cc97c9bf-mnmfn:/mnt/dumpsdata/xmldatadumps/temp/e/enwiki$ cd ../../../private/enwiki/20261001/
www-data@mediawiki-dumps-legacy-dump-toolbox-59cc97c9bf-mnmfn:/mnt/dumpsdata/xmldatadumps/private/enwiki/20261001$ ls -l
total 385517
-rw-r--r-- 1 www-data www-data 394768854 Oct  6 20:18 dumplog.txt
www-data@mediawiki-dumps-legacy-dump-toolbox-59cc97c9bf-mnmfn:/mnt/dumpsdata/xmldatadumps/private/enwiki/20261001$ rm *

btullis@clouddumps1001:/srv/dumps/xmldatadumps/public/enwiki/20261001$ sudo rm *

btullis@clouddumps1002:/srv/dumps/xmldatadumps/public/enwiki/20261001$ sudo rm *

The external storage servers are too loaded because of the dumps

Go to https://airflow-test-k8s.wikimedia.org/pool/list/ and reduce the slots of the mediawiki-dumps-legacy-regular and mediawiki-dumps-legacy-large pool slots by 2.

The default values are 32 slots for mediawiki-dumps-legacy-regular and 32 slots for mediawiki-dumps-legacy-large
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Dumps/Airflow/Operations&oldid=2464827"