Data Platform/Systems/Dealing with webrequest data loss

We monitor webrequest using 2 mechanisms:

• The actual number of rows computed using count by hostname should match the expected one, computed using MAX(sequence) - MIN(sequence) by hostname (sequence being an incremental integer assign by the cache host). By match, we mean the difference should be smaller than 5%. If it's higher than that, see below.
• The percentage of rows that edge servers mark as <BADREQ>. We have not taken any action when these rose above our 25% "threshold", but it is worrying that we process so many bad requests.

Data problems include:

• Some records have dt = '-', we think this happens because Varnish only does "best effort" reporting of the date we are logging, and when it's under load it skips some records (Open Question: is this still happening after the HA proxy move?).
• False positives: hosts are sometimes reset and the sequence numbers can have an unexpected gap. This shows up in our statistics as percent_loss but it's a false positive. The procedure below helps identify false positives.
• True positives: these would look just like the above, but the missing rows are not found in the adjacent hours by the checker, so it's actual loss.
• Many records with <BADREQ>, labeled this way at the edge

Example of trouble-shooting webrequest errors: https://phabricator.wikimedia.org/T208752

Check dataloss False positives

It is possible that a data loss alarm is triggered because data for one hour appears on the previous or next hour. You can verify that with this scrip

• if missing less than 2%, too small, don't worry about it (and normally no email is sent)
• if missing more than 2% of data, an alert email is sent -- check if data-loss (hostname, sequence) are not a false positive
  • Wait until the hour AFTER the one in alert has been refined
  • Connect to a server with the analytics kerberos keytab (eg. an-launcher1003)
  • Run the following command with:
    • TABLE: if the email was a warning, data got refined, use wmf.webrequest. If the email was an error, data didn't get refined, use wmf_raw.webrequest.
    • SOURCE, YEAR, MONTH, DAY, HOUR: updated accordingly to the alert.

sudo -u analytics kerberos-run-command analytics spark3-sql --master yarn -S \
         --jars /usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core.jar \
         -f /srv/deployment/analytics/refinery/hive/webrequest/check_dataloss_false_positives.sparksql \
         -d table_name=TABLE            \
         -d webrequest_source=SOURCE    \
         -d year=YEAR                   \
         -d month=MONTH                 \
         -d day=DAY                     \
         -d hour=HOUR

• If the output of this query contains rows that have the false_positivefield to false, there is real data-loss. Contact someone from Data Platform Engineering (DPE) to look more in details.
• If the output of this query contains only rows that have false_positive field set to true, and no rows in the hours before or after, then you might be looking at the wrong kind of data loss, it might be that the records are missing a timestamp. You can check this by reading the original alert email and the attached file.

Side Note: When checking for false-positives, it is interesting to understand the data-pattern of rows without dt in hadoop partitions. The rows end up not having a dt because the initial requests to varnish might have timeout and thus the dt , which is added when the request completes it is just not there. Since Gobblin falls back to using current-timestamp if dt is not set, rows that don't have a valid dtcan end up either in hour or hour+1, depending on when it is generated and when Gobblin runs. This pattern explains why we filter no-dt rows in the current-hour but not in the next-hour in the dataloss-script: for current hour, no-dt rows have sequence numbers for either previous-hour or current-hour, while next-hour has no-dt rows for current-hour and next-hour. The script is actually interested in those rows with no-dt whith sequence numbers belonging to current-hour.