Monitoring/Netbox DNS uncommitted changes

From Wikitech

What it does

This Icinga check runs the Python script that generates all the DNS records from Netbox (the same script run by the sre.dns.netbox cookbook) and checks if there are any uncommitted changes in the Netbox#DNS exported repository. This ensures that changes in Netbox are reflected into the DNS repository and hence in production in a timely manner and that there are no leftover changes that will show up to the next person running the sre.dns.netbox cookbook.

Statuses

  • OK: there are no uncommitted changes, the autogenerated DNS repository is in sync with Netbox data.
  • WARNING: there are uncommitted DNS changes but the last edit in Netbox was made few minutes ago (at most 30 minutes as of Sep. 2020) hence it's not yet considered critical because most likely someone is still editing Netbox and will propagate the changes soon.
  • CRITICAL: there are uncommitted DNS changes and the last edit in Netbox was a while ago (at least 30 minutes as of Sep. 2020).

What to do

To see what are the pending DNS changes that are uncommitted, run in DRY-RUN mode the sre.dns.netbox cookbook as follows (see Spicerack/Cookbooks#Run a single Cookbook for more context on how to run a cookbook):

sudo cookbook -d sre.dns.netbox "test"

It will show all the pending DNS changes not yet committed.

If you are missing context get in touch with the people/team most likely related to those changes asking in the various SRE IRC channels.

You can also look at the Netbox Change Log.

Only if you are sure that those changes can be committed, follow DNS/Netbox#Update generated records.

This will propagate DNS changes to production, hence care must be taken to make sure they are correct and would not cause an outage.