Jump to content

Incidents/20150227-LabsOutage

From Wikitech

Summary

At 02:30UTC on 2015-02-27 all of the labs instances hosted on virt1012 lost network connectivity. A reboot and restart of instances restored them within 40 minutes, but the same problem recurred at 05:20UTC. During the second outage, critical tools instances were migrated to other hosts and virt1012 was upgraded from Ubuntu Precise to Ubuntu Trusty. By 08:45 virt1012 was back up and running and all services were restored.

This problem was almost certainly a duplicate of the one that caused the outage earlier in the week. The cause is still unknown.

Timeline

  • [02:30] Instances drop off network -- shinken reports failure of many Tools instances. Based on our findings during the previous similar failure, Marc attempts to suspend all instances (which fails, as before) and then reboots virt1012.
  • [03:05] Marc resets and reboots all affected instances and services are restored.
  • [03:20] Andrew and Yuvi begin preparations against a repeat of this failure, creating duplicate tools-webproxy hosts on other virt hosts. Not fast enough!
  • [05:20] Shinken reports tools instance failure, and all virt1012 VMs drop off the network again. Andrew opts to upgrade the server to Trusty rather than just reboot. (There was no strong reason to believe that an upgrade would address the issue, other than it having a chance to move us away from a potentially buggy kernel or library.)
  • [05:30-08:30] Unfortunately, the upgrade to Trusty takes an extremely long time, due to Ubuntu having recently released a security-patched kernel; the Ubuntu package repositories are extremely busy and slow. In the meantime, Andrew cold-migrates several hosts off of virt1012 to other virt hosts: tools-webproxy, dynamicproxy-gateway and tools-submit.
  • [07:10] Tools-webproxy resumes operation on virt1010. May web-based tools are restored. Note that the cold-migrate of tools-webproxy required multiple attempts, as the required base image wasn't present on the first target host.
  • [07:20] Tools-submit resumes operation on virt1010, restoring all normal Tools services.
  • [08:05] dynamicproxy-gateway resumes on virt1010, restoring many labs-based web services.
  • [08:30] virt1012 is finally full upgraded and rebooted; Andrew restarts all remaining instances on virt1012
  • [08:45] All VMs are restored to active state.

Conclusions

It's still unclear what is causing this problem. The upgrade to Trusty seems an unlikely solution since this issue is specific to a single server; virt1010 and virt1011 are running the same software that virt1012 was.

Actionables

Yuvi has already created replacement instances for tools-webproxy. Currently two of them are on the same host though (virt1010) and switching over between them requires manual intervention.

The tools-webproxy backups need to be redistributed among three different hosts. Similar swappable tools instances should be constructed and distributed. The same should probably also be done for project deployment-prep.

We should acquire new hardware soon, in order to have available space should we need to abandon virt1012 entirely.

Vulnerable Instances

Should the virt1012 issue recur, the following instances are vulnerable: 

+--------------------------------------+---------------------------+---------+------------+-------------+-------------------------------------+
| ID                                   | Name                      | Status  | Task State | Power State | Networks                            |
+--------------------------------------+---------------------------+---------+------------+-------------+-------------------------------------+
| 684e5f6f-3fbf-42a1-a44e-2953448448fd | accounts-mwoauth          | ACTIVE  | -          | Running     | public=10.68.17.44                  |
| 6f147938-fe0c-4868-abd6-da7d0667c07f | bastion2                  | ACTIVE  | -          | Running     | public=10.68.16.66, 208.80.155.153  |
| 8b1c0173-bfad-45d4-a464-1f6772339b66 | bbdevel                   | ACTIVE  | -          | Running     | public=10.68.17.31                  |
| b141a571-5fb0-4c62-90b2-e76268511a6b | catalogcompiler           | ACTIVE  | -          | Running     | public=10.68.16.24                  |
| eded2579-f205-4d34-b935-d8fa0361c237 | cephticon3                | SHUTOFF | -          | Shutdown    | public=10.68.16.132                 |
| d33dba53-45b6-4525-943a-ad24b100cd48 | cg-puppetmaster           | ACTIVE  | -          | Running     | public=10.68.17.133                 |
| e7985789-f4fb-4993-b0c4-23699be455b8 | cvn-apache6               | ACTIVE  | -          | Running     | public=10.68.17.6                   |
| e97213bf-a56e-4d79-a6e3-508e6157d19c | cvresearch-web            | ACTIVE  | -          | Running     | public=10.68.16.91                  |
| f8d3db4c-6fd3-4ea7-9b80-ef1c50e4b0f8 | dannyb                    | ACTIVE  | -          | Running     | public=10.68.16.139                 |
| d0da5ac8-34ca-43a9-b63d-64ee438c29cc | deployment-cxserver03     | ACTIVE  | -          | Running     | public=10.68.16.150                 |
| aec2245c-e965-4f06-8496-a0d9abf519ee | deployment-db1            | ACTIVE  | -          | Running     | public=10.68.16.193                 |
| 0294f3af-eba5-4ac8-9205-7c1aba8808d9 | deployment-logstash1      | ACTIVE  | -          | Running     | public=10.68.16.134                 |
| cec6f6dc-5ab0-420e-8bc0-871ad3c9999b | deployment-parsoid01-test | ACTIVE  | -          | Running     | public=10.68.17.215                 |
| 09bbebac-631d-4126-ad3e-48ef64e72eb8 | deployment-restbase02     | ACTIVE  | -          | Running     | public=10.68.16.234                 |
| fcaa135e-3fce-4fae-afe0-ded789fe6f6a | deployment-sca01          | ACTIVE  | -          | Running     | public=10.68.17.54                  |
| 4b75951b-d874-4805-a4a7-28bd5c3ba6b4 | designate-devel           | ACTIVE  | -          | Running     | public=10.68.17.102                 |
| 0740df98-0a0b-45c4-9e33-6f0bbe126109 | designate-tests-juno      | SHUTOFF | -          | Shutdown    | public=10.68.16.216                 |
| 62b61eb0-c6b0-47a1-90e2-69ac5c7f26ab | designate-tests-trusty    | SHUTOFF | -          | Shutdown    | public=10.68.16.169                 |
| 07976e9b-6676-46d8-afc2-ee1cfd3a25b1 | dns-test-dzahn            | ACTIVE  | -          | Running     | public=10.68.16.232                 |
| 64d1952b-8b9e-4693-ab05-301245d6cc99 | drmf                      | ACTIVE  | -          | Running     | public=10.68.17.17                  |
| 5d3003d5-2d9a-4c88-80a5-b9a3db8415ea | etcd02                    | ACTIVE  | -          | Running     | public=10.68.17.157                 |
| 44a6a994-71dd-4ace-a4ba-30274883644e | filippo-swift-test        | ACTIVE  | -          | Running     | public=10.68.17.146                 |
| 8eb0fd07-292d-448e-9124-76b5e51317b5 | grantreview-dev           | ACTIVE  | -          | Running     | public=10.68.16.158                 |
| 51dc707e-f9f8-4e59-a4d9-5073908d22e1 | hovercards                | ACTIVE  | -          | Running     | public=10.68.17.136                 |
| 33da85da-eea6-4bf5-977c-5a75cc32215f | incident-test             | ACTIVE  | -          | Running     | public=10.68.17.109                 |
| 2198a8cf-da89-416b-bed5-a0521b122d0f | integration-slave1010     | ACTIVE  | -          | Running     | public=10.68.17.200                 |
| 1f700fb7-7d90-4df6-a849-6bfd7e4c1049 | integration-slave1201     | ACTIVE  | -          | Running     | public=10.68.17.179                 |
| a352d133-b08d-4f87-aca4-443fa9af57dc | integration-slave1202     | ACTIVE  | -          | Running     | public=10.68.17.189                 |
| c3c55df6-dfe0-4549-8987-b9c3e3e6b05b | integration-slave1203     | ACTIVE  | -          | Running     | public=10.68.17.195                 |
| eac1020b-f152-4ece-af2b-348bb6b04691 | integration-slave1204     | ACTIVE  | -          | Running     | public=10.68.17.207                 |
| 52cd4e56-c8a8-44dc-b4a4-d8f0e0210ac0 | integration-slave1401     | ACTIVE  | -          | Running     | public=10.68.17.208                 |
| 543cf5bd-5c5b-4bc3-a682-365ad2b709f8 | integration-slave1402     | ACTIVE  | -          | Running     | public=10.68.17.209                 |
| 7f8e4841-9f31-44ab-8c2c-d61c3fa466cc | integration-slave1403     | ACTIVE  | -          | Running     | public=10.68.17.210                 |
| e1086138-e898-40a9-a2f7-aa7e5e96fcbd | integration-slave1404     | ACTIVE  | -          | Running     | public=10.68.17.211                 |
| 3f5489e8-97b1-4f57-8481-e5ea31874535 | integration-slave1405     | ACTIVE  | -          | Running     | public=10.68.17.212                 |
| 77a8ccdd-f29e-4d07-89e4-ab9f13f0248b | ipsec-c8                  | ACTIVE  | -          | Running     | public=10.68.16.38                  |
| 0db39308-f680-414a-8d15-48fc21d65859 | ipsec-c9                  | ACTIVE  | -          | Running     | public=10.68.17.202                 |
| 42cdf096-eedf-4f64-8995-ab9874976a5f | jawiki-echo               | SHUTOFF | -          | Shutdown    | public=10.68.16.174                 |
| e9ac64d1-5594-49dc-98fb-573ba98d29ef | joal-hadoop-master        | ACTIVE  | -          | Running     | public=10.68.16.220                 |
| d6ff0d45-023e-4299-9b27-cdc93500cf30 | joal-hadoop-worker1       | ACTIVE  | -          | Running     | public=10.68.17.203                 |
| 7f65413f-b449-458b-968d-89afb15f8915 | joal-hadoop-worker2       | ACTIVE  | -          | Running     | public=10.68.17.206                 |
| a25019a1-fa5f-4c5a-be57-ff975c2e4552 | legoktm                   | ACTIVE  | -          | Running     | public=10.68.17.84                  |
| 3cc2200a-5432-4fe0-8db6-33ec35732b12 | map                       | ACTIVE  | -          | Running     | public=10.68.16.181                 |
| e381861f-aa7e-4d7a-894e-c919e86c73e3 | mathoid2                  | ACTIVE  | -          | Running     | public=10.68.16.194                 |
| b7076a0a-008f-4282-88b2-c5c4ea0b3ace | megacron-two              | ACTIVE  | -          | Running     | public=10.68.16.49, 208.80.155.149  |
| e082a270-d3f1-4934-8659-3224d06de417 | mlp                       | ACTIVE  | -          | Running     | public=10.68.16.3                   |
| d05d5ec9-48d1-42cc-96de-b40c485bed51 | mwui                      | ACTIVE  | -          | Running     | public=10.68.16.61                  |
| 20853100-12c6-480d-9eb7-a3d9e1864280 | nemobis                   | ACTIVE  | -          | Running     | public=10.68.17.131                 |
| 0153e94a-2f43-4b87-ac06-d6be88cb0c6c | opengrok-web              | ERROR   | -          | Shutdown    | public=10.68.16.184                 |
| d65b95f9-d28f-484c-a98f-7119aa085b89 | openid-wiki               | SHUTOFF | -          | Shutdown    | public=10.68.16.185                 |
| ed66a882-4b9e-4db7-b657-f4e7423f1900 | openstack-juno-testing    | SHUTOFF | -          | Shutdown    | public=10.68.16.97                  |
| 1e4668ee-d3aa-4ff7-9dbe-819267f6ec84 | otto-cass2                | ACTIVE  | -          | Running     | public=10.68.17.60                  |
| 6dbee509-88fb-43ac-990c-f8e47f16d7e2 | pirsquared-dev            | ACTIVE  | -          | Running     | public=10.68.17.57                  |
| 9fcfcd5f-7467-43b3-bc54-bfd02e5fbfc3 | pubsubhubbub              | ACTIVE  | -          | Running     | public=10.68.16.69                  |
| 89040a4e-6eeb-466b-8a22-484fe37f2eb0 | rcstream                  | ACTIVE  | -          | Running     | public=10.68.17.114, 208.80.155.180 |
| 38177df6-aafa-4b02-9aa7-5e7a1d82541a | staging-palladium         | ACTIVE  | -          | Running     | public=10.68.16.112                 |
| e2cd24cf-b9c6-40db-b6f3-df6dbae85c95 | staging-tin               | ACTIVE  | -          | Running     | public=10.68.17.217                 |
| 64493322-ecf0-4cdd-8d13-c8cfcb894158 | test-mem01                | ACTIVE  | -          | Running     | public=10.68.17.87                  |
| 120cc401-ed7a-44c5-b905-2d0eae23b6af | tools-exec-03             | ACTIVE  | -          | Running     | public=10.68.16.32, 208.80.155.142  |
| 30b98f1d-1c5a-49c1-b800-f4c535addc12 | tools-exec-07             | ACTIVE  | -          | Running     | public=10.68.16.36, 208.80.155.146  |
| 5cd684db-d0a6-4241-a11f-daf4c1b2f717 | tools-exec-09             | ACTIVE  | -          | Running     | public=10.68.17.64, 208.80.155.152  |
| 523df61c-07f0-41ba-924d-e2b8e474b4d7 | tools-exec-cyberbot       | ACTIVE  | -          | Running     | public=10.68.16.39                  |
| 65516028-7f49-4976-962c-543113ca77eb | tools-uwsgi-02            | ACTIVE  | -          | Running     | public=10.68.17.216                 |
| cdce426b-ef6f-47e7-96e4-bcb3647f4709 | tools-webgrid-04          | ACTIVE  | -          | Running     | public=10.68.17.174                 |
| 7fa7d90f-b783-409f-aa46-5cba75283645 | tools-webgrid-generic-02  | ACTIVE  | -          | Running     | public=10.68.16.215                 |
| 79aeb31c-a1c1-41af-9e00-df2c7e248924 | tools-webgrid-tomcat      | ACTIVE  | -          | Running     | public=10.68.16.29                  |
| 8df6ac98-05b8-4d60-bc5c-15b3c206c450 | toolsbeta-mail            | ACTIVE  | -          | Running     | public=10.68.16.113                 |
| 5584c52e-619d-424b-b060-14b385754147 | toolsbeta-master          | ACTIVE  | -          | Running     | public=10.68.16.146                 |
| 31e8206d-fa5c-4e62-a805-8cfb7def1f46 | toolsbeta-puppetmaster3   | ACTIVE  | -          | Running     | public=10.68.16.92                  |
| a0a49d0b-0fae-45bf-938b-7c942567fa8c | upload-wizard             | SHUTOFF | -          | Shutdown    | public=10.68.16.228                 |
| 29704829-3c25-4f79-b6f6-ed985946ec1f | wdq-bg1                   | ACTIVE  | -          | Running     | public=10.68.16.156                 |
| 0bb0601e-381d-491d-a580-c13ca963da8c | wdq-bg2                   | ACTIVE  | -          | Running     | public=10.68.16.159                 |
| 29f1b7a1-b3a5-4be7-8af3-81e620d69555 | wdq-bg3                   | ACTIVE  | -          | Running     | public=10.68.16.168                 |
| 9271df41-9f7a-4b28-95a1-dc691f0e0076 | wdq-wikidata              | ACTIVE  | -          | Running     | public=10.68.17.219                 |
| 09733bff-d485-40d8-9a7f-4a3670509741 | wikidata-reports          | SHUTOFF | -          | Shutdown    | public=10.68.17.34                  |
| 303fcd47-12dd-4c00-8961-13dd82d03ee7 | wikimetrics-staging1      | ACTIVE  | -          | Running     | public=10.68.16.77                  |
+--------------------------------------+---------------------------+---------+------------+-------------+-------------------------------------+
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Incidents/20150227-LabsOutage&oldid=1965564"