Talk:Incidents/2018-11-29 ores

Rendered with Parsoid
From Wikitech
(Redirected from Talk:Incident documentation/2018-11-29 ores)
Latest comment: 5 years ago by Mobrovac

I'd like to propose an alternate takeaway: Application configuration should ship with the application, and should not be in puppet. Had we followed that model, the change would have been merged, deployed immediately (as opposed to being deployed asynchronously by puppet), and the issue would have been discovered right away by someone with both context on what had been changed and expertise needed to fix it. - Ian Marlier

By "with the application" do you mean packaged as a hardcoded configuration inside a configuration file inside a package? That requires the build/release pipeline to accomodate different deployments (one package with configuration for each place where the application is deployed?). How do you control deviations in the configuration (e.g. some changes it in place)? -- GTirloni
Scap has this possibility, and in fact we do it for most services. The configuration lives in the respective deploy repos, so a config change is applied only when someone explictly deploys the application. It would be good to see the same approach used for ORES, +1. Mobrovac (talk)
User:GTirloni I can't speak to the full range of scenarios, but one that would potentially address some would be using the wikimedia-cluster value. EG, imagine having a 'config.yaml' file, and then a 'config-eqiad.yaml' and a 'config-codfw.yaml', with the appropriate one of the latter being imported and used as an override depending on the contents of /etc/wikimedia-cluster. (I acknowledge that this only address one specific case, and that I've never looked in to the general case. For example, I have no idea how the scap functionality that Marko mentioned works, or how that would interact.

User:Imarlier and User:Mobrovac: It's a good idea but the problem in this case is that some configs of ores are private (like password to the redis node) so there is no way we can ship those scap/helm. Ladsgroup (talk) 14:53, 30 November 2018 (UTC)Reply

Ladsgroup Private configs are definitely a complicating factor. What about having puppet drop those private configs somewhere (/etc, /srv, I'm not sure what the best option would be), and then have ORES read them in from there? Then the rest of the config could come from a file that would be shipped with the application.
Ladsgroup, Scap allows you to combine private parts from ops/puppet or ops/private with your config. You should talk to RelEng and see how to incorporate that into ORES. We do that routinely for virtually all Node services, where part of the variables come from the deploy repo and part of the variables come from ops/puppet and/or ops/private. Mobrovac (talk) 17:25, 4 December 2018 (UTC)Reply