Portal:Toolforge/Admin/Monthly meeting/2026-09-29

Attendees

Andrew Bogott, Arthur Puthin, Belinda Liviero, David Caro, Francesco Negri, Raymond Ndibe, Seyram Komla Sapaty, Taavi Väänänen

Notes

• Recurring items:
  • K8s upgrades
    • Andrew will work on this next quarter
  • Push to deploy
    • Feature was presented last week in a Discord call (slides)
    • Very positive feedback received during call (about 20 participants between staff and volunteers)
    • Next step is support in webservice CLI and Components-api
    • Target for stable release: end of October. Other improvements will follow.
    • Alerting service: user asked if they could define own alerts during runtime, using their own metric
    • More calls in the future (inside or outside Discord)
      • Will it be possible for one or more people from tools-infra to join the future calls?
      • One request (from same person) for multi-tenancy w/ Opensearch
• Moving logging away from NFS
  • When will we have resources for it?
  • Not currently mapped in the roadmap in phab:project/board/8755/
  • Once we move pre-built jobs to components, we can make them log to loki by default
  • We can also change the default for jobs (already tracked in Phab, but not part of the push-to-deploy stable release)
• Capacity of k8s cluster
  • Last week users were not able to schedule pods because CPU was at 90%
  • Some metrics disagree with each other, Andrew is looking at this
  • We have 8 very big new nodes, that should help
  • Do we have a metric that shows how long jobs wait before being scheduled?
  • We’re reserving 10x the CPU we are actually using it, but the default reservation is already quite small
  • What is the biggest pod we want to support?
  • Graph with pending pods