Swift/Other Processes

From Wikitech
Jump to: navigation, search

Current

Ganglia Logtailer

See swift::base in puppet for this.

  • ganglia-logtailer process running on all swift nodes via cron (front and back ends)
  • parses logs to gather performance and throughput information, feeds that into ganglia

swift-ganglia-report-global-stats

See role::swift::pmtpa-prod::ganglia_reporter in puppet for this.

  • runs on one server per cluster, currently ms-fe1
  • reports cluster-wide statistics to ganglia
    • object count, bytes used, change in # objects
    • these aren't related to a specific server but instead reflect cluster-wide stats

Old

purgethumbs.py

Unused; can't find source to this anywhere; does it still exist?

  • purges thumbnails from all media stores
  • takes in a list of images to purge and attempts to purge all thumbnails for that list of images from all three locations: ms5, swift, and squid
  • used to clean out unreferenced thumbnails generated by ariel's script

delete-containerless-objects

Can't find source to this anywhere; does it still exist?

  • script to crawl the object contents in a swift backend, compare this to a list of what's in the container, and delete objects that exist but aren't in the container list.
    • note that this shouldn't happen in the first place

Swift Cleaner

No longer running.

  • compared thumbnails on swift to those on ms5. deleted things on swift that didn't match what's on ms5.
  • two instances used to run on iron - full scan and incremental scan.
    • incremental took about 2 hours
    • full took about 2 weeks
  • was installed in /opt/swiftcleaner. canonical source in git:operations/software/swiftcleaner