From Wikitech
Jump to navigation Jump to search
This page contains historical information. It may be outdated or unreliable.

wmsync is the provisional name for a replacement for scap/sync-file


Our current scap/sync-file solution is a bit long in the tooth. This would be a more robust replacement that we hope makes deploys go more smoothly. A few goals:

  • No NFS dependency, thus eliminates a significant single point of failure
  • Non-root sync will actually work, instead of failing in a lot of subtle ways as it does currently.
  • Allows atomic multi-file sync, so avoiding the flood of fatal errors we sometimes see when running scap
  • Should be faster than rsync, and should scale well with increasing file count and size.
  • Eliminates the subtle differences between scap and sync-file, such as the fact that the former rebuilds message caches and the latter doesn't.
  • Simplifies monitoring for servers with out-of-date copies of MediaWiki, by providing an overall content hash.
  • Not written in shell script (i.e. easier to read, maintain, and extend for our staff)
  • Baby step toward Heterogeneous deployment

Example usage

Recursive push of current directory to all servers:

wmsync -r .

Recursive push of whole tree:

wmsync -r /home/wikipedia/common

Push of single file:

wmsync CommonSettings.php

Planned features

  • Single-file sync
  • Atomic subtree incremental sync
  • SSH setup, shared secret config, whole tree pull
  • Local make scripts, remote make scripts
  • Any member can push -- configured within tree.
  • Network-aware distribution
  • Compact, prioritised error display
  • Add member, delete member. Node group update.
  • Whole tree integrity check on subtree/file sync, to fix servers which were disconnected.
  • File catalog to determine which files have changed

See also