Help:Toolforge/Redis for Toolforge
|This page is currently a draft.|
More information and discussion about changes to this draft on the talk page.
|Help improve this page: https://phabricator.wikimedia.org/T232515|
This page contains information about Redis for Toolforge.
Redis for Toolforge
Redis is a key-value store similar to memcache, but with more features. It can be easily used to do publish/subscribe between processes, and also maintain persistent queues. Stored values can be different data structures, such as hash tables, lists, queues, etc. Stored data persists across service restarts. For more information, please see the Wikipedia article on Redis.
Note: Memcache is not available for Toolforge. Use Redis instead.
A Redis instance that can be used by all tools is available via the service name
tools-redis.svc.eqiad.wmflabs, on the standard port
6379. It has been allocated a maximum of 12G of memory, which should be enough for most usage. You can set limits for how long your data stays in Redis; otherwise it will be evicted when memory limits are exceeded. See the Redis documentation for a list of available
Libraries for interacting with Redis from PHP (
phpredis), Python (
redis-py), and Perl (
perl-redis) are installed on all the bastions and grid engine nodes. For an example of a bot using Redis, see gerrit-to-redis.
For quick & dirty debugging, you can connect directly to the Redis server with
nc -C tools-redis.svc.eqiad.wmflabs 6379 and execute commands (for example "INFO").
The redis service can be used as a broker between the worker and the web frontend to run a celery worker in a kubernetes container, as a continuous job (for instance to execute long-running tasks triggered by a web frontend).
Make sure you use your own queue name so that your tasks get sent to the right workers.
Redis has no access control mechanism, so other users can accidentally/intentionally overwrite and access the keys you set. Even if you are not worried about security, it is highly probable that multiple tools will try to use the same key (such as
lastupdated, etc). To prevent
this, it is highly recommended that you prefix all your keys with an application-specific, lengthy, randomly generated secret key.
PLEASE PREFIX YOUR KEYS! We have also disabled the Redis commands that let users 'list' keys. This protection however should not be trusted to protect any secret data. Do not store plain text secrets or decryption keys in Redis for your own protection.
You can generate a prefix by running the following command:
openssl rand -base64 32
Communication and support
We communicate and provide support through several primary channels. Please reach out with questions and to join the conversation.
|Phabricator Workboard||#Cloud-Services||Task tracking and bug reporting|
|IRC Channel||#wikimedia-cloud connect||General discussion and support|
|Mailing List||cloud@||Information about ongoing initiatives, general discussion and support|
|Announcement emails||cloud-announce@||Information about critical changes (all messages mirrored to cloud@)|
|News wiki page||News||Information about major near-term plans|
|Blog||Clouds & Unicorns||Learning more details about some of our work|