Help:Toolforge/Redis for Toolforge
Overview
This page contains information about Redis for Toolforge.
Redis for Toolforge
Redis, is a key-value store that can be used to implement publish/subscribe protocols between processes and maintain persistent queues. Stored values can be different data structures, such as hash tables, lists, queues, etc. Stored data persists across service restarts.
Redis instances
A Redis instance that can be used by all tools is available via the service name redis.svc.tools.eqiad1.wikimedia.cloud, on the standard port 6379. It has been allocated a maximum of 12G of memory, which should be enough for most usage. You can set limits for how long your data stays in Redis; otherwise it will be evicted when memory limits are exceeded. See the Redis documentation for a list of available
commands.
Redis libraries
Libraries for interacting with Redis from PHP (phpredis), Python (redis-py), and Perl (perl-redis) are installed on all the bastions and grid engine nodes. For an example of a bot using Redis, see gerrit-to-redis.
For quick & dirty debugging, you can connect directly to the Redis server with nc -C redis.svc.tools.eqiad1.wikimedia.cloud 6379 and execute commands (for example "INFO") or using redis-cli (also installed on the grid-engine nodes) redis-cli -h redis.svc.tools.eqiad1.wikimedia.cloud.
Celery
The redis service can be used as a broker between the worker and the web frontend to run a celery worker in a kubernetes container, as continuous job, see the documentation for Kubernetes (for instance to execute long-running tasks triggered by a web frontend).
 | Make sure you use a unique queue name so that your tasks get sent to the right workers. |
If using Django, assuming you use following definition of the namespace in your celery.py file:
app.config_from_object('django.conf:settings', namespace='CELERY')
Then, you can adapt the settings.py as following:
REDIS_PASSWORD=""
REDIS_HOST="redis.svc.tools.eqiad1.wikimedia.cloud"
REDIS_PORT="6379"
REDIS_DB=0
REDIS_URL = ':%s@%s:%s/%d' % (
REDIS_PASSWORD,
REDIS_HOST,
REDIS_PORT,
REDIS_DB)
CELERY_BROKER_URL = 'redis://'+REDIS_URL
CELERY_DEFAULT_QUEUE = "your-tool-name.default" # Change 'your-tool-name'
The deployment file for Kubernetes to start the worker can be written as (see Help:Toolforge/Kubernetes#Kubernetes continuous jobs):
| /data/project/your-tool-name/www/python/src/deployment.yaml |
|---|
| The following content has been placed in a collapsed box for improved usability. |
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: your-tool-name.worker
namespace: tool-your-tool-name
labels:
name: your-tool-name.worker
# The toolforge=tool label will cause $HOME and other paths to be mounted from Toolforge
toolforge: tool
spec:
replicas: 1
selector:
matchLabels:
name: your-tool-name.worker
toolforge: tool
template:
metadata:
labels:
name: your-tool-name.worker
toolforge: tool
spec:
containers:
- name: bot
image: docker-registry.tools.wmflabs.org/toolforge-python37-sssd-base:latest
command: [ "/data/project/your-tool-name/www/python/src/yourscript.sh" ]
workingDir: /data/project/your-tool-name/www/python/src/
env:
- name: HOME
value: /data/project/your-tool-name
imagePullPolicy: Always
Note: for debugging purpose, you can replace the command line with : command: ['sh', '-c', 'echo Hello, Kubernetes!']
Note : don't forget to give execution permission to your script with "chmod +x". |
| The above content has been placed in a collapsed box for improved usability. |
And the script for starting celery (yourscript.sh) can have following form:
#!/bin/bash
/data/project/your-tool-name/www/python/venv/bin/celery -A yourDjangoApp worker
Of course, if your virtual environment is at a different position, adapt the path.
To start your worker, type:
kubectl create --validate=true -f $HOME/www/python/src/deployment.yml
If everything is fine, "kubectl logs" followed from the name of the created pods should display typical celery starting feedback.
Security
Redis has no access control mechanism, so other users can accidentally/intentionally overwrite and access the keys you set. Even if you are not worried about security, it is highly probable that multiple tools will try to use the same key (such as lastupdated, etc). To prevent
this, it is highly recommended that you prefix all your keys with an application-specific, lengthy, randomly generated secret key.
PLEASE PREFIX YOUR KEYS! We have also disabled the Redis commands that let users 'list' keys. This protection however should not be trusted to protect any secret data. Do not store plain text secrets or decryption keys in Redis for your own protection.
You can generate a prefix by running the following command:
openssl rand -base64 32
Disabled Redis commands
Some built-in Redis commands have been disabled in an attempt to make Redis safer for multi-tenant usage:
- CLIENT
- CONFIG
- DEBUG
- FLUSHALL
- FLUSHDB
- KEYS
- MONITOR
- RANDOMKEY
- SCAN
- SHUTDOWN
- SLAVEOF
See Also
Communication and support
Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia Movement volunteers. Please reach out with questions and join the conversation:
- Chat in real time in the IRC channel #wikimedia-cloud connect, the bridged Telegram group, or the bridged Mattermost channel
- Discuss via email after you subscribed to the cloud@ mailing list