people.wikimedia.org
people.wikimedia.org hosts some of the user public files, in the form of people.wikimedia.org/~username.
Connecting
- You need shell access in production before you can use
people.wikimedia.org.
You can lookup what the DNS name peopleweb.discovery.wmnet points to to check which is the currently active server that you should use.
Connect to people.wikimedia.org over ssh by proxying through a bastion (e.g. bast1003.wikimedia.org). As with other hosts, ensure you have ProxyJump configured in .ssh/config (see Production shell access). Then connect as follows:
user@laptop:~$ ssh people.eqiad.wmnet
Service
The web server runs on people1004 with people2003 as the fail-over.[1] It previously ran on people2002, people1003, people1002, people1001, and rutherfordium.
How to upload files
Once you have production SSH configured with ProxyJump, you can use scp to copy files from your workstation (via a bastion) to your home directory on people.codfw.wmnet (which is a CNAME, aliased to people2002.codfw.wmnet currently).
Put files into a directory called "public_html". If that doesn't exist yet, create it with mkdir.
How To add SSO Authentication
sites on people.wikimedia.org can be configuered to use the SSO service to authenticate users to their site via .htaccess files. for example
allow any authenticated users
AuthType CAS
Require valid-user
Allow all members of the NDA group
AuthType CAS
Require cas-attribute memberOf:cn=nda,ou=groups,dc=wikimedia,dc=org
Allow only a specific user
AuthType CAS
Require user jbond
Allow only users who have authenticated with u2f
AuthType CAS
Require cas-attribute successfulAuthenticationHandlers:U2FAuthenticationHandler
Who has access
If you are a member of any admin group that gets shell access, you also have a user on people1003. Access is automatically granted to the "all users" group. You can also ssh to the CNAME people.eqiad.wmnet to avoid having to remember a number that might change in the future.
How to test changes
If you want to test a change on a people.wm.org URL you can SSH to the current deployment server and use httpbb.
This example requires manually writing a test file from scratch or copying one to the deployment host. See the newer example below on how to use the already installed test file coming from a repository.
Example:
rzl@deploy1001:~$ cat people.yaml
https://people.wikimedia.org:
- path: /~rzl/
assert_status: 200
assert_body_contains: "👋"
rzl@deploy1001:~$ httpbb /home/rzl/people.yaml --hosts people1002.eqiad.wmnet
Sending to people.wikimedia.org...
PASS: 1 request sent to people.wikimedia.org. All assertions passed.
Nowadays (2021) we also have a test suite file for people for httpbb in the repository.
See `modules/profile/files/httpbb/people/test_people.yaml` in the `operations/puppet` repository if you would like to add or edit tests.
Example:
dzahn@deploy1002:~$ httpbb --hosts people1003.eqiad.wmnet,people2002.codfw.wmnet /srv/deployment/httpbb-tests/people/test_people.yaml
Sending to 2 hosts...
PASS: 5 requests sent to each of 2 hosts. All assertions passed.
How to perform a switch-over between data centers
- Determine the currently active server by looking up `peopleweb.discovery.wmnet` in DNS:
[mwmaint1002:~] $ host peopleweb.discovery.wmnet peopleweb.discovery.wmnet is an alias for people1004.eqiad.wmnet.
- As of today, 2023-07-14, the active machines are people1004 and people2003 (note the number mismatch), both on bookworm already.
- Look at `hieradata/role/common/microsites/peopleweb.yaml` and the "rsync_src_host" and "rsync_dst_host" values in there. The active server from before
should be the "src_host", it is the "source of truth" and where users should upload. You can also confirm this by the MOTD on these servers, on the inactive server puppet will add a warning.
- We are using `rsync::quickdatacopy` but with `auto_sync => false`, so rsyncd, firewall rules etc are setup but you need to manually run the actual rsync command. It also means that rsyncd will run on the _source_ ost.
- ssh to the _dest_ host and rsync over copying from the source host to you.
- You are allowed to copy the entire /home and the rsync module name is "people-home". (see /etc/rsync.d/frag-people-home on the source host), so rsync example command:
rsync -avp rsync://people1004.eqiad.wmnet:/people-home /home/
- Note how we are not using --delete here since it's possible some users have used the non-active host to dump files that are not served (in /home but not in /home/*/public_html) and we don't need to overwrite them. We are accepting the risk to overwrite files with identical file names.
- run puppet once on the dst_host to let it overwrite changes in dot files in the home dir, just in case
- Make a puppet change to switch rsync_src_host and rsync_dst_host, merge it, run puppet on both machines, see the MOTDs adjust accordingly
- Make a DNS change in templates/wmnet and change the entry for "peopleweb.discovery.wmnet". You will see one of the backends is just commented out with ; like this and you switch that around:
peopleweb 300 IN CNAME people1004.eqiad.wmnet. ;peopleweb 300 IN CNAME people2003.codfw.wmnet.
(note there is also a "people" entry in eqiad and codfw but that just nees to be changed if you introduce new server names and it's not relevant for where ATS sends traffic, peopleweb is the relevant entry for that)
- After deploying the DNS change (authdns-update) confirm you see traffic ending up on the new active backend, (tail -f /var/log/apache2/*.log) when opening some test URL in a browser. (random test URL: https://people.wikimedia.org/~jmm/pws)
- You can also run the httpbb tests for people on a deploy or cumin server:
[deploy1002:~] $ httpbb /srv/deployment/httpbb-tests/people/test_people.yaml --hosts=people1004.eqiad.wmnet,people2003.codfw.wmnet Sending to 2 hosts... PASS: 6 requests sent to each of 2 hosts. All assertions passed.
- Inform the users via a mailing list like "sre-at-large" (since that should cover shell users) that we switched and they should now upload to the other DC. You can remind them that there are the "people.eqiad.wmnet" and "people.codfw.wmnet" DNS names and also the "peopleweb.discovery.wmnet" but with the former they still have to use a different name and with the latter they have to expect a key change warning.
History
Backend switch-over June 2023
On June 14, 2023 the backend switched to people1004 as the new source of truth and active backend with people2003 as the failover.
Backend switch-over February 2023
On February 24, 2023 the backend switched from people1003 to people2002 in codfw as part of the data center switch over. The new place to upload to is people2002 and you can use peopleweb.discovery.wmnet to look up which is the currently active one. public_html directories have been rsynced over, files outside of public_html dirs have not been touched. You still have shell access to both but only the active one is serving HTTPS traffic.
Backend upgrade May 2021
On May 11, 2021 the backend switched from people1002 on Debian buster to a new machine people1003 on Debian bullseye. The DNS CNAME people.eqiad.wmnet can be used as before and has been switched over. Just expect the new fingerprints.
All files in home directories have been rsynced over to the new host.
Backend upgrade May 2020
On May 21, 2020 the backend switched from people1001 on Debian stretch to a new machine people1002 on Debian buster. The DNS CNAME people.eqiad.wmnet can be used as before and has been switched over. Just expect the new fingerprints.
All files in home directories have been rsynced over to the new host.
Backend upgrade November 2018
On November 29, 2018 the backend switched from rutherfordium on Debian jessie to a new machine people1001 on Debian stretch. The DNS CNAME people.eqiad.wmnet can be used as before and has been switched over. Just expect the new fingerprints.
All files in home directories were rsynced over to the new host.