Obsolete:svn.wikimedia.org
svn.wikimedia.org.
Hardware
- antimony
- SVN+SSH developer access
- SVN over HTTP public read-only access
- ViewVC
Related services
- svn-private - houses internal scripts and configurations that can't be publicly viewable
- codereview-proxy.wikimedia.org
Dependencies:
- lists.wikimedia.org - MediaWiki-CVS commit notification list
- Offsite -- CIA update notifications
Used by:
- codereview-proxy.wikimedia.org - indirection tool to access SVN logs from MediaWiki for CodeReview extension
Server setup
Packages
apt-get install some packages:
subversion # SVN core bits libsvn-notify-perl # For commit notification mails viewvc # For repository view python-subversion # Needed for viewvc python-pygments # Used by viewvc for syntax highlighting php5-cli # Used by svnusers.php doxygen # For doc generation apache2 # Web server! libapache2-svn # HTTP SVN module for apache libapache2-mod-php5 # For user.php
Custom software
/usr/local/bin/sillyshell # restricted shell; in tools/sillyshell in SVN /usr/local/bin/ciabot_svn.py # IRC notifications for SVN, available at cia.cx /usr/local/viewvc/* # ViewVC - viewvc.org
Doc generation
Handled by puppet.
# Set up an account to run the docs under adduser --system mwdocs su -s /bin/bash mwdocs cd && svn co file:///svnroot/mediawiki/trunk/phase3
# make a log file for the cronjob cd /var/log touch mwdocs.log chown mwdocs mwdocs.log
# fix path in apache config # add a cron job: crontab -e 30 0 * * * (cd /home/mwdocs/phase3 && svn up && php maintenance/mwdocgen.php --all) >> /var/log/mwdocs.log 2>&1
SVN users setup
Note: By default most LDAP commands try to run as the "scriptuser", which requires root privileges. In general most of these should run as root anyway. You'll need to use sudo to run these commands if you are not root.
Add users
Run add-ldap-user on terbium:
Usage: add-ldap-user [options] <username> <key(s)> example: add-ldap-user laner http://ryandlane.com/static/pubkey.key Options: -h, --help show this help message and exit -s, --self Use your credentials, rather than the proxyagent's -D BINDAS, --bindas=BINDAS Specify user to bind as -m, --directorymanager Use the Directory Manager's credentials, rather than your own --shell=LOGINSHELL The user's shell (default: /usr/bin/sillyshell) --gid=GIDNUMBER The user's gid (default: 500) --uid=UIDNUMBER The user's uid (default: next available uid) --home=HOMEDIRECTORY The user's home directory (default /home/username) --cn=CN The user's CN (default: firstname + lastname, or username, if first/last not set) --firstname=GIVENNAME The user's first name (default: username) --lastname=SN The user's last name (default: username)
The <key(s)> should be either a static URL or a local file.
Modify users
Run modify-ldap-user on terbium:
Usage: modify-ldap-user [options] <username> [--rename <newusername>] example: modify-ldap-user --replacekeys=http://ryandlane.com/static/pubkey.key laner Options: -h, --help show this help message and exit -s, --self Use your credentials, rather than the proxyagent's -D BINDAS, --bindas=BINDAS Specify user to bind as -m, --directorymanager Use the Directory Manager's credentials, rather than your own --shell=LOGINSHELL Set the user's shell --gid=GIDNUMBER Set the user's gid --uid=UIDNUMBER Set the user's uid --cn=CN Set the user's CN --firstname=GIVENNAME Set the user's first name --lastname=SN Set the user's last name --replacekeys=REPLACEKEYLOCATION Replaces all of the user's keys --addkeys=ADDKEYLOCATION Adds keys to the user's entry --rename Rename the user
Delete users
It is likely this should never be done. There are probably better ways to deny access. Removing a user's SSH keys should deny access everywhere. However, if you really want to do this, you can use delete-ldap-user:
Usage: delete-ldap-user [options] <username>
Options:
-h, --help show this help message and exit
-s, --self Use your credentials
-D BINDAS, --bindas=BINDAS
Specify user to bind as
--scriptuser Use the scriptusers' credentials
-m, --directorymanager
Use the Directory Manager's credentials, rather than
your own
--no-delete-home Don't delete the user's home directory
Checking LDAP information
You can check entries by using ldaplist, which is slightly easier to use than ldapsearch:
Usage: ldaplist [options] [database] [object-name]
example: ldaplist -l passwd ldap_user
Options:
--help show this help message and exit
-s, --self Use your credentials
-D BINDAS, --bindas=BINDAS
Specify user to bind as
-m, --directorymanager
Use the Directory Manager's credentials
--scriptuser Use the scriptusers' credentials
-v, --verbose Show the database and search filter used for this
search
-l, --longlisting List all the attributes for each entry matching the
search criteria. By default, ldaplist lists only the
Distiguished Name of the entries found.
-h Show available databases to search
-d, --showdatabase Show the base dn being used for this database
-a SHOWATTRIBUTES, --showattributes=SHOWATTRIBUTES
Show the given attributes
-r, --recursive Recurse netgroups
--like Search for objects that equal or sound like [object-
name]
For instance, to check a user's account:
ldaplist -l passwd <username>
To find a group and list its membership:
ldaplist -l group <some group name>
Access Control
By default, users have no access in /svnroot/mediawiki/conf/authz. Users can be added to the core, and mediawiki groups using svn-group; mediawiki is for extensions and tools access, whereas core is for all other access. Most people with core will also need mediawiki.
Using svn-group:
Usage: /usr/local/bin/svn-group {add|delete} <user> <group>
Example: /usr/local/bin/svn-group add laner core
Groups:
- mediawiki - only access to extensions, tools and branches
- core - access to core mediawiki
- wmf-deployment - access to deployment branch
When you give someone commit access, link their mediawiki.org username to their commit username by going to https://www.mediawiki.org/wiki/Special:Code/MediaWiki/author/commitname and using the "link this user" link. Then, give the mediawiki.org username "coder" rights via https://www.mediawiki.org/wiki/Special:UserRights/username. Send an email to wikitech-l announcing the new committer (ok to do this in 1 daily batch). Also add the committer to the mw:Developers page.
To see which SVN groups a user is currently in, look in /svnroot/mediawiki/conf/authz.
All SVN users have access to the pywikipediabot project, even those who aren't in /svnroot/mediawiki/conf/authz.
Troubleshooting
Bad ssh keys
[12:53] <Ryan_Lane> I think the way I'm parsing keys is occasionally screwed up [12:55] <Ryan_Lane> ^demon: if it imported the keys correctly, there should be a sshPublicKey attribute for every key [12:56] <Ryan_Lane> when I added rfaulk, his keys were all jacked up [12:56] <Ryan_Lane> with like 20 sshPublicKey attributes, each having one letter :) [12:57] <Ryan_Lane> ^demon: if you ever run into that problem, download the key from the URL, and use modify-ldap-user --replacekeys=<filename> [12:57] <Ryan_Lane> it always seems to work from a file
Dumps
Dumps are handled daily at 18:00 UTC and offloaded to Tridge via Amanda. Cron runs daily /usr/local/bin/svndump.php. Sunday performs a full dump, with daily incrementals. The previous day's dump is cleared before starting each day's dump.
Dumps of svn.wikimedia.org are available at https://dumps.wikimedia.org/other/misc/ .
Post-commit hooks
SVN Post commit hooks are used for commit e-mail notification, CIA and CodeReview, and can be found in /svnroot/svn tree/hooks/
CodeReview
# Hit CodeReview wget -q -O /dev/null --post-data="" \ "http://www.mediawiki.org/w/api.php?action=codeupdate&format=xml&repo=MediaWiki&rev=$REV" &
Git/Gerrit
An import of svn.wikimedia.org's mediawiki/trunk is available in Gerrit at https://gerrit.wikimedia.org/r/#/admin/projects/subversion/svn.wikimedia.org/mediawiki/trunk . This git repo contains (among other things) phase3 (i.e.: what's now called mediawiki/core) and all extensions. So if you need to bring a back-then-svn-hosted extension to gerrit, you need not reimport an svn dump, but you can just git clone this repo and then git filter-branch on it.