Tool:CommTech
CommTech | |
---|---|
Website | https://commtech.wmflabs.org |
Description | Hosts Community Tech's test wiki and bot |
Maintainer(s) | Community Tech (View all) |
License | MIT License |
Issues | Open tasks · Report a bug |
This page is for documentation related to the commtech
VPS project.
CommTech Wiki
CommTech Wiki is a MediaWiki-Vagrant instance used for internal testing of MediaWiki features. It lives on the VPS instance commtech-2.commtech.eqiad.wmflabs
. The wiki installation is located in the /srv/mediawiki-vagrant
directory on that server.
Community Tech bot
This section documents bot tasks that live on the commtech-bot.commtech.eqiad.wmflabs
instance.
Creating a new instance
- Create a new instance (m1.medium or larger)
- Install PHP along with some dependencies
sudo apt update sudo apt install -y php php-cli php-common php-curl php-json zip unzip php-zip php-mysql php-dom sudo apt install composer cron
- Create the bot directory with
sudo mkdir /var/bot
- Create the
commtech-bot
user and give it ownership of the bot directory.sudo useradd -M commtech-bot sudo usermod -L commtech-bot sudo chown commtech-bot:commtech-bot /var/bot
Popular Pages bot
License: MIT
After creating an instance (see above), do the following:
- Clone the repo in the bot directory:
sudo su commtech-bot # Make sure you're the bot user cd /var/bot git clone https://github.com/wikimedia/popularpages.git
- Follow the installation steps at https://github.com/wikimedia/popularpages
- Edit the crontab for commtech-bot with
crontab -e
then add the following to the bottom:The first entry continually runs the main bot task ever 30 minutes for most of the month (it takes that long!). Using flock ensures there's only ever one instance of the bot running at a time. The second entry refreshes the index page every two hours so that users can monitor progress of the bot.# enwiki */10 * 4-11 * * /usr/bin/flock -n /tmp/fcj.lockfile php /var/bot/popularpages/bin/checkReports.php en.wikipedia 0 */2 4-11 * * php /var/bot/popularpages/bin/generateIndex.php en.wikipedia # arwiki */10 * 3-6 * * /usr/bin/flock -n /tmp/fcj-arwiki.lockfile php /var/bot/popularpages/bin/checkReports.php ar.wikipedia 0 */2 3-6 * * php /var/bot/popularpages/bin/generateIndex.php ar.wikipedia
Note that if you make any modifications to the /var/bot directory as any other user than commtech-bot, you must restore ownership to commtech-bot. For instance to pull in the latest changes from master, you could use:
cd /var/bot
sudo git pull origin master
sudo chown commtech-bot:commtech-bot .
NSFW
Source: https://github.com/rahiel/open_nsfw--
License: https://github.com/rahiel/open_nsfw--/blob/master/LICENSE.txt
The NSFW instance is at commtech-nsfw.commtech.eqiad.wmflabs
. This is a NSFW classifier service, using Yahoo!'s open source algorithm, wrapped in a Docker container.
See https://github.com/rahiel/open_nsfw-- for installation instructions.
To query the service, make a POST request with cURL:
$ curl -d 'url=https://upload.wikimedia.org/wikipedia/commons/thumb/b/b3/Jimmy_Wales_in_August_2006.jpg/319px-Jimmy_Wales_in_August_2006.jpg' https://nsfw.wmflabs.org
0.006064464803785086
WikiWho
Creating a new instance
- Create a new m1.small instance.
- Run:
sudo apt-get update sudo apt-get install -y apt-transport-https lsb-release ca-certificates sudo apt-get install -y apache2 libxml2-dev
- Create the WikiWho credentials file with
sudo htpasswd -c /etc/apache2/passwd-nospecial "[username]" "[password]"
- Make note of the hash for the password with
echo -n "WhoWroteThat:[password]" | base64
replacing[password]
with the password for the WikiWho account. - Create
/etc/apache2/sites-available/wikiwho.conf
as root with the contents:replacing<VirtualHost *:80> ServerName wikiwho.wmflabs.org ServerAdmin tools.wikiwho@tools.wmflabs.org ErrorLog ${APACHE_LOG_DIR}/error.log CustomLog ${APACHE_LOG_DIR}/access.log combined <Directory /> AuthUserFile /etc/apache2/htpasswd AuthName "Identify" AuthType Basic require valid-user </Directory> RequestHeader set Authorization "Basic [hash]" SSLProxyEngine on ProxyPass / https://api.wikiwho.net/ ProxyPassReverse / https://api.wikiwho.net/ RewriteEngine On RewriteCond %{HTTP:X-Forwarded-Proto} !https RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R=301,L] </VirtualHost>
[hash]
with the output of step #4. - Run:
sudo a2enmod rewrite sudo a2ensite wikiwho sudo a2enmod proxy proxy_http proxy_ajp proxy_wstunnel rewrite deflate headers proxy_balancer proxy_connect proxy_html xml2enc ssl sudo service apache2 reload sudo service apache2 restart
Pageviews
Creating a new instance
- Create a new m1.small instance.
- Install Apache, PHP and some dependencies:
sudo apt update && sudo apt -y upgrade sudo apt -y install php php-common sudo apt -y install php-cli php-fpm php-json php-xml php-mysql php-intl php-zip php-curl php-apcu sudo apt -y install apache2 libapache2-mod-php zip unzip default-mysql-client
- Install composer by following these instructions, but make sure to install to the
/usr/local/bin
directory and with the filenamecomposer
, e.g.sudo php composer-setup.php --install-dir=/usr/local/bin --filename=composer
- Clone the repository, first removing the html directory created by Apache.
cd /var/www && sudo rm -rf html sudo git clone https://github.com/MusikAnimal/pageviews.git cd /var/www/pageviews
- Become the root user with
sudo su root
- Run
sudo composer install --no-dev -o
- Run
sudo cp config.sample.php config.php
and editconfig.php
accordingly (see inline comments). - Make sure that all the files in the repo are owned by www-data.
sudo chown -R www-data:www-data .
- Create the web server configuration file at
/etc/apache2/sites-available/pageviews.conf
with the following:<VirtualHost *:80> DocumentRoot /var/www/public_html/ ServerName pageviews.wmcloud.org CustomLog ${APACHE_LOG_DIR}/access.log pageviews ErrorLog ${APACHE_LOG_DIR}/error.log <Directory /var/www/public_html/> Options Indexes FollowSymLinks AllowOverride None Require all granted </Directory> </VirtualHost>
- Enable the mod-rewrite Apache module, and enable the web server configuration.
sudo a2dismod mpm_event sudo a2enmod php7.4 sudo a2enmod rewrite sudo a2ensite pageviews sudo service apache2 reload
- (Re)start Apache:Moving forward, you should use
sudo service apache2 restart
sudo service apache2 graceful
to restart the server. - [TODO] Add symlinks (this should probably go in version control)