Jump to content

Tool:CommTech

From Wikitech
Toolforge tools
CommTech
Website https://commtech.wmflabs.org
Description Hosts Community Tech's test wiki and bot
Maintainer(s) Community Tech (View all)
License MIT License
Issues Open tasks · Report a bug

This page is for documentation related to the commtech VPS project.

CommTech Wiki

CommTech Wiki is a MediaWiki-Vagrant instance used for internal testing of MediaWiki features. It lives on the VPS instance commtech-2.commtech.eqiad.wmflabs. The wiki installation is located in the /srv/mediawiki-vagrant directory on that server.

Community Tech bot

This section documents bot tasks that live on the commtech-bot.commtech.eqiad.wmflabs instance.

Creating a new instance

  1. Create a new instance (m1.medium or larger)
  2. Install PHP along with some dependencies
    sudo apt update
    sudo apt install -y php php-cli php-common php-curl php-json zip unzip php-zip php-mysql php-dom
    sudo apt install composer cron
    
  3. Create the bot directory with sudo mkdir /var/bot
  4. Create the commtech-bot user and give it ownership of the bot directory.
    sudo useradd -M commtech-bot 
    sudo usermod -L commtech-bot
    sudo chown commtech-bot:commtech-bot /var/bot
    

After creating an instance (see above), do the following:

  1. Clone the repo in the bot directory:
    sudo su commtech-bot # Make sure you're the bot user
    cd /var/bot
    git clone https://github.com/wikimedia/popularpages.git
    
  2. Follow the installation steps at https://github.com/wikimedia/popularpages
  3. Edit the crontab for commtech-bot with crontab -e then add the following to the bottom:
    # enwiki
    */10 * 4-11 * * /usr/bin/flock -n /tmp/fcj.lockfile php /var/bot/popularpages/bin/checkReports.php en.wikipedia
    0 */2 4-11 * * php /var/bot/popularpages/bin/generateIndex.php en.wikipedia
    
    # arwiki
    */10 * 3-6 * * /usr/bin/flock -n /tmp/fcj-arwiki.lockfile php /var/bot/popularpages/bin/checkReports.php ar.wikipedia
    0 */2 3-6 * * php /var/bot/popularpages/bin/generateIndex.php ar.wikipedia
    
    The first entry continually runs the main bot task ever 30 minutes for most of the month (it takes that long!). Using flock ensures there's only ever one instance of the bot running at a time. The second entry refreshes the index page every two hours so that users can monitor progress of the bot.

Note that if you make any modifications to the /var/bot directory as any other user than commtech-bot, you must restore ownership to commtech-bot. For instance to pull in the latest changes from master, you could use:

cd /var/bot
sudo git pull origin master
sudo chown commtech-bot:commtech-bot .

NSFW

The NSFW instance is at commtech-nsfw.commtech.eqiad.wmflabs. This is a NSFW classifier service, using Yahoo!'s open source algorithm, wrapped in a Docker container.

See https://github.com/rahiel/open_nsfw-- for installation instructions.

To query the service, make a POST request with cURL:

$ curl -d 'url=https://upload.wikimedia.org/wikipedia/commons/thumb/b/b3/Jimmy_Wales_in_August_2006.jpg/319px-Jimmy_Wales_in_August_2006.jpg' https://nsfw.wmflabs.org
0.006064464803785086

WikiWho

Creating a new instance

  1. Create a new m1.small instance.
  2. Run:
    sudo apt-get update
    sudo apt-get install -y apt-transport-https lsb-release ca-certificates
    sudo apt-get install -y apache2 libxml2-dev
    
  3. Create the WikiWho credentials file with sudo htpasswd -c /etc/apache2/passwd-nospecial "[username]" "[password]"
  4. Make note of the hash for the password with echo -n "WhoWroteThat:[password]" | base64 replacing [password] with the password for the WikiWho account.
  5. Create /etc/apache2/sites-available/wikiwho.conf as root with the contents:
    <VirtualHost *:80>
            ServerName wikiwho.wmflabs.org
            ServerAdmin tools.wikiwho@tools.wmflabs.org
    
            ErrorLog ${APACHE_LOG_DIR}/error.log
            CustomLog ${APACHE_LOG_DIR}/access.log combined
    
            <Directory />
                    AuthUserFile /etc/apache2/htpasswd
                    AuthName "Identify"
                    AuthType Basic
                    require valid-user
            </Directory>
    
            RequestHeader set Authorization "Basic [hash]"
    
            SSLProxyEngine on
            ProxyPass / https://api.wikiwho.net/
            ProxyPassReverse / https://api.wikiwho.net/
    
            RewriteEngine On
    
            RewriteCond %{HTTP:X-Forwarded-Proto} !https
            RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R=301,L]
    </VirtualHost>
    
    replacing [hash] with the output of step #4.
  6. Run:
    sudo a2enmod rewrite
    sudo a2ensite wikiwho
    sudo a2enmod proxy proxy_http proxy_ajp proxy_wstunnel rewrite deflate headers proxy_balancer proxy_connect proxy_html xml2enc ssl
    sudo service apache2 reload
    sudo service apache2 restart
    

Pageviews

Creating a new instance

  1. Create a new m1.small instance.
  2. Install Apache, PHP and some dependencies:
    sudo apt update && sudo apt -y upgrade
    sudo apt -y install php php-common
    sudo apt -y install php-cli php-fpm php-json php-xml php-mysql php-intl php-zip php-curl php-apcu
    sudo apt -y install apache2 libapache2-mod-php zip unzip default-mysql-client
    
  3. Install composer by following these instructions, but make sure to install to the /usr/local/bin directory and with the filename composer, e.g.
    sudo php composer-setup.php --install-dir=/usr/local/bin --filename=composer
    
  4. Clone the repository, first removing the html directory created by Apache.
    cd /var/www && sudo rm -rf html
    sudo git clone https://github.com/MusikAnimal/pageviews.git
    cd /var/www/pageviews
    
  5. Become the root user with sudo su root
  6. Run sudo composer install --no-dev -o
  7. Run sudo cp config.sample.php config.php and edit config.php accordingly (see inline comments).
  8. Make sure that all the files in the repo are owned by www-data.
    sudo chown -R www-data:www-data .
    
  9. Create the web server configuration file at /etc/apache2/sites-available/pageviews.conf with the following:
    <VirtualHost *:80>
            DocumentRoot /var/www/public_html/
            ServerName pageviews.wmcloud.org
    
            CustomLog ${APACHE_LOG_DIR}/access.log pageviews
            ErrorLog ${APACHE_LOG_DIR}/error.log
    
            <Directory /var/www/public_html/>
                    Options Indexes FollowSymLinks
                    AllowOverride None
                    Require all granted
            </Directory>
    </VirtualHost>
    
  10. Enable the mod-rewrite Apache module, and enable the web server configuration.
    sudo a2dismod mpm_event
    sudo a2enmod php7.4
    sudo a2enmod rewrite
    sudo a2ensite pageviews
    sudo service apache2 reload
    
  11. (Re)start Apache:
    sudo service apache2 restart
    
    Moving forward, you should use sudo service apache2 graceful to restart the server.
  12. [TODO] Add symlinks (this should probably go in version control)