Jump to content

Help:Toolforge/Running Pywikibot scripts (advanced)

From Wikitech
If you only need to run a script that is included with Pywikibot itself, it is recommended that you follow the running Pywikibot scripts tutorial instead.

This tutorial contains instructions on how to run custom Pywikibot scripts on Toolforge using the Toolforge jobs framework.

Prerequisites

To effectively use custom Pywikibot scripts on Toolforge, you need:

Installation and setup

  1. SSH to a Toolforge bastion
  2. Become your tool-account
    MAINTAINER@tools-sgebastion-10:~$ become TOOLNAME
    
  3. Create a setup script named pwb_venv.sh with the following content that will create a virtual environment and install Pywikibot's dependencies.
    tools.TOOLNAME@tools-sgebastion-10:~$ nano $HOME/pwb_venv.sh
    
    $HOME/pwb_venv.sh
    #!/bin/bash
    # Install Pywikibot in a virtual environment from git sources
    # (Pywikibot's PyPi package does not include scripts)
    
    # clean up directories if they already exist
    rm -fdr pwbvenv $HOME/pywikibot-core
    
    # create a new virtual environment
    python3 -m venv pwbvenv
    
    # activate it
    source pwbvenv/bin/activate
    
    # clone Pywikibot
    git clone --depth 1 --single-branch --recurse-submodules --shallow-submodules --branch stable "https://gerrit.wikimedia.org/r/pywikibot/core" $HOME/pywikibot-core
    
    # install dependencies
    pip install --upgrade pip setuptools wheel
    pip install $HOME/pywikibot-core[mwoauth,mysql]  # update as needed
    # pip install ... # any other dependencies here
    
  1. Make the pwb_venv.sh script executable.
    tools.TOOLNAME@tools-sgebastion-10:~$ chmod ug+x $HOME/pwb_venv.sh
    
  2. Run the setup script using the jobs framework.
    tools.TOOLNAME@tools-sgebastion-10:~$ toolforge jobs run setup-venv --command $HOME/pwb_venv.sh --image python3.11 --wait
    INFO: job 'setup-venv' completed
    
  3. Configure Pywikibot
    • Interactively:
      tools.TOOLNAME@tools-sgebastion-10:~$ webservice python3.11 shell  # start an interactive Kubernetes shell
      tools.TOOLNAME@shell-#########:~$ source pwbvenv/bin/activate  # activate the virtual environment
      (pwbvenv) tools.TOOLNAME@shell-#########:~$ pwb generate_user_files  # run the generate_user_files.py Pywikibot script
      
      user-config.py and (optionally) user-password.py are generated by following the prompts.
    • Manually: See the manual on MediaWiki wiki: user-config.py, BotPasswords, and OAuth.
  4. You may test the installation from an interactive shell.
    tools.TOOLNAME@tools-sgebastion-10:~$ webservice python3.11 shell  # start an interactive Kubernetes shell
    tools.TOOLNAME@shell-#########:~$ source pwbvenv/bin/activate  # activate the virtual environment
    (pwbvenv) tools.TOOLNAME@shell-#########:~$ pwb version  # run the version.py Pywikibot script
    Pywikibot: pywikibot/__init__.py (, -1 (unknown), 2024/03/13, 21:31:19, UNKNOWN)
    Release version: 9.0.0
    packaging version: 24.0
    mwparserfromhell version: 0.6.6
    wikitextparser version: n/a
    requests version: 2.31.0
      cacerts: /data/project/TOOLNAME/pwbvenv/lib/python3.11/site-packages/certifi/cacert.pem
        certificate test: ok
    Python: 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]
    PYWIKIBOT_DIR: Not set
    PYWIKIBOT_DIR_PWB: /data/project/TOOLNAME/pwbvenv/lib/python3.11/site-packages/pywikibot/scripts
    PYWIKIBOT_NO_USER_CONFIG: Not set
    Config base dir: /data/project/TOOLNAME
    Usernames for family 'commons':
            commons: Example bot
    

Create jobs

Main article: Help:Toolforge/Jobs framework

Example of running the Pywikibot touch.py script once:

tools.TOOLNAME@tools-sgebastion-10:~$ toolforge jobs run purge-example --command "$HOME/pwbvenv/bin/python3 $HOME/pywikibot-core/pwb.py touch -purge -forcelinkupdate -page:Example" --image python3.11

Example of running your own script on a schedule:

tools.TOOLNAME@tools-sgebastion-10:~$ toolforge jobs run my-cool-job --command "$HOME/pwbvenv/bin/python3 $HOME/my-cool-script.py" --image python3.11 --schedule "@daily"

Additional parameters for the job could include, for example, additional memory allocation (--mem MEM), job restart after being finished (--continuous), etc.

OAuth

Pywikibot can use OAuth v1.0a as the method of authenticating with wikis. To configure this behavior for Wikimedia wikis, follow the instructions in Manual:Pywikibot/OAuth/Wikimedia.

Caution: When using OAuth as the authentication method, be sure to set the correct permissions on the user-config.py file as described in Manual:Pywikibot/OAuth. This will keep your OAuth credentials more secure.

To learn more about OAuth in general, see OAuth/For Developers.

Next steps

Now that you know how to run your bot on Toolforge, read the script development and maintenance tips described in Help:Toolforge/Developing successful tools. These tips will help you to license, publish, and support your bot over time.

If you are looking for ideas on what types of scripts you can run using Pywikibot, see Manual:Pywikibot/Scripts. If you want to write your own script, see Manual:Pywikibot/Cookbook.

If you want to teach others how to run bots on Toolforge, see the How to host a bot on Toolforge - workshop materials.

Troubleshooting

ModuleNotFoundError: No module named '...', but that dependency is installed

This is sometimes caused by leftovers from a Toolforge Grid Engine setup. To fix it, remove or comment any lines starting with export PYTHONPATH= in the .bash_profile file in the tool home directory.

See also

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support
Stay aware of critical changes and plans
Track work tasks and report bugs

Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself

Read stories and WMCS blog posts

Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)