Wm-bot

wm-bot is an IRC bot that participates in several developer-related wiki channels. wm-bot is currently running simultaneously on libera.chat

All stars team

These folks are currently operating the bot, please insert your name if you are one of us:

Petr Bena User:Petrb petan on Libera.chat (server access)
Billinghurst User:Billinghurst sDrewth on Libera.chat (server access)
Matthew Bowker User:Matthewrbowker Matthew_ on Libera.chat (server access)
MacFan4000 User:MacFan4000 MacFan4000 on Libera.chat (server access)
User:Legoktm legoktm on Libera.chat (server access)

Where is it

Instance wm-bot-bookworm.wm-bot.eqiad1.wikimedia.cloud, you need root to operate the bot.

/opt/wm-bot (freenode, historical, not in operation)
/opt/wm-bot-lib (libera)

How does it work

Bot is using external bouncers (AKA IO buffer servers in its configuration). That means, you first start the bouncers, then you start restart script which keeps core running. The core is connected to bouncers, which are connected to network. In case you need to restart the core, bot stays connected to network.

This bot is using multiple instances but all are part of 1 huge system. The core handles all the instances but there is a bouncer for each of them.

 ┌──────────────────────┐
 │                      │
 │        Libera        │
 │                      │
 └──────────────────────┘
     ▲              ▲
     │              │
     │              │
     │              │
     │              │
 ┌───┴───┐      ┌───┴───┐
 │       │      │       │
 │ BNC 1 │      │ BNC 2 │
 │       │      │       │
 └───────┘      └───────┘
     ▲              ▲
     │              │
     └──────┐┌──────┘
            ││
    ┌───────┴┴───────┐
    │                │
    │    Bot core    │
    │                │
    └─┬────────────┬─┘
      │            │
      │            │
      │            │
      │            │
      │            │
      ▼            ▼
  ┌───────┐    ┌────────────┐
  │       │    │            │
  │ MySQL │    │ PostgreSQL │
  │       │    │            │
  └───────┘    └────────────┘

Paths

Freenode (historical, not in operation)

Binaries	/opt/wm-bot/wmib.exe - core
Binaries	/opt/wm-bot/bouncer.exe
Configuration	/opt/wm-bot/configuration
Logs	/opt/wm-bot/*.log
IRC Logs	/var/www/html/logs

Libera

Binaries	/opt/wm-bot-lib/wmib.exe - core
Binaries	/opt/wm-bot-lib/bouncer.exe
Configuration	/opt/wm-bot-lib/configuration
Logs	/opt/wm-bot-lib/*.log
IRC Logs	/var/www/libera/logs

How to start the bot

IMPORTANT: If the server has crashed and bot wasn't properly shut down you have to remove all pid files using

sudo rm -iv /opt/wm-bot-lib/*.pid

If bot isn't running you can start it using

sudo service wm-bot-lib start

Troubleshooting

First of all, check what is already running:

ps -ef | grep wm-bot

If nothing is running, it should be ok, if some processes are there, kill them

Ensure there are no *pid files in /opt/wm-bot-lib
Make sure there is no restart.lock file
There must be only 1 instance of core (restart.sh) running in same time

tail /opt/wm-bot-lib/wmib.log is your friend

How to restart the bot core (fixes most problems)

This is what you should do if you just want to fix some bot issues, this will restart the core only, bot will stay connected to all channels and this process is painless and quick. Preferred:

tail /opt/wm-bot-lib/wmib.log there must be no critical errors! If you see that bot couldn't write log files etc, do not kill it, or these data will be permanently lost!
join #wm-bot
type @restart

If you get permission denied

You forgot to put your cloak to admins file, just ssh to wm-bot instance and kill process running as wm-bot called "mono wmib.exe". This process will restart itself when it crashes.

How to shutdown the bot

Complete shutdown

sudo stop_all.sh

Core only

If you need to restart the bot because you changed the binaries and you need it to completely reload, you just need to execute "halt" or @restart in any channel. Keep in mind that doing restart after you changed the binary will likely fail (bot will crash instead of proper shutdown).

You can also shutdown the bot with signal (kill -15).

This command will also restart the bot, but it may not give it enough time to properly shut down (bot's proper shut down takes about 10 seconds)

sudo service wm-bot-lib restart

How to fix 1 or more disconnected instances

Secret password is in /opt/wm-bot-lib/configuration/admins (username is root) it's on first row third column

Connect to local sysadmin console (noting that the bot core needs to be running)

$ telnet localhost 2820

Now enter the username (root) and password

Check the status (info / help)

Successfuly logged in to wm-bot
info
Uptime: 0 days  0 hours since 7/18/2013 11:53:39 AM
Instances:
wm-bot channels: 50 connected: True working: True
wm-bot2 channels: 42 connected: True working: True

If you see that some instance is not working properly, no matter of status, shut it down:

kill wm-bot

Now, in second terminal, restart the corresponding bouncer, the bouncer is process bouncer.exe with parameter that correspond to broken instance, for example for wm-bot it's bouncer.exe 6667, list of all bouncers can be found at http://wm-bot.wmcloud.org/dump/systemdata.htm

Once you kill that, you need start it again:

nohup mono bouncer.exe 6667 &

Once the bouncer is running again you need to connect bot to it, type this in system console:

conn wm-bot

How to change the logs

Recommended: In case you want to change logs, you must disable bot for that moment.

Chmod the logfile with 000 (bot won't be able to write)
Fix it and change the permissions back
Connect to SQL database, using password in the pg_pass= variable in the /opt/wm-bot-lib/configuration/wmib.conf file:

IMPORTANT: PostgreSQL doesn't have any decent backups yet, so please be careful, damaging DB may not be reversible

root@wm-bot-pg2-bookworm:~# psql -h wm-bot-pg2-bookworm -U wm_bot_lib

# Removing single line - this query may take minutes
> select * from logs where contents like '%text_you_search_for%';
> update logs set contents = 'redacted' where id = number_you_got_from_previous_query;

Drop the logs for the given channel (complete removal):

wm_bot_lib=> delete from logs where channel='#mychannel';

How to patch the bot

sudo ./easydeploy.sh

that will recompile all source code in ../wikimedia-bot stop the bot, replace the binaries and start the bot

How to change the nickname of bot

Nickname that is chosen when bouncer reconnects to network is defined in configuration file (configuration/wmib.conf), each instance may have completely different nickname. If you want to change it only temporarily, or fix it back (in case nick is taken, bot will append random number to nick), you have two options:

Wait - bot will automatically regain its hardcoded nick, it will perform check during periodic network resync, which happens about once a day (((Configuration.Channels.Count * 2000) + 80000) seconds to be precise).
Fix it via telnet:

petrb@wm-bot:~$ telnet localhost 2820
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Enter username:

Enter password:

Successfuly logged in to wm-bot, I have 1 users logged in
send wm-bot NICK wm-bot
send wm-bot2 NICK wm-bot2
quit

Password can be found in configuration/admins file

Backups

Are done using rsnapshot, see /etc/rsnapshot.conf, everything is stored under FS /backups, intervals are defined in /etc/cron.d/rsnapshot