Summary

ORES uses a library called revscoring for building prediction models such as Edit quality, Article quality, Draft quality, Draft topic, Article topic. All of them are revscoring-based models. In Lift Wing, these models are hosted as Inference Services, in which models are structured as predicators and depend on revscoring to extract features, and return a prediction.

Develop and test revscoring models

In the section, we describe a simple way to develop and test revscoring models in your local environment.

Preparation

Prerequisites: Docker installed

Fist step is to download the production revscoring images. For example, here we download the Article quality image:

➜  ~ docker pull docker-registry.wikimedia.org/wikimedia/machinelearning-liftwing-inference-services-articlequality:stable

In your local environment, create an inference-test directory and put model.py inside. You could find the code in the Github mirror repositories:

Also, create a models directory and download the model binary inside. You could find model binary files in the Github repositories:

articlequality: https://github.com/wikimedia/articlequality/tree/master/models
editquality: https://github.com/wikimedia/editquality/tree/master/models
draftquality: https://github.com/wikimedia/draftquality/tree/master/models
topic: https://github.com/wikimedia/drafttopic/tree/master/models

Later when we run the docker container, we will mount these two directories inference-test and models to the container.

Now you can use your favorite IDE to open the model.py and modify the code as you want. Don't forget to change the model path which is the location kserve searches and loads model from to the model binary in your local environment.

def load(self):
    with open("/models/enwiki.nettrom_wp10.gradient_boosting.model") as f:
        self.model = Model.load(f)

After you finish code changes, save the model.py.

Run revscoring inference service

Open a terminal, type the following command to start a docker container:

➜  ~ docker run -p 8080:8080 -it -v "$(pwd)"/inference-test:/inference-code -v "$(pwd)"/models:/models --entrypoint=/bin/bash docker-registry.wikimedia.org/wikimedia/machinelearning-liftwing-inference-services-articlequality:stable

The command contains many parameters. I will explain one by one:

-p 8080:8080 - this means to expose port 8080 of the container to port 8080 of the host.
-it - short for --interactive + --tty
-v "$(pwd)"/inference-test:/inference-code - this means to mount inference-test directory in the host to /inference-code directory in the container (if it is not exist, create one). "$(pwd)" is your current working directory.
-v "$(pwd)"/models:/models - similar to the previous one, this means to mount models directory in the host to /models directory in the container.
--entrypoint=/bin/bash - this means to create a bash shell in the container and to overwrite the default initial commands of the image.
the last part is an image to derive the container from

If the container starts successfully, you will see a new prompt. This means you have entered the container!

somebody@f0dc42bf397c:/srv/articlequality$

You can check if inference-code and models directories exist under the root using commend ls /. If they exist, change your working directory to inference-code using cd /inference-code.

We need to set two environment variables, INFERENCE_NAME and WIKI_URL based on the wikipedia project and model type.

somebody@f0dc42bf397c:/inference-code$ export INFERENCE_NAME=enwiki-articlequality
somebody@f0dc42bf397c:/inference-code$ export WIKI_URL=https://en.wikipedia.org

Now we can run the inference service with:

somebody@f0dc42bf397c:/inference-code$ python3 model.py

If everything goes well, you should see messages like:

[I 220508 16:04:24 kfserver:150] Registering model: enwiki-articlequality
[I 220508 16:04:24 kfserver:120] Setting asyncio max_workers as 8
[I 220508 16:04:24 kfserver:127] Listening on port 8080
[I 220508 16:04:24 kfserver:129] Will fork 1 workers

Test revscoring models

After initiating the model server in the container, the model is listening on port 8080. Since we have exported the container's port 8080 to the host's 8080, we can send a request to the model server from the host machine.

First, create a file that contains input parameters the model needs. We call it input.json.

➜  ~ cat input.json
{"rev_id": 1083325118}

Open a terminal, send a request to port 8080 via curl using the following command:

➜  ~ curl localhost:8080/v1/models/enwiki-articlequality:predict -X POST -d @input.json

You should get the prediction result from the model.

{"predictions": {"prediction": "GA", "probability": {"B": 0.15992155417293058, "C": 0.22796325261056125, "FA": 0.015504683326725057, "GA": 0.5501430775309284, "Start": 0.04142996939619623, "Stub": 0.005037462962658527}}}

If you look at the terminal that runs the inference service, you would see something like:

[I 220508 16:17:09 web:2243] 200 POST /v1/models/enwiki-articlequality:predict (172.17.0.1) 1429.68ms