Luca: we could post to eventgate in our model.py for Lift Wing
Tobias: could we do this in Istio on our side? a bit like request logging right?
Luca: its just a simple POST request so we could do it in the python code, could maybe try Knative eventing but we are using an older version of Knative.
Tobias: agreed, doing it w/ a library or wrapper makes alot of sense, only tricky bit is you don't want to delay the call.
Andy: does consistency matter? can we just fire off a post request via asyncio and then return our prediction to the user?
Luca: that should be fine, if some are missing it's not super problematic.
also transformer + predictor both need to mount/load model into pod from storage
editquality will have 30+ isvcs running two large images
Chris: Do we want to use transformers on future models? Yes. The ORES models are a special case.
Kevin: the transformers seemed fine until we needed to load the models into the separate pods, now it seems really heavy.
Andy: my one argument for keeping the heavy transformers was that we could use it in an explainer, but that does not seem to work (maybe a kserve bug?)
Tobias: Forcing transformers architecture on revscoring models may not help us gain much, other than keeping it alive longer.
Luca: for editquality it might make sense to go back to predictor
the mw transformers are not async either so there is a bottleneck.
Kevin: Also, regarding maintainability, we are currently loading models + revscoring and all its dependencies in both the transformer and predictor. Loading them in only the predictor is much more maintainable. It's more DRY.
ORES migration
Chris: We should get all the models onto Lift Wing before ORES dies (hardware out of warranty, stretch is ending LTS in a few months, etc.) and then we can improve the models.
Tobias: I wish we could easily sample say 5% of current ORES-bound traffic and see what happens to it on LW
Luca: We could use changeprop and see how it handles on LW
Chris: We could do an experiment, although every single model we are not able to migrate over is a conversation we need to have with the community.
Andy: It would be pretty easy to migrate editquality over to LW. We just need to add the model binary files to Thanos Swift and then update the helm files.
Chris: Let's load every single ORES model into LW. We got 110+ of them, let's start moving them.
Luca: For editquality, let's get rid of transformer, move to predictor-only and then start spinning up pods.
Andy: I will make a task and then Kevin and I can split the work from there.
Staging Environment
Kevin: What is our staging environment going to be?
Chris: Should we do dev on staging?
Luca: We have ml-sandbox
Kevin: ml-sandbox is good
Andy: I think we were unsure of what the testing cluster would be used for. Also the cluster-local-gateway networking issues hadn't been solved on ml-sandbox yet, so we were unsure if we should continue maintaining our dev cluster. Things are good now and I think ml-sandbox is good for dev.
Luca: i think it happens during feature extraction?
Luca: adding more logging would help us figure out what is happening
Chris: this will help keep ores stable while we migrate to lift wing
Andy: this could be helpful for us to see if there is a bug buried somewhere.
Luca: maybe not fix the bug but help us know where the issue is
ORES deploy
Andy: we need to deploy the new nlwiki articlequality model, maybe this week?
Luca: let's include the logging updates for the spikes
Andy: +1, i'm happy to do the deploy, maybe we can record it and use it as a side-by-side comparison video w/ lift wing?
ORES clients
Luca: let's catalog all clients, bots, how people are calling ores (apis etc.) a good starting task for maybe Aiko?
Andy: +1 - we need local contacts for wiki communities too
Hal (privacy engineer) had suggested having 'service cards' that describe all downstream users for a model, might be good to have early-on for lift wing.
Luca: a preliminary list of users, tools, etc.
2022-01-12
Feast spike & hardware order
Luca: We should discuss our plans for the Feature store(s), we have to review procurement tasks for dcops this week. Current Plan:
3 redis-like nodes in eqiad
3 redis-like nodes in codfw
2 nodes in eqiad (for the offline store, but it was in early stages, not even sure if we need those)
Andy: the new version ranges for revscoring on the model class repos have some implications for the inference-service image builds. The 'easiest' solution so far has been to install model-class repo via git+ssh and pin to a specific commit in requirements.txt
Andy: Next step is to create transformers for editquality, draftquality and topic
These images might also be large. They may need the same assets as the predictor due to feature extraction process. Another reason to start work on a feature store soon!
