Push notifications
The push-notifications service accepts notification requests via HTTP for batching and submission to app push provider APIs. It it deployed in Kubernetes, and its request intake API endpoints are exposed only internally to the cluster. In the future it might be updated to handle web push (see phab:T266138).
Lifecycle of a push notification request
Push notifications build on the longstanding MediaWiki notification system implemented in the Echo extension. It adds a third notification type, currently labeled "Apps," to the preexisting "Web" and "Email" types. It is the responsibility of the app clients to register users' push subscriptions via the echopushsubscriptions API module and update their notification preferences in order to benefit from push notifications.
When an Echo event occurs and a recipient's notification preferences indicate that the user has chosen to be notified of that event type by push, Echo pushes a NotificationRequestJob onto the job queue. Upon execution of the job, the database is queried for all of the user's registered push subscriptions, and requests are made to the push-notifications service (to endpoints /v1/events/fcm and /v1/events/apns, for Firebase Cloud Messaging (Android) and Apple Push Notification Service (iOS), respectively) to request that a message be sent to each subscribed device.
Upon receiving a push notification request, the push-notification service enqueues the request to be batched with other requests for a configured period of time. This period can be either defined explicitly or configured to be randomized between a provided pair of lower and upper bounds. After the configured batching period passes, the request is consolidated with other requests received during the batching period into one or more multicast requests, and submitted to the push provider API. Consolidation with other messages into generic multicast requests is possible because the system currently sends only a generic message prompting the apps to contact the MediaWiki API to receive full message content.
[TO BE IMPLEMENTED: see T260247 and subtasks] When a push provider API responds that a subscriber token is invalid, the push-notifications service submits a request to the MediaWiki API to delete that subscription from the database.
Dashboards
The service is sending metrics to Prometheus via service-runner.
Job queue processing metrics for NotificationRequestJobs can also be seen on the JobQueue EventBus dashboard:
Connecting to push-notifications
Production
The service is exposed by service proxy at http://localhost:6012.
To access the service directly, by cluster:
To quickly verify that the service is up and running, request /_info:
mholloway-shell@deploy1001:~$ curl -s http://localhost:6012/_info | jq .
{
"name": "push-notifications",
"version": "0.0.1",
"description": "A service for sending push notification messages",
"home": "https://www.mediawiki.org/wiki/Wikimedia_Product_Infrastructure_team/Push_Notifications_Infrastructure"
}
Beta Cluster
A role::beta::docker_services instance of the service is running on deployment-push-notifications01 and exposed internally at http://deployment-push-notifications01.deployment-prep.eqiad1.wikimedia.cloud:8900.
Deployment process
Production
Beta Cluster
In Horizon, navigate to the deployment-prep (Beta Cluster) project and view the deployment-push-notifications01 instance. Click the "Puppet Configuration" tab and scroll to the "Hiera config" section. You will see a configuration definition like the following:
profile::docker::engine::declare_service: true
profile::docker::engine::packagename: docker.io
profile::docker::engine::settings: {}
profile::docker::engine::version: 18.09.1+dfsg1-7.1+deb10u2
profile::docker::runner::service_defs:
mediawiki-services-push-notifications:
config:
<service-runner service configuration>
namespace: wikimedia
override_cmd: -c /etc/mediawiki-services-push-notifications/config.yaml
port: 8900
version: 2020-09-25-202910-publish
puppetmaster: deployment-puppetmaster04.deployment-prep.eqiad.wmflabs
Update the version setting with the desired image tag from the pipeline. The update will be picked up and the service will be restarted with the new image on the next Puppet run.
In case of emergency
Push notifications support can be disabled without negative effects on other wiki operations. [TODO: Verify job queue retry behavior and ensure there are no spillover effects]
To disable push notifications support in MediaWiki (Echo), set $wgEchoEnablePush to false. This will result in the push subscription management API module being removed, and push notification request jobs no longer being sent to the job queue for submission to the push-notifications service.
Service level objectives (SLOs)
TODO: T264118