status.wikimedia.org

This page contains historical information. The service was decommisioned in July 2018 (reason: T199816).

Read about the new status page at wikimediastatus.net.

2018

status.wikimedia.org was a internal apache reverse proxy hosted on wikitech-static to provide secure web-interface interface to status monitoring provided by CA App Synthetic Monitor (formerly known as Watchmouse).

Service notes

I've moved the status.wm.o DNS to wikitech-static, and set up an apache reverse proxy there with a LetsEncrypt cert that auto-renews. It seems to work now, after much experimenting and mucking around!

For the record, since we have no puppet, in case we have to muck with this again, the basic things I did were:

Created a local acme user and group that can't log in
Copied acme-setup, acme_tiny.py, and x509-bundle from our puppet repo to /usr/local/sbin/
Commented out the self-verification portion of acme_tiny.py (this always seems to fail on challenge over redirect to self-signed for me).
Installed the letsencrypt X3 and X4 intermediates in /usr/local/share/ca-certificates and ran update-ca-certificates.
Enabled the following new apache2 modules: proxy, proxy_http, proxy_html
Set up the following as the sites-available/enabled file for status.wikimedia.org.conf (note especially the crazy html translation hacks for re-mapping links URLs, especially the mongocache one (which is for ajax data loaded from a separate HTTP-only URL belonging to CA...):

# vim: filetype=apache

<VirtualHost *:80>
	ServerAdmin noc@wikimedia.org
	ServerName status.wikimedia.org

	SSLEngine off

	RewriteEngine on
	RewriteCond %{SERVER_PORT} !^443$
	RewriteRule ^/(.*)$ https://status.wikimedia.org/$1 [L,R=301]

	ErrorLog /var/log/apache2/error.log

	# Possible values include: debug, info, notice, warn, error, crit,
	# alert, emerg.
	LogLevel warn

	CustomLog /var/log/apache2/access.log combined
	ServerSignature Off

</VirtualHost>
<VirtualHost *:443>
	ServerAdmin noc@wikimedia.org
	ServerName status.wikimedia.org

		SSLEngine on
		SSLCertificateFile /etc/acme/cert/status.chained.crt
		SSLCertificateKeyFile /etc/acme/key/status.key
		SSLProtocol all -SSLv2 -SSLv3
	SSLCipherSuite -ALL:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA
	SSLHonorCipherOrder On
	Header always set Strict-Transport-Security "max-age=31536000"

	<Location />
		ProxyPass "http://status.asm.ca.com/8777/"
		ProxyPassReverse "http://status.asm.ca.com/8777/"
		RequestHeader unset Accept-Encoding
		Header always set Content-Security-Policy upgrade-insecure-requests
		ProxyHTMLEnable On
		ProxyHTMLExtended On
		ProxyHTMLLinks	a		href
		ProxyHTMLLinks	area		href
		ProxyHTMLLinks	link		href
		ProxyHTMLLinks	img		src longdesc usemap
		ProxyHTMLLinks	object		classid codebase data usemap
		ProxyHTMLLinks	q		cite
		ProxyHTMLLinks	blockquote	cite
		ProxyHTMLLinks	ins		cite
		ProxyHTMLLinks	del		cite
		ProxyHTMLLinks	form		action
		ProxyHTMLLinks	input		src usemap
		ProxyHTMLLinks	head		profile
		ProxyHTMLLinks	base		href
		ProxyHTMLLinks	script		src for
		ProxyHTMLEvents	onclick ondblclick onmousedown onmouseup onmouseover onmousemove onmouseout onkeypress onkeydown onkeyup onfocus onblur onload onunload onsubmit onreset onselect onchange
		ProxyHTMLURLMap //status\.asm\.ca\.com/8777(/|$) //status.wikimedia.org/ [Ri]
		ProxyHTMLURLMap //mongocache.asm.ca.com/ //status.wikimedia.org/.mongocache/
		ProxyHTMLURLMap http:// https:// [i]
		SetOutputFilter proxy-html
	</Location>
	<Location /.mongocache>
		ProxyPass "http://mongocache.asm.ca.com/"
		ProxyPassReverse "http://mongocache.asm.ca.com/"
	</Location>

	<Location /.well-known/acme-challenge>
		ProxyPass "!"
	</Location>

	Alias "/.well-known/acme-challenge" "/var/acme/challenge"
	<IfVersion >= 2.4>
		<Directory "/var/acme/challenge">
			Require all granted
		</Directory>
	</IfVersion>

	ErrorLog /var/log/apache2/error.log

	# Possible values include: debug, info, notice, warn, error, crit,
	# alert, emerg.
	LogLevel debug

	CustomLog /var/log/apache2/access.log combined
	ServerSignature Off

</VirtualHost>

Ran the initial acme-setup for self-signed:

/usr/local/sbin/acme-setup -i status -s status.wikimedia.org -m self -u acme

Reloaded apache2
Re-run to get a real cert:

/usr/local/sbin/acme-setup -i status -s status.wikimedia.org -u acme -m acme -w apache2

Created a cronjob running exactly the above once a day at 17:17, which will auto-renew when necessary.

Service notes

See also