This page contains historical information. It is probably no longer true.
- This page appears to have moved to phabricator:T104681.
Scaling up SSL infrastructure
- push them to caching frontends
- do we have enough CPU on frontends for this?
- need (better) statistics + benchmark likely solutions
- Determine if load from logged-in users is sufficient for benchmarking. Switch certain projects to alternate software/ciphers/pfs/etc to test relative load (possibly with depooling)
- Faidon: Apache testing (vs nginx)
- Will it be cheaper to expand the frontends, or to expand the terminator cluster?
- Split Varnish frontend from backends (SSDs, lots of mem), put SSL terminators with frontends (just CPU)
- Switching to Varnish will eliminate IPV6 protocol proxies
- get rid (or replace) LVS source hashing
needs Varnish proxy protocol support(patch written by Mark, nearing completion)
- Apache 2.4 + memcached
- nginx + session sharing patch (to be written by us or Nginx Inc.)
- all above: session tickets need to be either disabled or shared, example strategy: https://blog.twitter.com/2013/forward-secrecy-at-twitter
- mark's wcsh LVS scheduler (weighted consistent source hashing)
switch to TLSv1.2done, with GCM cipher switch to PFS, ECDHE, where applicabledone
- move away from SHA1 to SHA256 in certificates (will get disabled in browsers at some point)
- rotate SSL session ticket master key daily or disable session tickets altogether
- remove wildcards/unified from misc services, switch to individual certs or SSL cluster
- investigate shorter expiry for certificates (e.g. 6 months)
- Fix broken Apache redirects
- Add HTTPS to Wikipedia Zero (needs involvement of the zero partner...)
- Zero team must be involved prior to anon default HTTPS (same partner carrier issues as non-HTTPS-countries)
- Certificate pinning
- Only implemented in Chrome right now, as far as I know
- box-to-box traffic (mark will investigate ipsec)
- stop serving full chains, omit root CAs already known to browsers
- switch to individual wildcard certs & SNI, use large unified only as fallback
- investigate OCSP stapling (esp. with regards to intermediate CAs)
- investigate other enhancements to fit initial negotiation in one RTT
- SPDY -- postpone for "v1.1"
- Poorly supported in nginx
- mod_spdy doesn't work for 2.4 yet - it's also a big hack with unmerged patches to Apache core
- None support ALPN as of this time
- Is also a security benefit thanks to pipelining for paylod-size attacks.
- Not without collapsing domains (all requests via one connection)
- Wait and see what development / standardization (HTTP 2.0) looks like when we're ready for HTTPS-by-default
- investigate ECDSA certificates -- faidon to investigate more, drop it for v1.1 if it's too complicated
- smaller than 2048-bit RSA, helps with performance (RTTs)
- more secure, 256-bit equivalent to 3072-bit RSA
- less computational cost
- where to procure? cost?
- how/when to fallback to RSA (Google requires SNI + *no* deflate support for ECDSA)
- SSL terminator UDP logging - Ken will talk to Toby and get the shutdown into the analytics plan
- The logs are duplicated. We can likely eliminate them if we track XFP
- Fixing udp2log is probably pointless at this point as it's about to be replaced with Kafka...
HTTPS by default
- only solution at the moment is to enable by language-wiki, which is suboptimal (especially in terms of feedback)
- GeoIP is in progress (see and
- separate IP to deal with China?
- Needs to be more than separate IP. If login happens over https, the redirection cookie is added. Also, login itself occurs per wiki and doesn't redirect to login.wm.o (iframes are used for that), so credentials would still be passed in the clear.
- Users in Iran can not access login.wikimedia over HTTPS
- ramp-up traffic; how?
- per language? per project? per wiki? Firefox search engine?
- deal with state firewalls blocking HTTPS
- language exceptions?
- custom blacklist, e.g. based on EventLogging data
- Carrier support for HTTPS/ip filtering is a problem
- Zero makes some requests for m.wm.o resources
|Feature||nginx 1.1||nginx 1.5||Apache 2.4||stud|
|Lightweight (no HTTP parsing)||N||N||N||Y|
|Shared session backend||N||N||Y||Y|
|Session ticket key rotation||N||kind of²||N||?|
|OCSP stapling||N||Y||Y (untested)||N|
|ECDSA certificates||?||N||Y (untested)||?|
¹ not upstream: https://github.com/eousphoros/mod-spdy ; https://code.google.com/p/mod-spdy/ does currently only support apache 2.2, see https://code.google.com/p/mod-spdy/issues/detail?id=64