Jump to content

HTTPS/Future work

From Wikitech
This page contains historical information. It may be outdated or unreliable.
This page appears to have moved to phabricator:T104681.

Scaling up SSL infrastructure

  • push them to caching frontends
  • do we have enough CPU on frontends for this?
  • need (better) statistics + benchmark likely solutions
  • Determine if load from logged-in users is sufficient for benchmarking. Switch certain projects to alternate software/ciphers/pfs/etc to test relative load (possibly with depooling)
  • Faidon: Apache testing (vs nginx)
  • Will it be cheaper to expand the frontends, or to expand the terminator cluster?
  • Split Varnish frontend from backends (SSDs, lots of mem), put SSL terminators with frontends (just CPU)
  • Switching to Varnish will eliminate IPV6 protocol proxies
  • get rid (or replace) LVS source hashing
  • stud? needs Varnish proxy protocol support (patch written by Mark, nearing completion)
  • Apache 2.4 + memcached
  • nginx + session sharing patch (to be written by us or Nginx Inc.)
  • all above: session tickets need to be either disabled or shared, example strategy: https://blog.twitter.com/2013/forward-secrecy-at-twitter
  • mark's wcsh LVS scheduler (weighted consistent source hashing)

Security enhancements

  • Zero team must be involved prior to anon default HTTPS (same partner carrier issues as non-HTTPS-countries)
  • Certificate pinning
  • Only implemented in Chrome right now, as far as I know
  • box-to-box traffic (mark will investigate ipsec)

Performance enhancements

  • stop serving full chains, omit root CAs already known to browsers
  • switch to individual wildcard certs & SNI, use large unified only as fallback
  • investigate OCSP stapling (esp. with regards to intermediate CAs)
  • investigate other enhancements to fit initial negotiation in one RTT

Other considerations

  • SPDY -- postpone for "v1.1"
  • Poorly supported in nginx
  • mod_spdy doesn't work for 2.4 yet - it's also a big hack with unmerged patches to Apache core
  • None support ALPN as of this time
  • Is also a security benefit thanks to pipelining for paylod-size attacks.
  • Not without collapsing domains (all requests via one connection)
  • Wait and see what development / standardization (HTTP 2.0) looks like when we're ready for HTTPS-by-default
  • investigate ECDSA certificates -- faidon to investigate more, drop it for v1.1 if it's too complicated
  • why?
  • smaller than 2048-bit RSA, helps with performance (RTTs)
  • more secure, 256-bit equivalent to 3072-bit RSA
  • less computational cost
  • where to procure? cost?
  • how/when to fallback to RSA (Google requires SNI + *no* deflate support for ECDSA)
  • SSL terminator UDP logging - Ken will talk to Toby and get the shutdown into the analytics plan
  • Remove?
  • The logs are duplicated. We can likely eliminate them if we track XFP
  • Fix?
  • Fixing udp2log is probably pointless at this point as it's about to be replaced with Kafka...

HTTPS by default

  • only solution at the moment is to enable by language-wiki, which is suboptimal (especially in terms of feedback)
  • login.wikimedia.org
  • separate IP to deal with China?
  • Needs to be more than separate IP. If login happens over https, the redirection cookie is added. Also, login itself occurs per wiki and doesn't redirect to login.wm.o (iframes are used for that), so credentials would still be passed in the clear.
  • Users in Iran can not access login.wikimedia over HTTPS
  • ramp-up traffic; how?
  • deal with state firewalls blocking HTTPS
  • language exceptions?
  • GeoIP?
  • custom blacklist, e.g. based on EventLogging data
  • Carrier support for HTTPS/ip filtering is a problem
  • Zero makes some requests for m.wm.o resources

Server comparison

Also see https://istlsfastyet.com/#server-performance

Feature nginx 1.1 nginx 1.5 Apache 2.4 stud
TLSv1.2 Y Y Y N
Performance + + ? ++?
Lightweight (no HTTP parsing) N N N Y
Shared session backend N N Y Y
Session ticket key rotation N kind of² N ?
OCSP stapling N Y Y (untested) N
ECDSA certificates ? N Y (untested) ?
SPDY N Y Y¹ (untested) N/A

¹ not upstream: https://github.com/eousphoros/mod-spdy ; https://code.google.com/p/mod-spdy/ does currently only support apache 2.2, see https://code.google.com/p/mod-spdy/issues/detail?id=64

² multiple keys can be specified; rotating those then needs to be triggered external of nginx; cf. [1] [2]