Wikimedia Cloud Services team/EnhancementProposals/cloudswift

From Wikitech

This page contains information on our plans to introduce the cloudswift service to Cloud VPS, a new Openstack API for object storage.

Why

Having Object-storage-as-a-Service (swift API) on Cloud VPS enables several possibilities and opens the door for supporting interesting use cases, such as:

  • storage decoupling for some Toolforge components (like docker registry, tools static files for webservice, etc)
  • options for backup/snapshot features for other Cloud VPS/Toolforge components
  • activation of additional functionalities on other openstack components that require it
  • adoption of other openstack components that require it
  • in the future, offering a Object-storage-as-a-Service product/API opened to the internet at large for the wikimedia technical community to use.

The openstack swift component is mature, and we are confident with adopting it. We anticipate high adoption of this service.

Other SRE teams on the WMF already use it for other purposes and in other contexts not related to WMCS, Cloud VPS or Toolforge (see swift).

How it works

We see swift as a simple REST API-based storage solution. You can GET/POST/PUT/DELETE/etc files, object blobs, and that's about it.
The openstack swift component enables quotas, multi-tenancy and all the other controls we need to successfully introduce the service on Cloud VPS.


Our idea for this service is to leverage our ceph distributed block storage cluster.
We will use a component called radosgw which will be the responsible for translating swift-API calls into ceph-backend storage actions.

Our plan is to have dedicated hardware servers for running radosgw, to isolate load, capacity and failure from cloudcontrol servers.

Proposed setup

The cloudswift servers are a special kind of API servers, similar to cloudcontrols, but with the particularity that they need direct and high bandwidth network connectivity with the backend ceph farm. The radosgw process can demand certain CPU/RAM capacity. The cloudcontrol servers already run galera, haproxy, and the rest of openstack services, so we decided against coupling radosgw and the swift API into cloudcontrol servers.

However, like the openstack APIs that live on cloudcontrol servers, we would like to expose the swift API to Cloud VPS VMs (and eventually the internet at large). This means the swift API must be reachable on a public IPv4 address.

The WMF SRE/InfraFoundation team requested for the 2 hardware servers to be racked in cloud-dedicated racks and therefore connected to cloud-dedicated switches (cloudsw). This is good for several reasons, but in particular, it will solve the high-bandwidth connectivity requirements, since the ceph farm is also connected to the same switches.

In summary:

  • we need the swift API to be reachable via public IPv4 address.
  • we need 10G full bandwidth to/from the ceph farm, with a connection as direct as we can get.
  • we decided to not colocate swift APIs (radosgw) in cloudcontrol hardware servers, therefore we're using 2 new hardware servers.

The proposed setup is to have each cloudswift hardware server dual-homed:

  • PRIMARY: 10G NIC connected to cloudsw switch on the cloud-host1-eqiad VLAN, with 10.x.x.x private addressing. This interface will be used for SSH management/install/dhcp/monitoring/etc and also for high-bandwidth connectivity with the ceph backend farm.
  • SECONDARY: 10G NIC connected to cloudsw swtich on a NEW vlan/subnet with public IPv4 addressing from the cloud pool (usually 185.15.56.x range). This interface will be use for service offering to Cloud VPS VMs (and potentially other clients over the internet later on).


The new service vlan/subnet created to host the public IPv4 addresses is, by all means, new:

  • the L3 gateway for that vlan/subnet should be the cloudgw servers, which already acts as L3 gateway for other cloud-dedicated public-IPv4 networks.
  • cloudgw allows for extended controls and firewalling over cloud-specific traffic. Cloud VPS VMs could contact the swift API without NAT being involved if required, given all that network traffic would not be leaving the cloud realm.

Additional notes

  • If the proposed setup works as expected, in the future we may consider migrating existing cloudcontrol servers to a similar setup (double-homed, one NIC for management other for API services).
  • We ran a swift PoC on codfw1dev. That PoC didn't use the new network layout described above, but instead a similar setup to what cloudcontrol servers were using. Once we integrate the new design, the codfw setup will be updated accordingly (see T292427).

See also

TODO

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Wikimedia_Cloud_Services_team/EnhancementProposals/cloudswift&oldid=1932640"