User:EEvans (WMF)/Sandbox/Storage Infrastructure
Note to Document Collaborators
The (tentative) objective of this document would be to shift left(ward) into a project's discovery & design phase, and set teams on a path to making sound decisions (vis-a-vis storage). Ideally, this document would encourage engineers to think about their use-cases abstractly, without bias (without —for example— preconceived ideas about storage technologies, patterns, etc).
It cannot be comprehensive (to do so would be tantamount to encapsulating all of "engineering" into one wiki page), but perhaps we can set expectations and inspire folks to reach out to us for clarification and guidance earlier.
Introduction
Applications have state, state can be persisted (or shared). How you gonna do that?
Questions
What is the structure —or shape— of your data?
Can it be expressed as objects with attributes (users, pages, etc), or is it opaque (think: images, or video)? What relationships (if any) exist between objects? What data types make the most sense? What constraints?
How will your data be queried?
Will you perform discreet look-ups using a unique key, or will it be necessary to use queries with predicates to match other attributes?
What guarantees are needed?
Is this primary, or secondary data? How serious is data loss? How tolerant is this to incorrectness (transient or otherwise)? How sensitive is your application to latency?
What is the expected storage size?
How big will your data set be? What about growth rate?
What is the expected throughput?
What do you expect for read/write request rate?
Types (data)
Primary
The canonical copy of something; The source of truth. Primary data cannot be regenerated, recalculated, or derived; A loss of primary data constitutes an outage and requires a restoration from backup.
Secondary
Data which is a copy or derivation of primary data. Examples might include cache or materialized views.
Categories (implementations)
Relational Database
A relational database management system (RDBMS), models data into one or more tables of rows containing columns, and a (unique) primary key. Typically, tables represent a type, rows instances of the type, and columns attributes of an instance. Rows in a table can be linked to rows of another using foreign keys. Correctness is made possible through the use of ACID transactions. SQL (Structured Query Language) is used to store and retrieve data. Master-slave replication is possible.
Low-latency Key/value
Stores values that are addressable by a unique, arbitrary key. Value sizes are constrained; There is no support for streaming or paging of results. Storage and retrieval operations are inexpensive and low-latency, but durability is weak or altogether non-existent. Replication is possible, typically on a best-effort asynchronous master-slave basis.
Durable Key/value
Stores values addressable by a unique, arbitrary key. Value sizes are constrained; There is no support for streaming or paging of results. Durability of writes is guaranteed. Replication is possible, typically on a best-effort asynchronous master-slave basis.
Object Storage
An object store is a special case of Durable Key/value, one where values are of arbitrary size and can be streamed (or paged) to the client.
Multi-master Row Storage
A distributed, partitioned row store with tunable consistency. Rows are identified by a (unique) primary key, the first component of which is used in partitioning the dataset. Additional attributes of a primary key can be used to establish sorted one-to-many relationships with other attributes. Additional indexing is also possible. ACID semantics are not supported, and joins and sub-queries are not possible; data de-normalization is emphasized. Replication is master-master, and network-topology aware. Clusters support high write volumes, high availability, fault-tolerance, and are capable of geographic distribution.
Content Distribution Network (HTTP caching)
Geographically distributed network of caching HTTP proxy servers.
Full-text Search
Partitioned (distributed) full-text search indexing.
Graph Storage
Provides for the maintenance and retrieval of complex hierarchical structures (graphs).