How Instagram Serves 2 Billion Users on Postgres: Sharding, PgBouncer and Snowflake IDs

The video opens with a provocation: Instagram serves 2 billion users on Postgres, while Reddit, Notion, Discord and Strava keep large parts of their load on Postgres too. All of them evaluated the alternatives, all of them stayed. The question follows naturally: what do these teams know that the 'Postgres does not scale' mantra misses?

The story starts in 2010: three engineers, rented virtual machines, an object store for photos and a single Postgres for everything else (accounts, metadata, comments, likes, the follow graph). At 10 million users in 2011 the picture is unchanged. At the 2012 billion-dollar acquisition there are 27 million users on one 2-terabyte database. The lesson the video saves for the end germinates here: well-run boring technology beats poorly-run new technology.

By 2012 the single machine hits the ceiling: the biggest rentable server of the era is running out of memory, disk throughput is saturated, there is nowhere left to scale vertically. The reflex the video recommends matters: instead of panic-switching databases, find exactly what is clogged and fix that. And the first clog turns out different from what most people guess.

The first wall is not data size, it is connections. Each connection holds about 1.3 megabytes; when dozens of app servers each open their own small pool (the worked example is 50 servers times 30 connections) you get 1500 connections eating 2 gigabytes before a single useful query runs. The fix is PgBouncer: a lightweight proxy between app and database that multiplexes thousands of incoming connections onto a few dozen real ones. Memory goes back to page cache, sorting and query planning. The claim is sharp: for any serious Postgres setup without a pooler, this is the highest-leverage fix of the week.

Pooling buys breathing room, but load keeps growing and the standard chorus arrives: 'time for NoSQL.' The video rejects the prescription, arguing that partitioning pain does not disappear on Cassandra, it just hides behind another abstraction. And there is no return ticket: once you switch, coming back is nearly impossible. Instagram instead decides to make Postgres itself horizontal.

The sharding decision is made, but the shard key seals its fate. The natural candidate is the user id: one user's photos and likes live on one shard, 'show my profile' hits a single shard, fast and clean. But the defining query of a social network is 'show photos from people I follow,' and when 200 follows scatter over 50 shards it becomes a nightmare: fanout to 50 databases, merge and sort in the app layer, a feed as slow as the slowest shard, brand-new failure modes. The tradeoff is permanent; the team bets it can solve cross-user queries in the app layer with caching, smart fanout and precomputed feeds.

Here comes the idea the video calls 'the architecture itself': separate how data is partitioned from where it physically lives. The team defines a few thousand logical shards (Postgres schemas with identical tables), all sitting on one physical machine at first. The app only asks 'which logical shard owns this user.' When a machine fills up, no data is rewritten; logical shards are copied to a new box with built-in streaming replication and the mapping table is updated (in the example, half of a nearly-full node's shard range moves to the new node and both settle at half capacity). With the shard count never hardcoded, growth becomes a configuration change.

Classic auto-increment counters collide instantly in this layout: if every shard mints its own 1-2-3 series, two different photos share one id. Random 128-bit ids are huge and unordered, which makes time queries expensive; a central ticket server is a single point of failure; an external Snowflake service is one more thing to monitor. Instagram's answer is a three-part 64-bit number: a 41-bit millisecond timestamp from a custom epoch (41 years of room), a 13-bit shard id (up to 8 thousand shards), a 10-bit per-millisecond sequence (hundreds of thousands of ids per second per shard). It runs as a small Postgres function on every shard, no coordination, no external dependency; with the timestamp in the high bits, sorting by id means newest first, no separate time index needed. The pattern later spreads across the industry to Discord and Slack; the practical warning is to adopt it on day one, not after a billion rows.

The video also spotlights three underused features already inside Postgres. Partial indexes: indexing only the last 30 days of a billion-row table shrinks the index tenfold and keeps it small as old data ages out. Functional indexes: indexing the first 8 characters of a 64-character token instead of the whole string keeps lookups fast at a tenth of the size. Logical replication: streaming every insert, update and delete to the search index, cache invalidation and analytics warehouse in real time, instead of hand-building dual writes and queues in the app.

One honesty note: this is the 2010-2015 story; after the acquisition Instagram's follow graph moved into Meta infrastructure and today lives in TAO, a distributed graph system. But the video's five rules still hold: do not shard until forced (the team waited until 27 million users), separate logical from physical, adopt Snowflake-style ids on day one, fix connection pooling first, and buy new infrastructure only for genuinely new problems (vector search, time series, graph). The closing thesis: the bottleneck is never the database, it is the architecture around it.