Vitess vs Hadoop: Stop Comparing Them, Start Designing the Right Data Backbone

Vitess vs Hadoop: Stop Comparing Them, Start Designing the Right Data Backbone

In 2020, Slack reported Vitess serving 2.3 million QPS and handling 99% of query load after a multi year migration that started in 2017. That’s OLTP scale most teams won’t ever hit, but plenty of us still have to plan like we might. In the same kind of org, Hadoop clusters can still be running nightly batch jobs that feed finance, risk, and ML.

“Vitess vs Hadoop” sounds like a choice. It’s not. Vitess scales MySQL for online traffic. Hadoop runs distributed batch and analytics. Teams get into trouble when they treat them as substitutes, then staff the wrong team, set the wrong SLOs, and build the wrong cost model.

Vitess vs Hadoop in plain English (and what each one is for)

Most CTOs I talk to get stuck on one basic question: are we scaling transactions, or are we scaling analytics? Answer that first and the rest stops being fuzzy.

Vitess is a layer that makes many MySQL servers look like one database. A Hacker News thread described the pattern as a proxy layer that sits on top of existing RDBMS instances, then routes queries across servers so apps see one logical database (Hacker News explanation). PlanetScale’s docs lay out the core building blocks: VTGate for query routing, VTTablet as the MySQL sidecar, and keyspaces and shards for partitioning (PlanetScale Vitess overview).

Hadoop is a distributed storage and compute stack built around HDFS and MapReduce, plus an ecosystem of query engines and storage layers. Starburst summarizes the classic setup as HDFS storage, MapReduce processing, and SQL engines like Hive and Impala on top (Starburst on Hadoop to lakehouses).

Here’s the mental model I use.

• Vitess runs your product database at scale.
  • Primary job: serve reads and writes for user traffic.
  • Core unit: MySQL shard with a primary and replicas.
  • Failure mode: wrong sharding key, cross shard queries, migration mistakes.
• Hadoop runs your data platform at scale.
  • Primary job: store and process large datasets for batch and analytics.
  • Core unit: files and partitions in HDFS, plus jobs that scan them.
  • Failure mode: cluster sprawl, slow queries under concurrency, high ops load.

A framing line that works in planning meetings: Vitess is for transaction scale, Hadoop is for data scan scale.

What people mean by “Vitess vs Hadoop” (and the category error)

The comparison usually shows up in two real situations.

First: a product org outgrows a single MySQL instance. Someone sees “distributed systems” and assumes Hadoop-style tooling will help. Hadoop won’t fix OLTP pain. Hadoop can store events and snapshots, but it won’t serve your checkout flow.

Second: a data org wants “one platform.” Leaders see Vitess success stories and assume Vitess can replace the data lake. Vitess won’t run your 200 TB scan for quarterly revenue recognition.

So what should a CTO compare instead?

Compare Vitess to other OLTP scaling options.

• Proxy and sharding layers: Vitess, ProxySQL, Citus (Postgres), Timescale.
• Distributed SQL databases: CockroachDB, TiDB, YugabyteDB, Spanner.

Compare Hadoop to other analytics and lakehouse options.

• Cloud warehouses: BigQuery, Snowflake, Redshift.
• Lakehouse stacks: Iceberg plus Trino, Delta Lake plus Spark.

Starburst calls out a common Hadoop pain point: Hive and Impala can bottleneck under high concurrency, and the stack needs a lot of manual care across interdependent systems (Starburst on limitations). Adastra frames the same issue as technical debt, where teams keep buying hardware and paying ops costs for systems that no longer pay back (Adastra on Hadoop debt).

The right question isn’t “Vitess vs Hadoop.” The right question is “Which backbone serves online traffic, which backbone serves analytics, and how do they share data?”

Vitess vs Hadoop decision matrix for CTOs

I use a simple model in exec reviews. I call it the Two Backbones Model.

• System of Record Backbone: the database that owns truth for product state.
• System of Insight Backbone: the platform that owns truth for analytics and ML.

Vitess can be a strong System of Record backbone. Hadoop can be a System of Insight backbone. Plenty of companies run both.

Here’s a decision matrix you can drop into your next architecture review.

A practical rule: if the top pain is “the primary DB is at 80% CPU and writes lag,” start with Vitess-style options. If the top pain is “analysts wait 20 minutes for a dashboard and clusters are fragile,” start with Hadoop modernization.

Want a quotable definition for your team?

Vitess is a control plane for sharded MySQL. Hadoop is a batch oriented data platform built around distributed storage.

How Vitess wins and where it bites you

Slack, Etsy, and Shopify all published the parts that matter to CTOs: migration mechanics, org impact, and the sharp edges.

Vitess at Slack: QPS scale and long migrations

Slack wrote that Vitess went from 0% to 99% adoption and from 0 QPS to 2.3M QPS over about three years, starting in 2017 (Slack engineering). That detail sets expectations. Vitess adoption isn’t a weekend project.

Slack also called out the real work: picking sharding keys, retrofitting apps, and learning to operate Vitess at scale. That’s engineering labor, not a vendor invoice.

Etsy: reducing manual scaling and hiding shards from developers

Etsy described a classic pre Vitess sharding setup with an index database that became a single point of failure. Etsy moved to Vitess and reported that scaling operations shifted from “months” to “days,” and shard infrastructure became hidden from developers (Etsy Code as Craft).

Etsy also described the hard part: auditing a decade of ORM queries and migrating incrementally, one table at a time, using vindexes.

Shopify: schema migrations and the cost of data model cleanup

Shopify’s Shop app team described running Vitess V14 and hitting a bug where throttled migrations could terminate after 10 minutes. The team raised the issue in the community Slack, and a fix landed. Shopify now uses the Vitess migration strategy and reports schema migrations taking “hours instead of weeks” after the move (Shopify engineering).

Shopify also shared the part CTOs should tattoo on the roadmap: the team had to add user_id columns and backfill large tables, including tables with billions of rows. Data model cleanup becomes the hidden critical path.

The Vitess gotchas CTOs should plan for

• Sharding key mistakes: wrong key forces cross shard queries and hot shards.
• Query constraints: some MySQL patterns become expensive or unsupported.
• Migration tooling maturity: version specific bugs happen, as Shopify saw on V14.
• Org friction: app teams need new rules for schema design and query patterns.

If you want a concrete playbook for safe cutovers, the “live traffic splits” pattern in Vitess resharding is worth studying. The pattern copies data, keeps it in sync via replication, then shifts reads and writes in phases (Vitess migration patterns).

Why Hadoop keeps getting replaced (and what replaces it)

Hadoop solved a 2010 problem: store lots of data on commodity disks and run batch jobs. Plenty of orgs still run that workload. The pain usually comes from everything around it.

Starburst describes the classic Hadoop stack as strong for batch, but weak for real time analytics and high concurrency. Starburst also points to manual management across many interdependent systems as a core cost driver (Starburst).

Adastra frames the business angle: Hadoop clusters become technical debt when teams keep buying hardware and staffing ops for systems that don’t fit modern cloud and governance needs (Adastra).

A LinkedIn history piece makes a useful point for CTOs: Spark inherited batch centric assumptions from the Hadoop era, and newer systems push toward vectorized engines and simpler components. The same piece calls out governance, lineage, and observability as table stakes by 2025 (History of open source data platforms).

Even older benchmarks still teach a lesson. CERN tested formats and engines on a Cloudera CDH 5.7.0 cluster with Hadoop 2.6.0, Impala 2.5.0, and Hive 1.1.0. CERN found HBase slow in their test, and pointed to long row keys as a likely cause (CERN Databases blog). Format and key design still matter, even when you “just run a cluster.”

So what replaces Hadoop in practice?

• Lakehouse stacks: Iceberg tables plus Trino, or Delta tables plus Spark.
• Cloud warehouses: BigQuery, Snowflake, Redshift.
• Streaming first pipelines: Kafka plus Flink, then batch as correction.

Starburst pitches a path from Hadoop to a lakehouse by swapping Hive and Impala for Trino and using Iceberg for table formats and schema evolution (Starburst). Vendor pitch aside, the architecture pattern is real.

Enterprise implications: what changes for CTOs

1. Budget splits into two cost curves. Vitess cost tracks QPS, write volume, and shard count. Hadoop cost tracks storage growth, scan volume, and concurrency. Finance teams hate blended budgets, so set two owners and two forecasts.

2. Talent markets diverge fast. Vitess needs strong MySQL, SRE, and app performance skills. Hadoop modernization needs data engineering, table formats, and governance skills. A single “data platform team” often fails because the skill sets don’t overlap enough.

3. Incident patterns change. Vitess incidents look like hot shards, replication lag, and bad migrations. Hadoop incidents look like queue starvation, broken partitions, and runaway scans. On call rotations and runbooks should reflect that split. Our guide to incident postmortems can help standardize the learning loop across both stacks (/tools/incident-postmortem).

4. Migrations become a leadership test. Slack’s multi year Vitess migration and Shopify’s billion row backfills point to the same truth: migrations compete with product work. CTOs need a capacity plan, not a hope. Track migration work in Command Center so the board sees risk and progress in one place (/command-center).

CTO recommendations: what to do next

Immediate Actions

1. Name the backbones. Write down your System of Record and System of Insight owners. Put names next to them. Shared ownership means no ownership.

2. Inventory the top 20 tables and top 20 queries. Use production traces and slow query logs. Tag each query as OLTP or analytics. That list drives your Vitess sharding key choice and your lakehouse partition plan.

3. Set two SLO sets. Create OLTP SLOs for p95 latency and error rate. Create analytics SLOs for freshness, job success rate, and dashboard latency. Track both in an engineering metrics dashboard so teams stop arguing from anecdotes (/tools/engineering-metrics-dashboard).

4. Run a migration spike with real load. Shopify hit migration edge cases under throttling. Run a rehearsal with production-like data sizes and peak hour constraints.

Policy Framework

1. Sharding policy: Key choice must map to a business entity like user_id or account_id. Shopify’s backfill story shows why missing keys become a schedule killer (Shopify).

2. Query policy: Cross shard joins need an explicit review. Put the review in your PR template. Treat cross shard reads like calling a remote service.

3. Data contract policy: Event schemas need versioning and ownership. Data teams can’t govern what product teams treat as “best effort.”

4. Build vs buy policy: Platform selection should run through a repeatable rubric. Use our Build vs Buy Matrix to document why you run Vitess yourself, use PlanetScale, or pick a distributed SQL database (/tools/build-vs-buy-matrix).

Architecture Principles

1. Separation of concerns: OLTP stays OLTP. Keep user facing writes out of Hadoop style systems. Keep heavy scans out of Vitess.

2. Incremental migration: Traffic shifting beats flag day cutovers. Etsy’s incremental vindex rollout and Vitess reshard workflows show the pattern (Etsy, migration patterns).

3. Table formats as a product: Iceberg or Delta becomes a long term contract. Starburst’s lakehouse framing highlights schema evolution and transactional consistency as core needs (Starburst).

4. Cost visibility: Unit economics should be visible per backbone. Use a cloud cost model for the analytics side and shard growth forecasts for the OLTP side. Our cloud cost estimator can help teams model the analytics platform cost curve (/tools/cloud-cost-estimator).

Bigger picture: the org design behind the tech

Vitess and Hadoop sit on opposite sides of the same company promise. Product teams promise fast, correct user actions. Data teams promise trusted numbers and models. Both promises fail when leadership blurs the boundaries.

I’ve watched platform teams collapse under a “one stack” mandate. The mandate sounds tidy. The mandate creates a backlog that never clears. A better pattern is two platform charters with a shared contract: events out, aggregates back.

What happens when the org hits 10x traffic and 10x data volume in the same year? The architecture only holds if the ownership model holds. Which leader owns the System of Record backbone, and which leader owns the System of Insight backbone?

Sources

1. Transitioning from Hadoop to modern lakehouses | Starburst
2. Can someone please explain how Vitess works, in plain English? | Hacker News
3. Why Hadoop No Longer Fits in the Modern Data Ecosystem | Adastra
4. A History of Open-Source Data Platforms: From Hadoop to Modern Data Lakehouses | LinkedIn
5. Vitess | PlanetScale
6. Scaling Datastores at Slack with Vitess | Slack Engineering
7. Migrating Etsy’s database sharding to Vitess | Etsy Code as Craft
8. 7 Real-Time Migrations With Vitess That Don’t Wake SREs | Medium
9. Horizontally scaling the Rails backend of Shop app with Vitess | Shopify Engineering
10. Performance comparison of different file formats and storage engines in the Hadoop ecosystem | CERN Databases blog