EKS + Databricks quotas: Decide your stateful migration path
Stateful and high-throughput systems are increasingly being migrated onto managed platforms (EKS, Databricks), with success determined less by app code and more by storage identity, quota management,...

Table of Contents
Decide whether your next stateful migration will be “data-move” or “storage-attach”, and budget engineering time for managed-service quota work before you commit to dates. InfoQ’s write-up of Yelp’s Cassandra move to Amazon EKS shows a migration that succeeded by moving volume identity, not data, while Databricks’ own post on raising cluster rate limits shows scale work that succeeded by treating quotas as a product surface, not an ops afterthought.
The common failure mode is predictable: CTOs approve a platform migration on the assumption that application compatibility is the hard part, then the program stalls on storage semantics, quotas, and operational constraints that were never on the Gantt chart.
Should we migrate Cassandra to EKS by moving EBS volumes or by streaming data?
“Moving EBS volumes” means preserving the underlying Amazon Elastic Block Store disks and re-attaching them to new Kubernetes pods, instead of copying data node-to-node during a cluster rebuild. In November 2026, InfoQ reported on Yelp migrating Cassandra to EKS by re-pointing existing EBS volumes, explicitly choosing attachment mechanics over data streaming.
That choice matters because stateful migrations fail in two expensive ways: (1) data movement takes longer than the maintenance window, and (2) partial moves create long-lived dual systems that become the new source of incidents.
What Yelp’s approach signals is a broader shift in how mature teams migrate stateful systems onto managed Kubernetes:
- Treat the disk as the unit of continuity. Volume identity becomes the “truth” that preserves data locality and reduces copy time.
- Reduce the number of moving parts during the cutover. Re-attaching volumes avoids a second distributed system problem (streaming) while you are also changing the control plane (legacy Kubernetes to EKS).
- Pull risk forward into rehearsal. Volume re-attachment still has sharp edges (scheduling, AZ constraints, CSI behavior, StatefulSet identity), but those are testable with dry runs.
A CTO-level implication: the migration strategy is not a purely technical preference. The strategy determines whether you need weeks of parallel run, how you roll back, and whether you can give the business a credible cutover date.
What do Databricks cluster rate limits have to do with platform architecture?
A “cluster rate limit” is a platform-enforced ceiling on how many clusters can be created, started, or modified in a given time window, plus related control-plane throughput constraints. In November 2026, Databricks described how its Compute Platform team raised Databricks cluster rate limits by 7.5×, framing the work as infrastructure engineering to remove scaling bottlenecks.
Quota and rate-limit work looks like plumbing until it blocks a launch.
The through-line with the Yelp EKS migration is that managed platforms have “physics” you cannot wish away:
- Control planes are shared and protected. Even if your data plane is fine, control-plane throughput can throttle you.
- Your org’s growth shows up as bursty automation. CI, autoscaling, ephemeral environments, and scheduled jobs all create synchronized demand spikes.
- The platform boundary moves constraints into contracts. The moment you rely on a managed service, you inherit its limit model and escalation path.
CTOs should read the Databricks post as a warning label: if your roadmap includes more automation, more ephemeral compute, or more teams self-serving infrastructure, quota management becomes architecture.
Which hidden constraints will break our migration plan (EKS, EBS, and managed quotas)?
A “hidden constraint” is any platform limit or attachment rule that is absent from your design doc but present in production behavior. The sources in this batch point to three classes of constraints that repeatedly derail plans.
1) Storage identity constraints (EBS and Kubernetes scheduling).
Yelp’s EBS re-attachment approach works when you can control placement and keep volumes attachable to the right nodes and Availability Zones. The constraint is not Cassandra. The constraint is volume locality and orchestration behavior. InfoQ’s Yelp write-up is a reminder to explicitly model:
- AZ pinning and failure domains
- StatefulSet identity and PVC/PV lifecycle
- Rollback plan that preserves volume integrity
2) Managed-service quota constraints (rate limits, concurrency ceilings, API throttles).
The Databricks rate-limit work shows a mature pattern: treat limits as tunable system design parameters, not as support tickets you file when something breaks. Databricks’ 7.5× rate limit post should push CTOs to require a “limit budget” in any plan that increases automation.
3) Operational constraints (blast radius and change coupling).
LeadDev’s postmortem-style analysis, “Microsoft’s outage started with a fix”, reinforces the operational side: migrations and platform changes tend to ship as “safe” fixes until they meet production coupling. The lesson for stateful and quota-bound systems is simple. Change control has to match blast radius.
One short sentence: rehearsal is cheaper.
How should a CTO budget time and risk for managed-platform migrations?
The tactical work is specific, but the decision pattern is general.
-
Pick your continuity unit early. If your continuity unit is “data”, you are signing up for streaming, dual-write, or long replication windows. If your continuity unit is “disk”, you are signing up for storage attachment constraints and orchestration rehearsal. Yelp picked disk continuity for Cassandra on EKS, and the choice shaped the whole migration plan (InfoQ on Yelp’s approach).
-
Put quotas in the architecture review. A platform limit that you discover in production is not an ops issue. The limit is an architectural constraint that should be documented like any other non-functional requirement. Databricks’ rate-limit work is a concrete example of investing in control-plane scalability before it becomes a business outage (Databricks on raising cluster rate limits).
-
Demand a rollback path that matches the migration strategy. Disk-attach migrations roll back differently than data-stream migrations. Quota-driven scale changes roll back differently than code changes. LeadDev’s outage analysis is a useful forcing function: if a “fix” can cause a large outage, a migration can too (LeadDev on the Microsoft outage starting with a fix).
-
Treat managed platforms as products you integrate with, not just infrastructure you rent. Escalation paths, limit increases, and support contracts become part of delivery capability. Platform selection is no longer only about features and price.
What to decide this quarter
-
Choose a stateful migration strategy (CTO, deadline: next architecture review). Decide “storage-attach” vs “data-move” for each stateful system, and require a written rollback plan per system before approving a cutover date.
-
Run a quota and limit audit for your top two managed platforms (Head of Platform or SRE Manager, trigger: before the next major scale event or migration kickoff). Document cluster creation/start limits, API throttles, and any support escalation lead times.
-
Schedule one full cutover rehearsal for the highest-risk stateful migration (Engineering Manager owning the system, deadline: 30 days before production cutover). The rehearsal must include failure injection for AZ or node loss and a timed rollback.
-
Instrument control-plane health and throttling signals (SRE lead, deadline: end of quarter). Add dashboards and alerts for the platform APIs you depend on (cluster start latency, throttling rates, attach/detach errors).
-
Use the Vendor Lock-in Exit Strategy tool (CTO or Principal Engineer, trigger: before signing a 12-month commitment). The tool forces an explicit exit plan and measurable triggers for when portability work starts, which is the only practical defense against quota surprises turning into long-term dependency.
Sources
▶ Interactive tool
Put this into practice — free, no sign-up
Run your own numbers in these interactive tools built for exactly this decision.