Quobyte International: Strong consistency across countries for unparalleled convenience and availability

“Success isn’t always about greatness. It’s about consistency.” — Dwayne “The Rock” Johnson.

Persistent, Consistent File Storage

The death of file systems has been greatly exaggerated.

Despite four decades and countless instances of critics declaring that the end is nigh for everything from object storage to database-oriented solutions, file systems are alive, and well, and (according to Gartner) growing in popularity.

File systems remain popular for three main reasons:

1. Users love hierarchical data organization. It’s predictable and easy to understand.
2. Application developers love the simple and ubiquitous interface for working with files. This remains consistent across virtually any programming or scripting language for all operating systems.
3. File systems on Linux — at least those that are POSIX or NFS — provide strong consistency guarantees. When you write to a file, there’s a guarantee that what was written will be exactly what later gets read. You can overwrite anywhere in the file with the same guarantee. This even works with NFS across multiple machines, albeit with closing and reopening the file. Such simplicity makes even writing a distributed application using the file system’s strong consistency guarantees remarkably easy.

What about relaxing consistency, as with an object store that has eventual consistency? This puts the burden of working with lower guarantees on the developer. Unless the use case is suitable for those lower guarantees (e.g., a shopping cart), lower consistency often means developers will waste their time reinventing the wheel.

While eventual consistency can meet firms’ needs in certain circumstances, many challenges call for persistent consistency via a single-solution, shared filesystem. A strongly consistent POSIX file system can solve certain problems that virtually no other architecture can in the current hierarchy of data center technologies.

In today’s world, we expect our online businesses to be always-on. “Five nines” or better uptime is now mandatory for any serious business. At the same time, while a local file system or an NFS server is convenient, both lack the high availability required for modern enterprises. Local machines fail. Data drops offline. Any NFS server/appliance becomes a single point of failure. Even redundant NFS appliance hardware doesn’t help, because modern enterprise scales mean being prepared for outages that can span whole data centers or even metro areas.

This is where real distributed storage comes in, based on software running across geographically disparate machines, all coordinated to ensure your file system provides what was promised: strong consistency.

However, achieving strong consistency across wide-area networks (think: internet or leased fiber lines) is much harder than it sounds because:

1. Hardware can fail.
2. Inter-system communication may suffer from packet loss or interruption. Sometimes, this results in strange patterns where you can only send a message in one direction or where several machines are able to communicate within their “partition” but not the outside world.
3. Entire servers can fail. In such cases, from the outside, you can’t immediately answer questions such as “is the server dead?”, “is it my network?”, “is it the network in between?”, or “is the machine just overloaded and really slow to respond?”

Achieving strong consistency is worth the effort, though. A strongly consistent wide-area distributed file system can be essential in a wide range of enterprise use cases. Yet, it’s surprisingly hard to find consistent storage systems that are easy to use and able to support clusters across international-scale geographies. In fact, there may only be one storage system that fits these simple criteria.

Quobyte is a software-storage system that implements strong consistency using quorum replication and peer-to-peer leader election. While most use cases for this replication involve low-latency, high-performance I/O inside a single data center, the algorithms Quobyte developed are so robust that you can stretch a Quobyte cluster geographically, easily handling latencies of 50 ms or more between replicas.

Wrong Takes on the Right Idea

The metro cluster is another common but flawed approach to achieving consistency. A metro cluster stretches a cluster across two or more data center locations, typically in disparate “metro” areas. Metro cluster users must devise ways to share states between locations in real-time. Frequently, though, the solution boils down to a form of RAID 1-type mirroring. Problems arise when a failure occurs. Given two systems, both are prone to thinking the other node is dead and will default to updating locally. This is a quick route to “split-brain” database forking. Manual intervention is often necessary during failover and recovery, which further degrades the approach’s timeliness and reliability.

Additionally, metro clusters often fail to maintain strong consistency in the face of distance and associated latency challenges. Metro clusters typically start to lose their consistency around 100 kilometers, or 2 milliseconds of latency, making them a poor fit for interstate or cross-border applications. In the same vein, IBM’s flagship storage product won’t tolerate more than 2 or 3 ms of latency. That’s simply not enough for a truly wide-area distributed storage system. Yet despite these drawbacks, the appliance vendors that offer metro clusters charge a (not so) small fortune for their so-called solutions.

Quobyte: A Better Approach to Distributed Storage

Quobyte, the storage platform, is built on 15 years of distributed systems research. It was designed from the ground up to provide sound, reliable replication, with resilience baked into the core rather than pasted onto archaic solutions in a patchwork fashion. In fact, Quobyte can tolerate far higher latencies than conventional distributed storage approaches and deliver resilience superior to any other market option.

Quobyte succeeds because its solid, distributed systems approach provides four key benefits:

1. Resiliency against failures
2. Resiliency against latency
3. Resiliency against machine misbehavior
4. Unlimited scalability

For developers, Quobyte also represents more opportunity in terms of both time and application potential. The more people try Quobyte, the more useful things they find to do with it, in part because the interface is so simple to program to. Quobyte leverages a programming interface that has been common for 40 years and exists in countless programming textbooks. Developers already know how to write to Quobyte, and they can very quickly come up with mechanisms for using it. The software is simple, easy to use, and loaded with properties and tools that would otherwise be very expensive.

For enterprises needing a scalable file system with strong consistency, Quobyte is simply the best tool available — and developers know that having the right tools translates into the highest project efficiency. As many engineers have learned, strong consistency makes life easier. Having a file system abstracted into a form that’s easy to use allows them to spend less time-solving problems outside their job description and more time focused on core challenges. You wouldn’t put a scientist in a lab with little more than rulers and calculators and say, “Here, just build anything else you need.” You would supply them with the best tools possible and allow them to stand on the shoulders of giants.

Deploying widely distributed storage with strong consistency is like outfitting a science lab with the most cutting-edge, productive infrastructure. If you’re a manager, VP, or CIO, it’s time to empower your IT team with the right tools for success.

Originally posted on Quobyte’s blog on July 10, 2023.