Unified S3 and File System Storage Without Compromise: An In-Depth Look at Quobyte’s S3 Gateway

As part of our vision to make high-performance storage accessible to all, we are pleased to introduce our latest product update — Quobyte’s S3 gateway. It is custom-built for full integration with Quobyte’s parallel distributed file system and high performance, representing our distinct approach to integrating S3 with a parallel file system. Unlike many other products that just “slap on” an open-source gateway, Quobyte’s S3 gateway is deeply integrated with the file system, access control, and Quobyte’s advanced policy engine. This overview highlights the five fundamental features and capabilities of Quobyte’s S3 gateway:

Easy to configure and scale out in minutes

Designed for rapid deployment, the Quobyte S3 gateway requires minimal setup. To instantly access your data via S3, just configure a DNS record and start Quobyte S3 services on your desired number of machines. Since Quobyte’s S3 gateways work with standard HTTP(S) tools such as load balancers (no sticky sessions required!) or firewalls, it’s easy to set up a secure, scalable production environment in minutes.

Unified Storage without Footnotes

The gateway facilitates simultaneous data access from the file system and the S3 interface. Quobyte volumes are accessible as buckets, with the flexibility to map new buckets to new Quobyte volumes or subdirectories. Users and applications can seamlessly share data across all interfaces that Quobyte supports, i.e., S3, Linux, Windows, MacOS, NFS 3 and 4, and HDFS, allowing for versatile data movement.

Better Security with Unified Access Control

Quobyte uses and enforces the same file system ACLs on S3 as well, making it easy to keep track of user access and modification tracking of files and objects. There is no need to manage multiple ACLs for each interface. For even easier management, you can modify permissions and ACLs concurrently via the file system or S3, which is truly unified.

Unlike traditional S3, Quobyte also enforces hierarchical file system permissions on S3. This makes it easy to enforce fine-grained access control over subdirectories even within S3 and facilitates bulk modifications to a large number of files and objects, enhancing overall security and administration efficiency.

High Performance and Cost-efficient S3

Quobyte’s S3 gateways provide high-performance S3 access for gets and puts, ensuring low latency for small objects on flash storage and high throughput for large objects across both flash and HDD, achieving speeds of up to 10GB/s per gateway node.

Thanks to the tight integration with Quobyte’s policy engine, S3 objects can be stored optimally — on flash for performance and on HDD with erasure coding for efficiency. Features like automatic tiering and recoding further contribute to optimizing storage costs.

Multi-tenancy with Self-service

Quobyte’s S3 service is fully integrated with Quobyte’s multi-tenancy capability, simplifying S3 storage management for multiple tenants from the same cluster. In addition, advanced features like the hardware isolation for tenants in the Quobyte policy engine also apply to the S3 storage, ensuring secure, uninterrupted data performance for each tenant.

Why our Customers Love the Quobyte S3 Gateway

Seamless data sharing between applications that use the file system and the S3 interface to access data. In sectors like life sciences and Media and Entertainment, our customers use a variety of S3-exclusive applications that need to share data with human users and other applications/systems that are reliant on the file system. For Machine Learning, many customers operate pipelines that use S3 for data ingestion, alongside big data tools primarily accessing data through S3.

Secure data sharing with external collaborators. Many of our research customers have to share data with external collaborators, which can be a challenging task that requires a balance of avoiding data duplication while making it easy for researchers to access data from the compute cluster, even remotely. The Quobyte S3 Gateway offers a secure, easy-to-use, and high-performance solution to make high-performance storage accessible to researchers from other institutions over the internet without having to open up the network for direct file system access. With features like built-in multi-tenancy and self-service capabilities, administrators find managing this aspect straightforward and efficient.

Originally posted on Quobyte’s blog on January 30, 2024.