Netflix has introduced RAW Hollow (Read After Write Hollow), a groundbreaking in-memory, co-located, compressed object store that represents a significant evolution in distributed data storage technology. This system extends Netflix’s original Hollow framework to support both read and write operations while maintaining the exceptional performance characteristics that made Hollow successful for read-only workloads. RAW Hollow addresses the critical need for stateful services that can handle small to medium datasets with ultra-low latency, high availability, and tunable consistency guarantees.

Introduction and Motivation

The Challenge of Modern Stateful Services

Netflix processes billions of requests daily across its global streaming platform, serving over 300 million users worldwide. While the company manages petabytes of data overall, many critical stateful services operate on relatively modest datasets that are orders of magnitude smaller than platforms with user-generated content. These services require exceptional reliability and performance characteristics that traditional database solutions struggle to provide consistently.

The primary challenges Netflix identified with existing solutions include performance unpredictability with SQL databases, cache synchronization complexities when supplementing primary stores with caching layers, and network overhead even with fully in-memory databases that require remote calls for every request. These limitations led to the development of RAW Hollow as a next-generation solution for stateful applications.

Evolution from Hollow

RAW Hollow builds upon Netflix’s proven Hollow technology, which has been battle-tested for over ten years as a read-only, high-density near-cache solution. The original Hollow employs sophisticated compression and memory pooling techniques, enabling applications to cache and query entire datasets in main memory with minimal heap pressure. RAW Hollow extends these capabilities to support write operations while adding robust data store guarantees such as atomicity and tunable consistency.

RAW Hollow: Technical Architecture and Core Components

RAW Hollow implements a sophisticated distributed architecture designed around a single-tenant model where each dataset namespace is served by its own deployment. The system consists of four distinct instance roles working in coordination with external services.

RAW Hollow Architecture: Core components and data flow showing how Netflix’s in-memory object store achieves high performance and
strong consistency

Writers: The Write Coordination Layer

Writers serve as the primary entry point for all write operations in a RAW Hollow deployment. The system maintains multiple writer instances with leader election managed through ZooKeeper, ensuring only one writer is active at any given time. Each active writer instance contains three critical components: a Hollow Consumer that wraps the base dataset, an internal hash table managing in-flight changes, and a message queue for coordinating operations.

The write process follows a rigorous protocol where each incoming request receives a monotonically increasing offset and is added to the message queue. The system implements synchronous replication, meaning the writer only acknowledges a write operation after all available logkeepers have successfully committed the message. This approach ensures strong durability guarantees while maintaining high write throughput of up to 1,000 writes per second with 10 KB serialized payloads.

Logkeepers: Simplified Durability Infrastructure

Logkeepers represent one of RAW Hollow’s most innovative architectural decisions, implementing an intentionally simple design focused on reliability and consistency. Each logkeeper operates as an in-memory circular log with a static 1 GB configuration, requiring no dynamic memory allocation during operation. When receiving messages from writers, logkeepers simply update internal pointers, making the commit operation extremely efficient.

The distributed nature of logkeepers provides exceptional fault tolerance, with instances deployed across multiple availability zones within an AWS region. The system maintains quorum-based coordination through ZooKeeper, automatically updating the quorum when logkeepers become unavailable and ensuring writes fail if the quorum falls below the configured minimum threshold. This design choice eliminates the need for logkeepers to scale linearly with application read requests, as local clients handle all read operations independently.

Producers: Dataset Synchronization and Distribution

Producers bridge the gap between the real-time write path and the batch distribution mechanism inherited from the original Hollow architecture. A single active producer, coordinated through ZooKeeper, pulls the latest log entries from logkeepers and updates the base Hollow dataset. These updates propagate through Gutenberg, Netflix’s pub/sub service, to both writers and all local clients throughout the system.

The producer implementation includes sophisticated coordination mechanisms that trigger logkeeper log pointer resets when writers and clients receive new Hollow updates, ensuring synchronized state transitions across the entire system. Additionally, producers create comprehensive snapshots of in-flight changes every 30 seconds, uploading them to Gutenberg for persistence in S3, which provides the system’s ultimate durability guarantee with Amazon’s advertised 11 nines of reliability.

Local Clients: Materialized Views with Real-Time Updates

Local clients represent RAW Hollow’s most distinctive architectural feature, enabling any application instance to maintain a complete, materialized view of the dataset with real-time updates. Each client combines a Hollow Consumer with a hash table of in-flight changes, similar to writer instances but optimized for read operations. The system generates Java APIs for local clients, allowing applications to add custom indexes for arbitrary field access patterns.

Clients maintain constant connectivity with logkeepers through long polling mechanisms, ensuring real-time propagation of changes with single-digit millisecond latencies. This architecture enables RAW Hollow to scale datasets up to 100 million records per entity while maintaining microsecond read latencies for local operations. The co-located nature of data eliminates network round trips for read operations, providing exceptional performance characteristics that traditional distributed databases cannot match.

RAW Hollow: ACID Properties and Consistency Guarantees

RAW Hollow implements comprehensive ACID properties while providing flexible consistency models that allow applications to balance performance with correctness requirements based on specific use cases.

Atomicity Through Bulk Operations

RAW Hollow ensures atomicity through its Bulk Update API, which groups multiple operations into atomic transactions where all operations either succeed together or fail as a unit. This mechanism prevents partial updates that could lead to inconsistent application state, particularly important for complex operations that span multiple data entities. The system’s atomicity guarantees extend across the entire distributed architecture, ensuring that partial failures in any component cannot result in corrupted or incomplete state transitions.

Consistency and Schema Validation

The system maintains consistency by ensuring all dataset transitions move from one valid state to another through comprehensive schema validation and constraint checking. RAW Hollow supports strong schema enforcement with validation on primary keys, though it does not currently support foreign key constraints. The Conditional Bulk Update API provides additional consistency guarantees by allowing applications to specify preconditions that must be met before operations execute, preventing race conditions and ensuring application-level invariants.

Isolation and Transaction Control

RAW Hollow implements Read Committed isolation levels, ensuring that only committed changes become visible outside transaction boundaries. This approach prevents dirty reads while maintaining high performance for concurrent operations. The system’s isolation guarantees work in conjunction with its distributed architecture to ensure that concurrent transactions cannot interfere with each other’s execution, even across multiple writer instances during failover scenarios.

Durability Through Multi-Layer Persistence

The durability implementation combines multiple persistence mechanisms to ensure acknowledged changes survive system failures. Synchronous replication to all available logkeepers provides immediate durability for acknowledged writes, while the regular snapshot mechanism to S3 provides long-term persistence guarantees. The system’s durability has proven acceptable for tier-0 services at Netflix, demonstrating its production readiness for mission-critical applications.

RAW Hollow: CAP Theorem Trade-offs and Consistency Models

RAW Hollow’s most sophisticated feature lies in its ability to dynamically balance CAP theorem trade-offs based on application requirements, providing both eventually consistent and strongly consistent operational modes.

RAW Hollow CAP Theorem Trade-offs: Comparison between Default AP mode and Optional CP mode showing consistency vs
availability choices

Eventually Consistent Mode (AP System)

By default, RAW Hollow operates as an AP (Available and Partition-tolerant) system, prioritizing high availability and fault tolerance over immediate consistency. In this mode, each local client maintains a complete copy of the dataset, ensuring the system remains highly available even during network partitions or component failures. Read operations execute entirely locally with microsecond latencies, and the system propagates updates throughout the cluster with single-digit millisecond latencies under normal conditions.

This mode proves ideal for applications where temporary inconsistencies are acceptable in exchange for exceptional performance and availability. Examples include content catalogs, user preference systems, and recommendation engines where slight delays in consistency do not impact user experience significantly.

Strong Consistency Mode (CP System)

RAW Hollow allows applications to request strong consistency at the granularity of individual requests, transforming the system into a CP (Consistent and Partition-tolerant) mode for those specific operations. When clients receive requests requiring strong consistency, they temporarily halt processing to ensure complete synchronization with the latest changes before proceeding. Subsequent operations within the same request scope continue to benefit from local execution, amortizing the consistency overhead across multiple operations.

This selective approach enables applications to use strong consistency only when necessary, such as for financial transactions, user authentication, or other operations where immediate consistency is critical. The ability to choose consistency levels per request provides unprecedented flexibility for application developers to optimize performance while maintaining correctness where required.

RAW Hollow: Performance Characteristics and Scalability

RAW Hollow delivers exceptional performance metrics that demonstrate its suitability for latency-sensitive applications at massive scale.

Read Performance

Local read operations achieve microsecond latencies due to the co-located architecture that eliminates network round trips. The system can handle queries across millions of records in milliseconds, enabled by the compressed in-memory representation and optional custom indexing. This performance characteristic makes RAW Hollow particularly suitable for applications requiring real-time data access patterns, such as personalization engines and content recommendation systems.

Write Performance and Throughput

The system demonstrates robust write capabilities, handling up to 1,000 writes per second with 10 KB serialized payloads. Write operations propagate throughout the cluster with single-digit millisecond latencies, ensuring that updates become visible across the system quickly. The synchronous replication to logkeepers ensures durability without significantly impacting write performance, making the system suitable for applications with moderate write loads requiring strong durability guarantees.

Dataset Scale and Memory Efficiency

RAW Hollow leverages advanced compression techniques inherited from the original Hollow framework to support datasets containing up to 100 million records per entity. The system’s memory efficiency stems from sophisticated encoding, bit packing, and deduplication algorithms that maximize data density while maintaining high query performance. This scale capability positions RAW Hollow as suitable for large-scale metadata management, user directories, and content catalogs that require both comprehensive coverage and high performance.

Production Deployment and Operational Excellence

Netflix Production Usage

Since its introduction in 2023, RAW Hollow has achieved remarkable adoption within Netflix, reaching over 500 deployments including more than 160 production deployments across critical tier-0 services. This rapid adoption demonstrates the system’s production readiness and operational stability in demanding real-world environments.

Operational Tools and Monitoring

RAW Hollow inherits Hollow’s comprehensive operational toolset, providing detailed insights into dataset health, change history, and version diffing capabilities. The system supports zero-copy cloning for rapid environment setup and testing, while comprehensive change tracking enables precise identification of data modifications and their timing. These operational features prove essential for maintaining large-scale production deployments with complex data dependencies.

Disaster Recovery and Rollback Capabilities

The system’s versioned data model enables rapid rollback capabilities, allowing operators to quickly revert to previous dataset states when issues are detected. Because state transitions execute quickly across the entire fleet, corrective actions can take effect within seconds of detection, minimizing the impact of operational issues. This rapid recovery capability proves crucial for maintaining high availability in production environments.

RAW Hollow: Real-World Applications and Use Cases

RAW Hollow’s unique characteristics make it particularly suitable for a wide range of applications that require high performance, reliability, and scalability.

Real-world application domains for RAW Hollow-style in-memory object stores showing relative applicability and key characteristics

Streaming Platform Infrastructure

Video Metadata Management

Streaming platforms like Netflix require ultra-fast access to vast catalogs of content metadata including titles, descriptions, cast information, and technical specifications. RAW Hollow excels in this domain by providing microsecond access to comprehensive metadata while supporting real-time updates as new content is added or existing content is modified. The system’s ability to handle up to 100 million records per entity makes it ideal for large content catalogs that need to support millions of concurrent users.

Content Recommendation Systems

Modern streaming platforms rely heavily on sophisticated recommendation algorithms that need immediate access to user preferences, viewing history, and content relationships. RAW Hollow’s co-located architecture eliminates the latency introduced by network calls to remote databases, enabling recommendation engines to evaluate complex algorithms across millions of data points in real-time. The system’s eventual consistency model proves sufficient for recommendation use cases where slight delays in incorporating new user interactions do not significantly impact recommendation quality.

User Personalization and Preferences

Personalization systems require rapid access to user profiles, preferences, and behavioral data to customize user experiences dynamically. RAW Hollow’s local read capabilities enable personalization engines to process user-specific customizations without introducing latency that could degrade user experience. The system’s write performance characteristics support real-time updates to user preferences and behavioral patterns while maintaining consistency across user sessions.

Content Delivery Network Operations

Network Topology and Routing

Content delivery networks require real-time access to network topology information, server status, and routing decisions to optimize content delivery paths. RAW Hollow’s implementation in Netflix’s Open Connect platform demonstrates its effectiveness in storing network metadata that enables control plane instances to rapidly initialize and maintain complex in-memory data structures reflecting network topology. The system’s low-latency characteristics prove essential for live streaming applications where routing decisions must be made in real-time.

Edge Server Management

CDN operations require continuous monitoring and management of thousands of edge servers across global deployments. RAW Hollow can store server status information, capacity metrics, and configuration data, enabling centralized management systems to make intelligent decisions about traffic routing and load balancing. The system’s fault tolerance ensures that CDN operations continue even when individual components experience failures.

Geographic Distribution and Caching Strategies

Global CDN deployments require sophisticated coordination of caching strategies across geographically distributed points of presence. RAW Hollow’s ability to maintain synchronized datasets across multiple regions while providing local access to critical operational data makes it ideal for coordinating global CDN operations. The system can store cache policies, content popularity metrics, and regional preferences while ensuring that updates propagate efficiently across global deployments.

Identity and Access Management

Universal Identity Directories

Identity management systems require high-performance access to user authentication information, permissions, and organizational hierarchies. Netflix’s OneID implementation demonstrates RAW Hollow’s effectiveness in this domain, where the system replaced a complex multi-layered persistence stack comprising Cassandra, ElasticSearch, and NeptuneDB while improving overall latencies. The system’s strong consistency options prove essential for authentication operations where accuracy is critical.

Session Management and Authentication

Web applications require rapid access to session information and authentication states to validate user requests without introducing significant latency. RAW Hollow’s microsecond read performance makes it ideal for session validation operations that occur on every user request. The system’s write performance supports real-time session updates while maintaining consistency across multiple application instances.

Permission and Authorization Systems

Complex authorization systems require rapid evaluation of user permissions against resource access patterns. RAW Hollow’s ability to store and query large permission datasets locally eliminates the latency traditionally associated with authorization checks. The system’s consistency guarantees ensure that permission changes propagate appropriately across all application instances while maintaining security requirements.

Gaming and Real-Time Applications

Player Profile and Statistics Management

Gaming platforms require immediate access to player profiles, statistics, and achievement data to provide responsive user experiences. RAW Hollow’s co-located architecture enables gaming applications to access comprehensive player data without network delays that could impact gameplay responsiveness. The system’s write performance supports real-time updates to player statistics and achievements while maintaining data consistency across game sessions.

Leaderboards and Competitive Features

Real-time leaderboards and competitive gaming features require immediate updates and queries across large player populations. RAW Hollow’s ability to handle millions of records with microsecond query performance makes it ideal for implementing global leaderboards that update in real-time. The system’s eventual consistency model proves sufficient for most gaming applications where slight delays in leaderboard updates do not impact competitive fairness.

Match History and Analytics

Gaming platforms collect vast amounts of match data and player analytics that need to be accessible for both real-time and historical analysis. RAW Hollow can store comprehensive match history data while providing rapid access for generating player statistics, identifying trends, and supporting competitive analysis. The system’s compression capabilities enable storage of detailed match data without excessive memory requirements.

Financial Services and Trading

Market Data and Portfolio Management

Financial trading applications require microsecond access to market data, portfolio positions, and risk calculations. RAW Hollow’s strong consistency options prove essential for financial applications where data accuracy directly impacts trading decisions and regulatory compliance. The system’s local read capabilities eliminate network latency that could impact high-frequency trading algorithms or real-time risk management systems.

Fraud Detection and Risk Assessment

Financial institutions require real-time analysis of transaction patterns to identify fraudulent activity and assess risk. RAW Hollow’s ability to store comprehensive customer profiles and transaction history locally enables fraud detection algorithms to evaluate complex patterns without network delays that could allow fraudulent transactions to complete. The system’s write performance supports real-time updates to risk profiles and fraud indicators.

Compliance and Regulatory Reporting

Financial institutions must maintain comprehensive audit trails and support regulatory reporting requirements that demand immediate access to transaction history and customer data. RAW Hollow’s durability guarantees and change tracking capabilities support compliance requirements while providing the performance necessary for real-time regulatory reporting and audit trail generation.

E-commerce and Catalog Management

Product Catalog and Inventory Systems

E-commerce platforms require immediate access to product catalogs, inventory levels, and pricing information to support customer browsing and purchasing decisions. RAW Hollow’s ability to handle large product catalogs with real-time updates makes it ideal for e-commerce applications that need to reflect inventory changes and pricing updates immediately across all customer touchpoints.

Personalization and Recommendation Engines

E-commerce personalization requires rapid access to customer purchase history, browsing patterns, and product relationships to generate relevant recommendations. RAW Hollow’s local read capabilities enable personalization engines to evaluate complex algorithms across customer data without introducing latency that could impact conversion rates. The system’s write performance supports real-time updates to customer behavior profiles and recommendation models.

Search and Discovery Systems

E-commerce search functionality requires immediate access to product metadata, customer reviews, and availability information. RAW Hollow can store comprehensive search indexes and product attributes locally, enabling search algorithms to deliver results with minimal latency while supporting complex filtering and sorting operations across large product catalogs.

Implementation Considerations and Best Practices

Dataset Size and Modeling

RAW Hollow performs optimally with small to medium datasets that can fit comfortably in memory across application instances. Organizations should carefully evaluate their data requirements and access patterns to determine whether RAW Hollow’s co-located architecture provides benefits over traditional distributed database approaches. The system’s compression capabilities extend the practical size limits, but applications dealing with truly “big data” may require alternative solutions.

Consistency Requirements Analysis

Applications should carefully analyze their consistency requirements to optimize the balance between performance and correctness. Operations that require immediate consistency should use RAW Hollow’s strong consistency mode sparingly, as frequent strong consistency requests can impact overall system performance. Most applications can benefit from the eventual consistency model for the majority of operations while reserving strong consistency for critical updates.

Operational Complexity and Monitoring

While RAW Hollow simplifies many aspects of distributed data management, it introduces its own operational complexity through the coordination of multiple instance roles and external dependencies. Organizations should invest in comprehensive monitoring and alerting systems to track system health across writers, logkeepers, producers, and clients. The system’s built-in operational tools provide valuable insights, but production deployments require additional monitoring infrastructure.

Future Directions and Evolution

RAW Hollow represents a significant evolution in distributed data storage technology, particularly for applications that require high performance with moderate data sizes. The system’s success at Netflix demonstrates the viability of co-located, in-memory architectures for production workloads at massive scale. As memory costs continue to decline and application performance requirements increase, RAW Hollow’s approach may become increasingly attractive for a broader range of applications.

The system’s flexible consistency model provides a template for future distributed systems that need to balance CAP theorem trade-offs dynamically based on application requirements. This approach represents a mature understanding of distributed systems challenges and offers a practical solution for applications that cannot accept the rigid consistency models of traditional database systems.

Conclusion

RAW Hollow represents a fundamental advancement in distributed data storage technology, demonstrating how careful architectural decisions can deliver exceptional performance while maintaining the reliability and consistency guarantees required for production systems. Netflix’s success in deploying RAW Hollow across hundreds of production services validates the approach for real-world applications that require high performance, high availability, and flexible consistency guarantees.

The system’s unique combination of co-located data storage, tunable consistency, and comprehensive operational tooling makes it particularly suitable for modern applications that serve global user bases while managing moderate-sized datasets. As demonstrated through Netflix’s production deployments in critical infrastructure like Open Connect, identity management, and content management systems, RAW Hollow proves that in-memory, co-located architectures can deliver both the performance characteristics of local data access and the reliability guarantees of distributed systems.

For organizations evaluating distributed data storage solutions, RAW Hollow provides a compelling alternative to traditional approaches, particularly for applications where read performance is critical and dataset sizes remain within the bounds of modern memory capacities. The system’s success demonstrates that the future of distributed data storage may lie not in purely remote architectures, but in hybrid approaches that leverage local data access while maintaining distributed consistency and durability guarantees.