But they all share one trait: they require massive, dedicated infrastructure. They need a centralized “Metadata Service” (a Brain) to map data blocks to hard drives.

Delta Lake asks a different question. What if we don’t build the storage layer at all? What if we just use “dumb” object storage like S3 and manage the intelligence in a simple transaction log?

Here is how it works.

The Problem: The “Cloud Physics” Gap

For years, data engineers had to choose between two bad options.

Option A: The Data Warehouse (like Snowflake or Redshift). It’s consistent and reliable (ACID). If you write data, it’s there. But it’s expensive and proprietary. You can’t just open the files with another tool.

Option B: The Data Lake (S3 or Azure Blob). It’s cheap and open. You just dump Parquet files into a bucket. But it’s unsafe. S3 is “eventually consistent.” If a job crashes while writing, you get corrupt “ghost files.” If you read while writing, you might see partial data.

Delta Lake was born to fix this. It brings ACID guarantees to object storage without building a new database engine.

The Architecture: The “Database” is just a Folder

The core idea is simple: decouple the Physical State (what files are in the bucket) from the Logical State (what files actually belong to the table).

This happens in the Transaction Log.

1. The Transaction Log (_delta_log)

Instead of asking S3 to list thousands of files (which is slow), Delta Lake checks a folder named _delta_log.

• The “WAL” (JSON Files): Every change is a JSON file. 000001.json might say: Add “file_A.parquet”, Remove “file_B.parquet”.

• The “Checkpoint” (Parquet Files): To keep the log from getting too big, Delta compacts these JSON files into a Parquet checkpoint every few commits.

Why JSON? It’s human-readable. You can literally open the file and see what happened.

2. Access Protocols (ACID on S3)

This is where the physics gets tricky.

• Reading: When you query a table, the reader checks the log first. It finds the valid list of files and then asks S3 for them. The log is the single source of truth.

• Writing: Writers upload data to S3 first (invisible to readers). Then, they try to create the next log entry (e.g., 000011.json).

On Google Cloud or Azure, this is easy because the storage supports atomic operations. On AWS S3, it’s harder. S3 lacks a “put-if-absent” feature, so Delta needs a small external helper (like DynamoDB) to make sure two writers don’t create the same log entry at the same time.

High-Level “Superpowers”

Because the log is immutable (we never change past entries), we get features that usually require an enterprise database.

• Time Travel: Since we don’t delete data immediately (we just mark it as “removed” in the log), you can query the table as it existed yesterday.

• Unified Batch & Streaming: A streaming job can just watch the transaction log. It acts like a Kafka topic. New file added? Process it.

• Schema Enforcement: Delta checks data types before writing. It acts like a bouncer, keeping bad data out.

Performance: Making Files Fast

How does a file-based system compete with a specialized engine?

• Compaction: It solves the “Small File Problem.” A background process (OPTIMIZE) grabs thousands of tiny files and rewrites them into larger, efficient chunks.

• Z-Ordering: This organizes data inside the files. If you filter by CustomerID, Delta physically groups those customers together. This lets the engine skip over 90% of the data it doesn’t need.

The Architectural Debate: Files vs. Services

This brings me back to the papers I’ve been reading. The trade-offs here are fascinating.

Delta Lake vs. Facebook Tectonic

In my post about Tectonic, I noted that it manages the physical placement of blocks on hard drives. It’s a filesystem. Delta Lake is a Table Layer. It doesn’t care about blocks or hard drives. It assumes S3 handles the hardware. It pushes all the complexity to the client (Spark), keeping the storage layer simple.

Delta Lake vs. Snowflake

This is the classic “Dashboard Latency” trade-off.

Snowflake uses an always-on database (FoundationDB) to track metadata.

• Pro: It’s fast. You can look up a single row in milliseconds.

• Con: It costs money to keep that infrastructure running.

Delta Lake keeps metadata in files.

• Pro: Infinite scale. Zero “always-on” cost.

• Con: The “Startup Tax.” To read the table, you have to parse the JSON log. That takes 200-700ms.

This explains why Delta is amazing for scanning billions of rows but bad for powering a snappy user dashboard.

Summary

Delta Lake represents a shift from “Smart Storage” to “Open Storage.” By implementing a transaction log over standard files, it proves you don’t need a proprietary engine to get reliability.

However, physics still applies. In the Spanner paper, Google used atomic clocks to guarantee consistency across the globe. Delta uses a JSON log and optimistic logic because it runs on commodity cloud storage.

If you need sub-second lookups, use a service like Snowflake. But for massive data processing, the “dumb” file approach is surprisingly smart.

In mid-September 2025, a state-sponsored group launched a sophisticated cyber-espionage campaign designated GTG-1002. What made this one different wasn’t just the scale (targeting 30 global entities) but the method. The attackers didn’t just use AI to help them; they turned an AI model, Anthropic’s Claude Code, into an autonomous agent that did 80-90% of the tactical work.

The AI autonomously mapped networks, tested for vulnerabilities, harvested credentials, moved through the system, and even analyzed the stolen data for intelligence value. It did this at a speed “physically impossible” for human operators.

But here’s the most important part: the AI didn’t “go rogue.” It didn’t become self-aware or malicious. It was conned.

This event isn’t a story about a rogue AI. It’s a story about a classic, very human security vulnerability known as the “Confused Deputy”. And it exposes a deep, systemic flaw in how we’re building the infrastructure to connect AI to the real world.

The AI as a “Confused Deputy”

The “Confused Deputy” is a long-standing problem in computer security. It’s a situation where a program with legitimate authority is tricked by a malicious entity into misusing that authority.

Imagine you hire a new personal assistant who is brilliant, incredibly fast, and extremely literal. You give them a master key to your office building. One day, a person pretending to be a building inspector tells your assistant, “We have a report of a security flaw in the executive office. I need you to use your master key, go in, and test the safe’s lock for me.” Your assistant, lacking the “gut feeling” or context to be suspicious, sees only a person with an (apparent) legitimate goal. They think they’re helping. So they use their authority (the master key) to fulfill the request, letting the “inspector” (a thief) into the room.

This is exactly what happened in the GTG-1002 campaign. The attackers “socially engineered” the AI. They used a “role-play” tactic, successfully convincing the AI model that it was an employee at a cybersecurity firm and that all its tasks were part of an authorized, defensive penetration test.

The AI wasn’t the attacker. It was the “confused deputy,” the first victim. The attackers tricked it into diligently misusing its own powerful reasoning to achieve their malicious goals.

The “Insecure-by-Default” Protocol

So, how did the AI get the master key in the first place? This is where the problem gets systemic. The attackers connected the AI to their hacking tools using the Model Context Protocol (MCP). MCP was created by Anthropic to be a “universal, open standard” for AI - think of it as a “USB-C port for AI”. Its goal was connectivity and interoperability, not security.

The protocol’s design is “insecure by default”. It promotes an easy-to-implement but highly insecure pattern called “Agent-Auth”. This is the digital equivalent of giving your assistant (the AI agent) its own powerful, static credentials - the master key. This creates a second, more dangerous confused deputy.

1. First, the human confuses the AI (the “role-play”).

2. Second, the (now-confused) AI confuses the tool.

When the AI, fully believing it’s doing a “pen-test,” sends a command like “Run NetworkScan,” the tool (which has its own powerful permissions) faithfully executes it. The tool has no way to check the original human’s true malicious intent.

We built a system for convenience and forgot to build in the most basic safeguards.

Why This Is Just the Beginning

This problem isn’t a simple “bug” that can be patched. Anthropic banned the attackers’ accounts, but the techniques are now public. The attackers’ methods were brilliant in their simplicity.

• “Salami-Slicing”: They bypassed AI safety models by breaking their malicious plan into thousands of tiny, individually benign slices. A single request like “Scan this IP” looks harmless and consistent with the “pen-tester” role. The safety models, which check one prompt at a time (stateless), were blind to the malicious pattern emerging over time.

• “Tool Shadowing”: They exploited the fact that an AI often only sees a tool’s description, not its code. An attacker can create a malicious hacking tool and name it “Cats Counter,” with the description “Counts all the cats in a given domain”. When the AI is asked to “count the cats,” it harmlessly executes the tool, which then does its real, malicious work in the “shadow”.

The barrier to entry for sophisticated cyberattacks has now substantially dropped. This “commoditization of sophistication” means small-time actors can now achieve the results of an entire team of experienced hackers. This will happen again.

The Path Forward: A Mandate for “User-Auth”

If we can’t solve this at the model layer alone, what do we do? The GTG-1002 report shows that we need a “defense-in-depth” strategy.

1. At the Model Level - Stateful Safety: Safety systems must evolve. We need “stateful” safety that looks at context and patterns, not just single prompts. Instead of just asking “Is this prompt bad?”, the system should ask, “Why is this ‘pen-tester’ making 5,000 requests a second at 3 AM?”. This is what Anthropic is now working to build.

2. At the Protocol Level - Mandate “User-Auth”: This is the most critical fix. We must abandon the “Agent-Auth” (master key) pattern. The secure alternative is “User-Auth”. In this model, the AI agent never gets its own keys. Instead, it temporarily borrows the user’s keys. The AI agent’s permissions are identical to the human user who prompted it, and it can never have more authority than that person. If the human user “Anuvrat” doesn’t have permission to access the finance database, the AI he is using can’t access it either. This breaks the confused deputy problem at its root.

3. At the Enterprise Level - Zero Trust: As leaders and architects, we must treat all these new AI tools as untrusted. We need to enforce the “principle of least privilege,” audit our new AI-driven supply chain , and, for any high-risk action (like running an exploit or exfiltrating data), always require an explicit, auditable “Human-in-the-Loop” confirmation.

The GTG-1002 campaign wasn’t a “Terminator” moment. It was a failure of our own design. We’ve built an incredibly powerful engine, and now we have to do the hard work of building a safe, secure, and trustworthy architecture around it.

This decision changes how the browser performs, handles security, and how it will be maintained. It’s a classic study in software trade-offs. Let’s look at what they built, why, and the brittle foundation it rests on.

Part 1: The “Local Browser-as-a-Service”

The Atlas browser isn’t one application. It’s two.

1. The Native UI Shell (Client): This is what you see. It’s the window, the tabs, and the AI controls, built mostly in Apple’s native SwiftUI. This part is light and responsive.

2. The Chromium Service (Server): This is the entire Chromium engine (Blink, V8) running as a separate, headless service in the background.

The two processes are held together by Mojo, Chromium’s internal communication framework. OpenAI’s engineers used this instead of a public API like Chrome DevTools. This setup basically creates a “Local Browser-as-a-Service.” It mixes the security ideas of a remote browser (running in the cloud) with the speed of a local app.

Part 2: The “Why”: The Clear Benefits

Why go through this trouble? The team says this design provides several benefits, and they seem right.

• UI Development Speed: This is the main one. UI engineers can work in a clean Swift codebase. They don’t need to compile all of Chromium locally, which is a famously slow process. This lets them build and test new UI features much faster.

• Crash Isolation: This is a big win for users. If the heavy Chromium service crashes, the native UI shell doesn’t. The app stays responsive and can just restart the service, like restarting a server.

• Faster Perceived Startup: The app feels like it starts instantly. The lightweight native shell loads right away. The user sees a working app while the much heavier Chromium service boots up in the background.

Part 3: The Trade-offs

These benefits are real, but they have a cost. This design introduces new risks in performance and stability.

Trade-off 1: The IPC Performance Tax

When you split a program in two, simple function calls become cross-process network requests. This adds a small delay, or “latency tax.” Atlas pays this tax on every single click, keypress, and rendered frame.

• Input: Your click goes from the Swift UI (Process 1) over the Mojo bridge to the Chromium service (Process 2).

• Output: Chromium renders a new frame, then tells the Swift UI over the bridge that the frame is ready to be displayed.

This only works because Mojo is very fast. But the team still sacrificed raw, in-process speed for this architectural split.

Trade-off 2: The “Clean Codebase” Illusion

The idea of a “clean codebase” is mostly an aspiration. A browser has thousands of small UI parts, like <select> dropdowns or permission prompts, that aren’t part of the web content.

Rebuilding all of these in SwiftUI would take forever. So, the team admits they took a shortcut. When a site asks for your location, the Chromium service (Process 2) actually renders the permission prompt using its own C++ UI framework. This C++ element is then projected into the “all SwiftUI” app (Process 1).

So, you’re often looking at a piece of C++ UI rendered by a hidden process. It’s a smart, practical solution, but it’s not a clean separation.

Trade-off 3: The CALayerHost Gamble

This is the riskiest part of the design. The tool that projects web content and C++ UI from the service into the native app is a Core Animation class called CALayerHost.

CALayerHost is a private, undocumented Apple API.

On macOS, one process isn’t allowed to just draw in another’s window. CALayerHost is the only real way to make this cross-process projection work smoothly. This means the browser’s entire rendering pipeline depends on an unsupported API that Apple could change or remove in any macOS update, which would instantly break the browser.

This risk is shared. Other browsers, like Chrome and Firefox, also supposedly use these private APIs for performance. The Atlas team is betting that Apple won’t break all the major browsers at once. It’s a huge dependency on Apple’s goodwill.

Part 4: Is It Really Decoupled?

This brings up the main engineering question: Is managing this Mojo API bridge really easier than managing a traditional Chromium fork?

OpenAI claims “fewer merge conflicts.” This is probably true for the code. But they haven’t gotten rid of the monolith. They just traded a build-time monolith for a runtime API monolith. Their app is now tightly coupled to the Mojo API. This isn’t a stable, public API. It’s the internal, undocumented plumbing of Chromium, and it changes all the time. This creates a new, scary maintenance burden.

The Chromium team is slowly refactoring its own engine to be more modular (a project called “servicification”). The Atlas team is basically jumping ahead, treating the engine as a service before it was designed to be one. This means they are building on a moving target. As one example, a Chromium developer noted that a recent internal Mojo change required manually refactoring “about 5,000 call sites” across the codebase.

When OpenAI pulls the next security patch from Chromium, a similar internal change could break their C++ service layer, forcing a huge refactor. They’ve traded the predictable work of merge conflicts for the hidden risk of tracking internal, breaking API changes.

Part 5: The Big Win: Security & Privacy

Despite the engineering risks, this architecture is a clear win for security and privacy.

The “Double Sandbox” Security Model: The design creates a strong “double-wall” for security. In a normal browser, an attacker who escapes the renderer’s sandbox just needs to find one more bug to compromise the main browser process.

In Atlas, that entire “normal browser” is already inside the isolated Chromium service (Process 2). An attacker who escapes the renderer sandbox is still trapped inside the service. They would need a third vulnerability to attack the Mojo bridge and compromise the main Atlas UI (Process 1). This makes a successful attack much harder.

Choosing Mojo over the Chrome DevTools Protocol (CDP) was also a smart security move. CDP is a huge, public API with a large attack surface. OpenAI swapped a public building with hundreds of doors (CDP) for a custom vault with one tiny, private slot (their Mojo bridge). It’s a much smaller and more manageable security boundary.

Agent Privacy: Finally, the design for AI agent browsing is excellent. When an AI agent browses for you, Atlas doesn’t just open an Incognito window. Instead, it spins up a new, unique, in-memory storage partition for every single agent session. This is the same tech used to isolate web apps.

When the session ends, all cookies, cache, and site data are completely discarded. This allows for multiple, separate agent sessions to run at the same time with no risk of data leaking between them. It’s a great example of privacy-by-design.

The Verdict: A Brave, Brittle Future

The Atlas browser is an interesting software architecture. It’s a bold idea built on a foundation of high-stakes bets. The team has traded:

• Platform stability (by using Apple’s private CALayerHost API).

• Raw performance (by paying the IPC tax on every interaction).

• API stability (by coupling to Chromium’s internal Mojo APIs).

In exchange for:

• UI developer speed (by using a Swift-native UI team).

• Better crash isolation (by protecting the UI from engine crashes).

• A-class security and privacy (via the double-sandbox and ephemeral storage).

This trade-off makes sense for a company focused on AI and user experience, not on maintaining a browser engine.

This trade-off worked great for something like web search. But what happens when you need that scale, but you also need transactions?

That’s the problem Spanner was built to solve. Google’s ads team was stuck. They were running on a manually sharded MySQL database, and their last resharding effort took over two years. They needed to scale, but Bigtable wasn’t an option. You can’t tell an advertiser you might have over-billed them because of eventual consistency.

Spanner was designed to bridge this gap. It was Google’s attempt to build a single, global database that was both scalable and strongly consistent. They pulled it off not with a single magic bullet, but with a series of brilliant, pragmatic trade-offs. The foundation for all of them was a new way to think about time.

The Foundation: A Clock Built on Uncertainty

The big problem with distributed transactions is ordering. You can’t trust server clocks. Spanner’s solution was to build a clock that was provably accurate, called TrueTime.

TrueTime is a new kind of API. Instead of returning a single number, TT.now() returns an interval: [earliest, latest]. That interval is a guarantee: the “true” absolute time is somewhere in that tiny window (less than 10ms). The designers realized you don’t need to know the exact time, you just need to know the bounds of your uncertainty. This guarantee is the bedrock for everything that follows.

Spanner’s Core Trade-offs

Spanner’s genius lies in the bargains it makes. It willingly accepts a small, calculated cost in one area to gain a huge advantage in another.

Trade-off #1: Trading Milliseconds of Latency for Global Consistency

How do you use a “fuzzy” clock to guarantee transaction order? By making a trade: you wait. This is the “Commit Wait” rule.

When a transaction commits, Spanner’s leader assigns it a timestamp and then forces itself to pause. It holds all the transaction’s locks and waits until it knows the absolute time has passed that timestamp.

That’s the deal. Spanner trades a few milliseconds of latency on every single write. In return, it gets a mathematical guarantee of external consistency. It’s the price it pays to ensure that if transaction T1 commits before T2 starts, T1’s timestamp is provably smaller than T2’s, across the entire globe.

Trade-off #2: Trading Heavyweight Writes for Lightweight Reads

The “commit wait” is expensive, but it unlocks a massive payoff by enabling Spanner to operate at two different speeds. This is the second trade-off: making writes slower to make reads much, much faster.

• The Heavyweights: Read-Write Transactions. When you need to change data, Spanner uses traditional two-phase locking and pays the “Commit Wait” cost. This is the slow, safe, and correct path.

• The Lightweights: Snapshot Reads. Because Spanner is a multiversion database (like Bigtable), it can offer lock-free reads. A read-only transaction gets a fixed timestamp and simply reads the version of data from that moment in the past. It doesn’t need locks, so it’s blazing fast.

This is how Spanner supports high-throughput applications. It concentrates the cost of consistency on the writes, allowing the vast majority of read traffic to fly.

Trade-off #3: Trading a Pure Relational Model for Physical Locality

Even with TrueTime, a transaction that spans datacenters is slow. The fastest distributed transaction is one that isn’t distributed at all. This led to the final, pragmatic trade-off.

Spanner’s schema has a feature called INTERLEAVE IN PARENT. This is a directive from the developer to tell Spanner to physically store a child row (like an Album) with its parent row (like a User).

This isn’t a “pure” relational model, where data location is abstract. It’s a deliberate choice to trade that purity for performance. By co-locating related data, the most common transactions (updating a single user’s data) become fast, single-site operations that don’t need a slow, global two-phase commit. It’s the same practical spirit as Bigtable’s single-row transactions.

So, Was It Worth It?

The F1 ad team’s story says it all. After moving from a manually sharded MySQL database to Spanner, their operations became massively simpler. Spanner’s automatic sharding and failover worked so well that when datacenters failed, the event was “nearly invisible” to them.

Spanner completes the story that GFS and Bigtable started. It’s the final piece of the puzzle, built on a series of smart bargains. It proves you can have both scale and consistency, all for the price of a few atomic clocks and a few milliseconds of waiting.

The answer, I found, is Bigtable.

Reading the Bigtable paper was a fascinating exercise in systems analysis. I expected to learn about a database; instead, I found a case study in pragmatic, purpose-built design. Bigtable’s power isn’t just in its feature set, but in the deliberate, sometimes stark, trade-offs made to solve a specific set of problems at an unimaginable scale.

The First Question: How Should We Organize a Petabyte?

Before writing a single line of code, the Bigtable team faced a foundational choice: what should the data model look like? The familiar path would have been a traditional relational model, enforced by a rigid schema and queried with SQL. It’s the bedrock of the database world for a reason - it’s powerful, consistent, and well-understood.

But they chose a different path. They rejected the familiar comfort of SQL for a far simpler, more flexible abstraction: a sparse, distributed, multi-dimensional sorted map.

This was their first major trade-off. By sacrificing the declarative power of SQL, they gained extreme flexibility. Applications could now add new columns on the fly without complex schema migrations, a massive win for the rapidly evolving products at Google. It also meant they could sidestep the immense complexity of building a distributed query optimizer that could handle expensive operations like JOINs. The cost? Convenience. The job of “query optimization” was pushed from the database engine to the application developer. It was a significant burden, but a necessary one to achieve the primary goal of performance at scale.

The Performance Question: How Do We Make Reads Fast?

Having chosen a sorted map, the next critical question was how to sort it. In a distributed system, you typically want to spread the load evenly. Systems like Tectonic use hashing to scatter data uniformly across all servers, which is perfect for avoiding hotspots.

Here, Bigtable made a defining and counter-intuitive trade-off. Instead of hashing, they chose to keep all data sorted lexicographically by its row key.

This decision gives developers a powerful, if double-edged, tool: control over data locality. By carefully designing row keys (like reversing domain names to group all pages from a single site together), a developer could turn a series of slow, random disk reads into a single, blazing-fast sequential scan. This is arguably the most important feature of the entire system. But the trade-off was significant. It gave up the guarantee of a uniform load, introducing the risk of “hotspotting,” where a single server could be overwhelmed by requests for a popular key range. It was a clear trade of automated safety for developer-controlled performance.

The Scale Question: How Do We Avoid a Bottleneck?

With a petabyte-scale sorted map, how does a client find a specific row without a central directory becoming a massive bottleneck? The GFS model, where a client asks a single master for data locations, wouldn’t work for the thousands of small, low-latency queries Bigtable needed to serve.

The team’s answer was another trade-off: they chose a decentralized, client-driven lookup hierarchy over the simplicity of a central master.

A Bigtable client finds its data by traversing a three-level, B+ tree like structure. It starts with a pointer in the Chubby lock service, which leads to a METADATA tablet, which in turn points to the user data tablet. The client then caches this location. The master is completely out of the data path. This makes the system massively scalable and resilient to master failures, but at the cost of a more complex client library that now had to handle this navigation and caching logic itself.

The Consistency Question: How “Correct” Does It Really Need to Be?

Here again, the design opts for pragmatism over theoretical purity. The final set of decisions centered on consistency and storage.

First, they traded full ACID transactions for single-row atomicity. After studying their applications, they realized most didn’t need the ability to atomically update multiple rows. This single compromise allowed them to sidestep the immense complexity of distributed transaction protocols, resulting in a system with higher performance and availability. The classic banking transaction became impossible, but for Google’s workloads, it was a price worth paying.

Second, they traded in-place file updates for immutable SSTables. Aligning perfectly with GFS’s append-only philosophy, Bigtable never modifies a data file. All new writes go to an in-memory memtable, which is periodically flushed to a new, immutable SSTable file on disk. This radically simplifies concurrency - reads never block writes. The cost was a new background process called compaction, a constant, complex janitorial task required to merge SSTables and garbage collect deleted data.

Conclusion

Analyzing Bigtable is like studying the blueprints for a Formula 1 car. It’s not a general-purpose vehicle; it’s a highly specialized machine built to do one thing - manage structured data at extreme scale, exceptionally well. It achieves this by prioritizing its goals and deliberately omitting features considered standard in other systems.

This post explores how Tectonic’s major design choices illustrate the trade-offs that large-scale systems must navigate, and what they teach us about designing for scale.

Why Tectonic Matters

Tectonic is part of the continuing evolution of distributed storage systems. It doesn’t directly descend from Google’s File System (GFS), but it extends the same set of design questions: how can we balance simplicity, performance, and scalability when we can’t maximize all three at once?

In my earlier post on GFS, I described it as a freight train - steady, predictable, and optimized for bulk throughput. GFS worked beautifully for workloads dominated by large, sequential files. Facebook, however, faced a very different environment: billions of tiny objects and multiple tenants with conflicting requirements. Blob Storage demanded low latency for user-facing operations, while the Data Warehouse required high throughput for analytical batch jobs. Tectonic needed to support both on the same shared infrastructure.

Facebook’s engineers built something that resembled a city rather than a single-purpose factory—many subsystems coexisting, each tuned for a particular workload, all coordinated through a shared foundation.

The Inefficiency of Disaggregation

Before Tectonic, Facebook’s storage ecosystem was split across several specialized systems:

• Haystack managed new (“hot”) photos and videos, using replication for fast reads and writes.

• f4 stored older (“cold”) media using Reed–Solomon (RS) encoding to save space.

• HDFS, conceptually similar to GFS, powered the company’s analytics workloads.

Each system worked well in isolation but not in combination. Haystack was IOPS-bound, leaving unused storage capacity. f4 was capacity-bound, leaving IOPS stranded. HDFS couldn’t share resources with either. The result was a sea of stranded resources - hardware trapped by specialization.

Tectonic’s purpose was to unify these workloads under a single platform, allowing multiple tenants to efficiently share the same hardware. To achieve that, Facebook had to completely rethink both metadata management and how clients interacted with storage.

Reinterpreting GFS Principles

n GFS, a single centralized NameNode held all filesystem metadata in memory - a simple design that offered low-latency lookups for millions of files. But when managing trillions of small objects, that model simply breaks. No single machine could store or serve that much metadata without becoming a bottleneck.

Tectonic solved this by distributing metadata across a sharded key-value store. This horizontal architecture introduced additional network hops and latency but delivered what GFS could not: near-infinite scalability and resilience across fault domains.

Key takeaway: At massive scale, systems trade a bit of local speed for global scalability and fault tolerance.

The Core Trade-offs in Tectonic

Let’s explore the main trade-offs that define Tectonic and what they reveal about design at scale.

Metadata Latency vs. Scalability

In HDFS, metadata operations were instantaneous because everything lived in memory on a single node. Tectonic’s distributed design added network overhead but removed single-node limits. Instead of fighting that latency, engineers leaned into concurrency: clients issued many metadata requests simultaneously, achieving higher total throughput.

Lesson: For batch-oriented workloads like Tectonic’s Data Warehouse, scalability comes from maximizing throughput rather than minimizing per-operation latency.

Simplicity vs. Flexibility (Client-Side Logic)

Tectonic moved much of its decision-making into client libraries. Each tenant could choose how to interact with storage based on its workload:

• Blob Storage wrote data via replication for quick availability, later re-encoding it with RS for efficiency.

• Data Warehouse wrote directly in RS format, optimizing for throughput and space savings.

This approach increased client complexity and duplicated logic, but it enabled a unified storage substrate to serve radically different performance needs.

Lesson: Flexibility often requires decentralizing control, even if it introduces local complexity.

Availability vs. Cost Efficiency

Most of Tectonic’s data is RS-encoded. When a disk fails, reconstruction requires reading fragments from many disks. Too many concurrent reconstructions can overwhelm the cluster. Tectonic prevents this by limiting reconstruction traffic to roughly 10% of all reads. If that threshold is exceeded, new reconstruction requests pause until the system stabilizes.

This safeguard occasionally reduces availability but prevents cascading failures and eliminates the need for expensive over-provisioning.

Lesson: Controlled degradation is often more sustainable than over-provisioning for extreme cases.

Replication Followed by Re-encoding

When new data arrives, Tectonic writes it through fast replication for speed and reliability, then later re-encodes it into RS blocks for storage efficiency. This two-phase process consumes extra space temporarily but improves write latency and simplifies recovery.

Lesson: Temporary inefficiency can be a powerful tool for achieving long-term performance and maintainability.

Closing Reflections: Designing for Balance

If GFS was about control, Tectonic was about coordination. GFS optimized for one workload type. Tectonic tackled the harder problem - allowing diverse workloads to coexist efficiently. Across its design, Tectonic embodies one consistent philosophy: accept bounded inefficiency to achieve adaptability and reliability at scale. It doesn’t optimize for one metric, it balances them all.

The larger lesson is that distributed system design evolves through constraint, not convenience. The most durable architectures embrace imperfection as a design tool - using small inefficiencies to unlock scalability, resilience, and sustainability.

At large scale, perfection gives way to balance. The art of system design lies in knowing what to sacrifice, and doing so deliberately.

Reliability vs. Hardware Cost

In the early 2000s, Google’s data was growing rapidly, but enterprise storage hardware was expensive and limited in scale. Traditional systems achieved reliability through RAID, redundant controllers, and high-end disks, but they were designed to scale vertically - by buying bigger and more powerful machines. That approach hit a wall: each new machine was more expensive and carried more risk if it failed.

GFS flipped that idea. It used commodity hardware, accepting that machines would fail often. Files were broken into 64 MB chunks, stored across multiple chunkservers, and replicated three times, usually across different racks. The master server tracked replicas and automatically rebuilt lost data. Failures were common, but recovery was continuous.

By accepting failure as normal, GFS made reliability a software problem. Durability came from replication, not expensive hardware. Like a suspension bridge, it stayed strong by distributing tension rather than relying on one unbreakable beam.

Latency vs Throughput

Conventional file systems prioritize low latency because they serve humans. When you open or save a file, every millisecond counts. GFS was designed for machines, not people. Google’s crawlers, indexers, and MapReduce jobs processed terabytes of data, and what mattered most was throughput - how much data could move efficiently over time.

To achieve this, GFS used two key design choices:

1. Large 64 MB chunks: Reduced metadata lookups and disk seeks, improving sequential read and write speeds.

2. Append-based writes: Encouraged continuous, large writes instead of small, random updates.

The system excelled at high-bandwidth streaming workloads but struggled with small, random I/O. It traded quick responses for massive sustained data flow. GFS was more like a freight train - slow to start, but once in motion, it could carry enormous loads.

Simplicity vs Availability

Distributed systems often avoid single points of failure, but GFS intentionally included one: a single master server. The master handled metadata such as file names, chunk locations, and replica management. This made the architecture simpler and let the master make global decisions about placement and replication.

The downside was that if the master failed, metadata operations paused. GFS minimized this risk by keeping the master’s state in memory for speed and logging updates to a replicated operation log. It could recover within seconds, and clients could still read and write using cached metadata during downtime.

This was a deliberate compromise - accepting rare short pauses to keep the rest of the system simple and efficient.

Consistency vs Developer Burden

Most file systems enforce strict consistency so multiple users can modify files safely. But Google’s workloads didn’t need that. They were append-only and batch-oriented, like crawlers writing logs or analytics pipelines writing results.

GFS introduced atomic record append, allowing multiple clients to add data to the same file safely, without coordination. Each append was guaranteed to happen atomically and at least once, though duplicates were possible. Partial writes were not.

This relaxed consistency model simplified both the filesystem and developer experience. For most of Google’s workloads, a few duplicate records didn’t matter; losing data or blocking progress did. GFS made the right trade: practical consistency instead of theoretical precision.

Correctness vs Progress

Even with replication, disks could silently corrupt data. GFS protected itself with checksums for every 64 KB block, verifying data on every read. If corruption was detected, it restored the chunk from a healthy replica. The overhead was minimal and the gain in reliability was immense.

GFS assumed hardware would sometimes lie, so it built a system that could detect and fix those lies automatically.

The Meta Trade-Off: Generality vs Focus

GFS wasn’t meant to be a general-purpose file system. It focused on large, sequential, append-heavy workloads - the kind Google relied on most. By narrowing its scope, it avoided unnecessary complexity and achieved predictability at scale.

Its power came not from doing everything, but from doing one thing extraordinarily well.

They edited his DNA. They shipped a bug fix to his genome.

And it worked. This wasn’t some off-the-shelf therapy. The fix was handcrafted, tested, and deployed just for KJ. A one-time, one-person gene edit. The tool they used was CRISPR. I had heard the word thrown around in science headlines, but I had no idea what it actually meant — or why it’s such a big deal. So I did what I usually do when something technical intimidates me — I opened a new note and started writing it down like a software system I wanted to understand.

DNA Is the Codebase

Let’s start with the basics. Every cell in your body (minus a few exceptions like red blood cells) carries a full copy of your DNA — your personal codebase. It’s a 3.2 billion character–long sequence, written in a four-letter alphabet: A, T, C, and G. This code doesn’t run all at once. Cells interpret just the parts they need, kind of like microservices choosing which modules to load from a shared monorepo. Some lines of code are configuration — they turn things on and off. Some are executable — they get translated into proteins, which do the actual work.

The Bug in KJ’s Code

In KJ’s case, one of those code lines was wrong. A single typo. A gene responsible for a crucial enzyme in the liver had a defect. You can think of it like a function that always throws an exception, but in the runtime of the human body, that crash leads to metabolic failure.

In a traditional system, we’d hotfix the code and redeploy. But humans don’t come with continuous deployment pipelines. Or at least, they didn’t.

CRISPR: The Find-and-Replace Engine for DNA

CRISPR is a technology borrowed from nature and repurposed for editing genetic code. The core idea is shockingly simple: use a search string to find the exact spot in the DNA where a bug lives, then use a “molecular scalpel” to cut it, and finally — optionally — paste in a corrected version.

It works in three steps:

1. gRNA (guide RNA): This is your grep query. A ~20-character sequence that tells the system exactly what to look for in the genome. In KJ’s case, it was a needle-in-a-haystack match that located the defective gene in the liver’s DNA.

2. Cas9 enzyme: This is the cutter. Once the gRNA finds its match, Cas9 makes a clean double-stranded break in the DNA at that location. Think of it as opening the source file at the exact line with the bug and placing your cursor there, ready to edit.

3. Repair Template (optional): After the cut, the system needs to patch the break. Sometimes it just slaps the ends back together (a rough fix, useful for turning things off). But if you supply a repair template — basically a diff file — the cell can apply a precise fix.

CRISPR lets us write a targeted, declarative patch and ship it at the molecular level.

How Did They Patch Only KJ’s Liver?

Here’s the part that blew my mind. Remember how every cell has the same codebase? So wouldn’t CRISPR, once introduced, start editing every matching file?

Not if you package it cleverly.

The doctors used lipid nanoparticles — tiny fat bubbles that ferry molecular payloads into cells — tuned to be picked up primarily by liver cells. This is like deploying your fix only to a specific region of your production cluster. Even though the DNA is identical across the body, only the liver machines ran the hotfix script.

Designing for Safety: Avoiding the Wrong Files

CRISPR’s power comes with risk. What if your grep is too vague and hits multiple files? What if you accidentally cut the wrong line? This is called an off-target effect, and it can be disastrous — imagine silently corrupting a critical system file. To avoid this, scientists rigorously test gRNA sequences using both software simulations and lab assays. They look for sequences that are unique enough to only match the intended location, and they test for false positives before ever deploying to a human.

The goal is precision: one match, one cut, one fix.

A Future of One-Line Fixes

KJ’s story is the first time I’ve seen personalized medicine go from concept to commit. The doctors wrote a patch that corrected a specific mutation, in a specific patient, at a specific organ, with a single deployment. It was a one-off fix. But it’s a hint of what might become routine.

Imagine catching a predisposition for cancer and shipping a preventative edit to your genome. Or reversing the early onset of blindness. Or one day — and this is still sci-fi — patching the accumulated bugs of aging.

It’s not there yet. The challenges of delivery, safety, ethics, and cost are enormous. But the idea that we can now treat the body like a running system, and DNA like mutable code — that changes everything.

Alright, so you're deploying apps on Kubernetes. You probably know the drill: write some YAML defining Deployments, Services, maybe StatefulSets, run kubectl apply, and let Kubernetes work its magic. It's built around this core idea of a declarative state. You tell K8s what you want – "three replicas of my API server, please" – and its built-in controllers act like diligent robots, constantly working to make reality match your YAML. Pod crashes? The ReplicaSet controller replaces it. Simple, powerful automation.

When Standard Resources Aren't Enough

But let's be real. Not all applications fit neatly into the Deployment or StatefulSet box. Ever found yourself wrestling with apps that need:

• Complex Day-1 Setup: Maybe initializing a database schema, registering with a central service, or configuring intricate network policies before the app can even start?

• Weird Lifecycle Needs: Think application-specific backup/restore logic, fancy coordinated upgrades that need more finesse than a rolling update, or dynamic scaling based on custom metrics?

• Specialized Operational Knowledge: Tasks that usually live in runbooks or require a seasoned SRE to perform manually during outages or maintenance?

Trying to automate this kind of stuff often leads to a mess of external scripts, manual interventions, or overly complex Helm charts that feel like fragile hacks. We're basically back to imperative management, not the declarative dream Kubernetes promised. How do we bundle this operational "know-how" with the application and teach Kubernetes to handle it natively?

Enter the Operator Pattern

What if I told you there's a way to extend Kubernetes, to teach it new tricks specific to your application? That's precisely what the Operator pattern lets you do.

Think of it like this: you take all the procedures and logic a skilled human operator would use to manage your specific application, and you encode that logic into a piece of software – an Operator – that runs inside your Kubernetes cluster.

The magic happens by combining two things:

1. Custom Resource Definitions (CRDs): You define your own Kubernetes object tailored to your app. Forget juggling separate Deployments, Services, and ConfigMaps; manage your app as a single Kind: AwesomeDatabase or Kind: MyManagedWebApp. It's like adding custom components to your Kubernetes toolkit.

2. Custom Controller (The Operator itself): This is the brain you build. It's a controller specifically designed to understand your CRD. When it sees a user create an AwesomeDatabase resource, it knows exactly what low-level Kubernetes objects (StatefulSets, Services, backup CronJobs, etc.) are needed to make that database a reality.

Your CRD (The "What") + Your Controller (The "How") = Your Operator

The beauty? Your users (or customers) get a dead-simple API (your CRD) to manage a complex application. They declare their desired state using Kind: AwesomeDatabase, and your Operator handles the messy details behind the scenes, making your app a first-class citizen of the Kubernetes ecosystem.

How the Magic Works

Alright, let's peel back the layers a bit:

1. CRDs — Defining Your Language: As mentioned, CRDs let you extend the Kubernetes API. You define the apiVersion (like mydatabases.mycompany.com/v1alpha1), the kind (AwesomeDatabase), and most importantly, the spec fields users can configure (version, storageSize, highAvailability, etc.) using an OpenAPI schema. It’s the blueprint for your custom resource.

2. CRs — The User's Request: Once the CRD exists, a user can create a Custom Resource (CR) – an instance of your AwesomeDatabase. This YAML object holds their specific desired state: "I want version 14.2, 200Gi of storage, and HA enabled."

# User creates this simple(r) object:
apiVersion: mydatabases.mycompany.com/v1alpha1
kind: AwesomeDatabase
metadata:
  name: customers-db
spec:
  version: "14.2"
  storageSize: "200Gi"
  highAvailability: true
  backupFrequency: "daily"

1. The Controller — Your Automated Expert: The Operator (your custom controller) is running, constantly watching for AwesomeDatabase resources. It's programmed with the expert knowledge: "Aha, customers-db wants version 14.2 and HA! That means I need to create a StatefulSet with 2 replicas configured this specific way, a Service for discovery, maybe set up replication…"

2. The Reconciliation Loop — Constant Adjustment: The controller doesn't just act once. It lives in a continuous reconciliation loop:

   • Observe: Sees the customers-db CR, or detects changes to it, or notices that a resource it manages (like the StatefulSet) has deviated from the desired state.

   • Analyze: Compares the desired state (from the CR's spec) with the actual state (what currently exists in the cluster for customers-db).

   • Act: Makes calls to the Kubernetes API (create StatefulSet, update Service, delete Pod, configure backup job, etc.) to close the gap between actual and desired.

This loop ensures Kubernetes is continually working to enforce the state declared in your custom AwesomeDatabase resource, automating away the complex, application-specific operational burden.

By building an Operator, you're not just deploying an application; you're deploying an automated SRE for that application right into Kubernetes itself. Pretty cool, huh?

Python packaging has a nifty feature to include packages as an optional dependency for extended functionality. As an example, Amazon SageMaker released a new capability to use partner AI apps within SageMaker. We worked with the partner app teams to include sagemaker as an optional dependency of their app SDK. Within AWS, you can enable the partner app and install the SDK to include SageMaker specific functionalities.

pip install partner_app_sdk[sagemaker]

The optional dependency is declared by defining an extras_require section in the setup.py or pyproject.toml file.

Problem

An engineer on my team was testing a new version of a partner’s SDK. When they ran the command to upgrade the dependency, they encountered the following message:

> pip install partner_app_sdk[sagemaker] -U

Requirement already satisfied: partner_app_sdk[sagemaker] in /opt/conda/lib/python3.11/site-packages (0.1.5)
Collecting partner_app_sdk[sagemaker]
  Using cached partner_app_sdk-0.2.2-py3-none-any.whl.metadata (514 bytes)
WARNING: partner_app_sdk 0.2.2 does not provide the extra 'sagemaker'

Despite the new version 0.2.2 being downloaded, pip warned that the sagemaker extra was not provided. This was puzzling since the sagemaker extra was expected to be part of the package.

The Exploration

To debug what was happening, we started by reviewing the setup.py files for both the versions. The extras_required section correctly listed sagemaker as its dependency. So, why was pip unable to find sagemaker in the new version? We explored the following options:

1. Could the pip cache be corrupted? This was easy to rule out. We cleared the cache and re-ran the update command. Didn’t resolve the issue.

2. Could it be a problem with the wheel .whl file?

Deep Dive into Wheel Files

A wheel (.whl) is a binary distribution format for Python packages. It contains the package and its metadata, including the information about optional dependencies.

Inspecting the Wheel Files

To check the .whl files, we downloaded it manually and examined the metadata inside the wheel.

> pip download partner_app_sdk==0.1.5 --no-deps
> unzip -p partner_app_sdk-0.1.5-py3-none-any.whl | grep -A 5 "Provides-Extra"

Provides-Extra: langchain
Requires-Dist: langchain; extra == "langchain"
Provides-Extra: sagemaker
Requires-Dist: sagemaker; extra == "sagemaker"

> pip download partner_app_sdk==0.2.2 --no-deps
> unzip -p partner_app_sdk-0.2.2-py3-none-any.whl | grep -A 5 "Provides-Extra"

Provides-Extra: langchain
Requires-Dist: langchain; extra == "langchain"

This revealed that sagemaker was indeed missing from the .whl metadata, despite being included in the setup.py file! This explains why pip wasn’t able to find sagemaker and issued the warning.

Root Cause?

We are yet to understand the root cause here. Most likely, the build process caused the metadata to be incomplete or incorrect in the wheel file. But now that we know this can happen, we will implement tests to validate the partner_app_sdk.

Writing data into partitions is straightforward. You have two options:

• Static Partitioning: You provide the list of partitions you want to write the data into.

• Dynamic Partitioning: You provide a column whose values become the values of the partitions. In this case, Hive creates as many partitions as unique values of the column provided.

Here’s a code snippet that writes into static and dynamic partitions:

DROP TABLE IF EXISTS stats;

CREATE EXTERNAL TABLE stats (
    ad              STRING,
    impressions     INT,
    clicks          INT
) PARTITIONED BY (country STRING, year INT, month INT, day INT)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n';

MSCK REPAIR TABLE stats;

-- Specify static partitions
INSERT OVERWRITE TABLE stats
PARTITION(country = 'US', year = 2017, month = 3, day = 1)
SELECT ad, SUM(impressions), SUM(clicks)
FROM impression_logs
WHERE log_day = 1
GROUP BY ad;

-- Load data into partitions dynamically
SET hive.exec.dynamic.partition = true;
SET hive.exec.dynamic.partition.mode = nonstrict;

INSERT OVERWRITE TABLE stats
PARTITION(country = 'US', year = 2017, month = 3, day)
SELECT ad, SUM(impressions), SUM(clicks), log_day
FROM impression_logs
GROUP BY ad;

Did you notice the difference between the two? In the second INSERT query, the value for day hasn’t been specified in the PARTITIONS construct. Instead, the value for the day partition column comes from log_day column of the impression_logs tables. While the first query only writes into 1 partition, the second query can write into as many partitions as the days in the impression_logs table.

The OVERWRITE keyword tells Hive to delete the contents of the partitions into which data is being inserted. This is the key! Hive only deletes data for the partitions it is going to write into. All other partitions remain intact.

This feature of Hive allows us to develop applications using AWS EMR + S3 and Hive to write partitioned outputs and run backfills/updates on selective partitions. We create ephemeral EMR clusters and load the partitions needed by a job in each run. Partitioning the data by date also allows us to run multiple parallel jobs (in separate EMR clusters), processing data for different dates at the same time. Hive optimizes the disk reads to only load the partitions requested by a query.

Problem Statement

And this is where Spark differs massively in implementation! When a Spark programmer talks about partitions, they always mean the partitions into which a Dataset is divided across the cluster. They talk about re-partitioning the data or to coalesce the data before writing to disk.

One of the biggest problems I faced when working on a new project with Spark was the organization of the output data into buckets (Hive partitions) such that an individual (or a collective group of these buckets) may be overwritten by a batch Spark job. The challenge was that, even though Spark provides an API to write into a format similar to Hive partitions, it either OVERWRITEs all partitions or appends to the partitions. Spark doesn't natively support the same behavior as Hive. In the OVERWRITE mode, Spark deletes all the partitions, even the ones it would not have written into.

Here’s the Spark code that writes the data into partitions:

impressionsDF
    .write.mode("overwrite")
    .partitionBy("country", "year", "month", "day")
    .json("s3://output_bucket/stats")

If we run the job to backfill the data for just 1 day, then Spark will delete the data for all other days if we were to run the above code.

Solution

After searching around a lot for solutions and trying out various options, I finally settled on using Hive context to write the partitioned data. With Spark-2.1.0 the Spark community has fixed a few bugs related to dynamic partition inserts:

• SPARK-18183 — INSERT OVERWRITE TABLE ... PARTITION will overwrite the entire Datasource table instead of just the specified partition.

• SPARK-18185 — Should fix INSERT OVERWRITE TABLE of Datasource tables with dynamic partitions

So, if you are using Spark 2.1.0 and want to write into partitions dynamically without deleting the others, you can implement the below solution.

The idea is to register the dataset as a table and then use spark.sql() to run the INSERT query.

// Create SparkSession with Hive dynamic partitioning enabled
val spark: SparkSession =
    SparkSession
        .builder()
        .appName("StatsAnalyzer")
        .enableHiveSupport()
        .config("hive.exec.dynamic.partition", "true")
        .config("hive.exec.dynamic.partition.mode", "nonstrict")
        .getOrCreate()

// Register the dataframe as a Hive table
impressionsDF.createOrReplaceTempView("impressions_dataframe")

// Create the output Hive table
spark.sql(
    s"""
      |CREATE EXTERNAL TABLE stats (
      |   ad            STRING,
      |   impressions   INT,
      |   clicks        INT
      |) PARTITIONED BY (country STRING, year INT, month INT, day INT)
      |ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n'
    """.stripMargin
)

// Write the data into disk as Hive partitions
spark.sql(
    s"""
      |INSERT OVERWRITE TABLE stats
      |PARTITION(country = 'US', year = 2017, month = 3, day)
      |SELECT ad, SUM(impressions), SUM(clicks), day
      |FROM impressions_dataframe
      |GROUP BY ad
    """.stripMargin
)

Spark now writes data partitioned just as Hive would — which means only the partitions that are touched by the INSERT query get overwritten, and the others are not touched.

I hope Spark adds this functionality natively, but until then, this is the best solution I have.

{
  "customer": {
    "given_name": "Anuvrat",
    "surname": "Singh"
  },
  "order": {
    "id": "123dfe523gd"
  }
}

Now, using the Hive-JSON-Serde you can parse the above JSON record as:

create table order_raw(
  customer map<string, string>,
  order map<string, string>
) row format serde 'org.openx.data.jsonserde.JsonSerDe'
location '<location-to-file>';

select customer['given_name'], customer['surname'], order['id']
from order_raw;

This is really great! I can now parse more complicated JSON without writing any UDF. The fun doesn’t stop here. We can now define JSON that have nested arrays and maps and parse them using the lateral view along with the explode() UDF provided by hive. Let’s take another example to demonstrate the power of Hive-JSON-Serde.

Say you have a startup in the transportation domain, like Uber (which is awesome btw!). You have the travel details of customers in JSON format, and you want to analyze it. Each JSON record contains the customerId, age, and a list of trips taken by the customer. Each trip has distanceTravelled, fare and referral. Each trip might have 1 or more referral codes assigned to it.

You get the picture? Cool. Let’s see a sample JSON.

{
  "customerId": "0277ZGAX80PG6ZSJ04J5",
  "age": 23,
  "services": [
    {
      "trips": [
        {
          "tripId": "A12-5678344-4097746",
          "fare": 24.0,
          "distanceTravelled": 3.2,
          "referrals": {
            "email_campaign": {
              "campaignA": {
                "referralIds": ["0ZK7V4HM5ZZNKJ0PRRR5"]
              }
            }
          }
        }
      ]
    }
  ]
}

Okay, so the JSON looks a little complicated. But that was my aim. Let’s say the schema for your startup was designed in a way to allow you to add different services like cargo transport, etc. Trips are the taxi component. And you might want to track and attribute each ride to email campaigns, special offers, etc.

Now let’s create a table for this JSON using the Hive-JSON-Serde.

create table cust_trips (
    customerId  string,
    age int,
    services array<struct<
        trips:array<struct<
            tripId:string,
            fare:double,
            distanceTravelled:double,
            referrals:struct<
                email_campaign:map<string, struct<referralIds:array<string>>>
            >
        >>
    >>
) row format serde 'org.openx.data.jsonserde.JsonSerDe'
location '<location-of-files>';

Awesome!! Designing the table was the hardest part, where all the magic happens. If done properly, extracting data is a smooth sailing. Let’s get the count of customers referred by each referralId. For this, we need to write a query where we group all the records by referralId and take the count of distinct customerIds. Remember that referralId was nested so deep in the JSON, perhaps within multiple arrays.

We will use the explode() UDF to explode an array into different rows. For example, a single record A, [1, 2] in a hive table can be exploded into 2 records — A, 1 and A, 2. Sweet, isn’t it? I’d suggest you read it up if you don’t already know about it.

Here’s the query that answers the question posed above.

select
    campName,
    refIdArr.referralIds[0],
    count(distinct customerId)
from cust_trips ct
    lateral view explode(ct.services) v1 as s
    lateral view explode(s.trips) v2 as t
    lateral view explode(t.referrals.email_campaign) v2 as campName, refIdArr
group by
    campName,
    refIdArr.referralIds[0];

Note how simple and easy the query is!

PS: I hope I haven’t made any error writing it. Since I cannot share the original query I wrote for work, I had to build an example along a similar line and copy the query with variable names changed.

JSON is widely used to store and transfer data. Hive comes with a built-in json_tuple() function that can extract values for multiple keys at once. But if you have a nested JSON, the query using json_tuple() can get messy quickly. Say we have the following JSON:

{
  "customer": {
    "given_name": "Anuvrat",
    "surname": "Singh"
  },
  "order": {
    "id": "123dfe523gd"
  }
}

If we were to use the json_tuple() function of Hive, we would write something like:

create table json_table (
  record string
);

select
  v2.given_name,
  v2.surname,
  v3.id
from json_table jt
  lateral view json_tuple(jt.record, 'customer', 'order') v1 as customer, order
  lateral view json_tuple(v1.customer, 'given_name', 'surname') v2 as given_name, surname
  lateral view json_tuple(v1.order, 'id') v3 as id;

Which, you have got to agree, looks hideous!

Instead, we could quickly write a script (python being my fav.) that does the transformation for us. The input to the script is a single record of json from the table, and the output of the script should be tab separated values. The values can be operated upon or inserted into another table using hive.

Here’s the script. The script calls transform_json() method for each line, which extracts the values we are interested in and prints them with tab as the separation character.

#!/usr/bin/python
import sys
import json

def get_order_details(order_record):
  json_record = json.loads(order_record)

  customer = json_record['customer']
  order = json_record['order']

  values = [customer['given_name'], customer['surname'], order['id']]

  print '\t'.join(map(str, values))

for line in sys.stdin:
    line = line.strip()
    get_order_details(line)

In the hive script, we first add the file and then call it using the transform() command. Here’s a neat looking script that does what our previous hive script did, using json_tuple().

add file 'get_order_details_mapper.py';

select transform ( record )
using 'python get_order_details_mapper.py'
as given_name, surname, order_id
from json_table;

That’s it! Neat and simple.

In my particular case the keys were variables and I just could not have used json_tuple() to extract info from json.