Google Open Source Blog

This Week in Open Source for October 2, 2026

2026-10-02T11:30:00.000-07:00

by Daryl Ducharme, Open Source Programs Office

This Week in Open Source for October 2, 2026

A look around the world of open source

As AI becomes part of everyday software development, the most compelling questions in open source are shifting from the models themselves to everything surrounding them. I’m always drawn to how technology affects individuals, and especially the workers who use and maintain it, for better or for worse (I definitely prefer better). That’s why I’m convinced the next phase of open-source AI won’t be won by chasing bigger benchmarks, but by building the open infrastructure, transparent local tooling, and community norms that keep developers and maintainers in control. This week’s reads explore what that looks like in practice.

Upcoming Events

🗓️ October 2026

ValkeyConf 2026 (October 5, 2026) — Prague, Czechia. Dedicated community conference advancing the open source Valkey in-memory data store, featuring a keynote by Valkey TSC member and Google Cloud engineer Jacob Murphy on open source in-memory data structures and scaling high-performance workloads.
MCP Dev Summit Toronto (October 5–6, 2026) — Toronto, Ontario, Canada. Dedicated Linux Foundation developer summit advancing the Model Context Protocol (MCP) and open agentic interoperability standards. Join Google OSPO's Daryl Ducharme (that's me!) on Tuesday, October 6 (11:00 AM EDT) for "Tag-Team Transmission: Navigating A2A and MCP for Optimum Orchestration," examining how the Agent2Agent (A2A) protocol and MCP interoperate across multi-agent architectures.
Open Source Summit Europe (October 7–9, 2026) — Prague, Czechia. The premier Linux Foundation gathering in Europe celebrating the 35th anniversary of Linux and connecting developers, technologists, and community leaders—including a keynote by Google OSPO's Erin McKean on Docsy and technical documentation for humans and AI agents.
Community Over Code (October 11–14, 2026) — Glasgow, Scotland. The flagship Apache Software Foundation conference bringing together project maintainers, committers, and users to collaborate on open governance, data architecture, and community-led software development.
BazelCon 2026 (October 13–15, 2026) — Amsterdam, Netherlands. Annual gathering of the Bazel community uniting build system engineers, maintainers, and ecosystem contributors around fast, reproducible, multi-language software builds.
All Things Open 2026 (October 19–20, 2026) — Raleigh, North Carolina, USA. One of the largest community-focused open source conferences on the U.S. East Coast exploring open source software, AI engineering, and maintainer sustainability—featuring a Google keynote on Generative UI and the Open Future alongside the Google Community Lounge with the Flutter team.
PyTorch Conference North America 2026 (October 20–21, 2026) — San Jose, California, USA. Two days of open source machine learning innovation covering training, inference, compiler toolchains, and hardware heterogeneity across the PyTorch ecosystem.
AGNTCon + MCPCon North America 2026 (October 22–23, 2026) — San Jose, California, USA. Linux Foundation conference focused on building and scaling production agentic AI systems with open standards, observability, and control, featuring a keynote by Google's Rao Surapaneni.
GitHub Universe 2026 (October 28–29, 2026) — San Francisco, California, USA & Virtual. Annual global developer gathering highlighting open source workflows, collaborative security, and AI-assisted software engineering.

🗓️ November 2026

Open Source in Finance Forum (OSFF) New York (November 4–5, 2026) — New York, New York, USA. Dedicated industry conference examining open source compliance, supply-chain security, and collaborative innovation across regulated financial institutions.
KubeCon + CloudNativeCon North America 2026 (November 9–12, 2026) — Salt Lake City, Utah, USA. The Cloud Native Computing Foundation's flagship conference uniting adopters and maintainers around Kubernetes, platform engineering, and cloud-native infrastructure.
SFSCON (South Tyrol Free Software Conference) 2026 (November 13–14, 2026) — Bolzano, Italy. One of Europe's longest-running Free Software conferences bringing together public-sector decision-makers and developers to advance digital sovereignty and open infrastructure.

Open Source Reads and Links

[Article] Mila and Mozilla announce new initiative to build trustworthy open source AI for everyone, with Canadian government support — As a Canadienthusiast, seeing Mozilla and Mila partner with the Canadian government on an open source AI foundation layer immediately grabbed my attention. The crossover of open source and public-sector sovereign systems seems clear on the surface, yet real-world adoption reveals unexpected decisions. How do we make open source and open models useful to governments—is it in the open source infrastructure around them?
[Article] Google’s open source EnvHarness lets AI agents train against environments that evolve with them — As AI adoption matures, the need for solid infrastructure around AI is becoming more evident. It is nice to see Google Research’s work leading to open source tools like EnvHarness, which adapts training sandboxes to an agent’s failure modes—and raises the question of how our evaluation infrastructure must evolve alongside the agents themselves.
[Blog] Inside LLM Inference: Every Calculation from Text to Token using Gemma 4 12B — LLM inference often feels like an opaque black box, making Amulya Bhatia’s calculation-by-calculation trace of a token moving through Google’s open source Gemma 4 12B fascinating to ponder. Seeing how hybrid attention bounds memory under the hood prompts a broader question: how differently do we design AI systems when we actually understand the math happening inside the model?
[Video] 100% Local RAG Without Internet: Qdrant Edge and Google LiteRT — GDE Tarun R Jain pairs Google LiteRT and Gemma 4 with Qdrant Edge to run retrieval-augmented generation 100% offline. It prompts an interesting question for system design: how differently do teams experiment with their knowledge bases when local prototyping is free—and how much of what we default to the cloud could stay on the edge?
[Post] UISurf: An Operator-Centric Multi-Agent Platform for Observable and Cross-Environment UI Automation — UISurf explores orchestrating UI agents across Web, Desktop, and Mobile boundaries using the open Agent2Agent (A2A) protocol. Beyond the tool itself, its lessons on sandboxing and human-in-the-loop oversight are worth pondering: when autonomous agents coordinate across environments, what level of observability do human operators actually need to stay in control?
[Article] PS5 Linux lead quits as open-source projects have become “a bunch of noobs using LLMs” that “they don’t even understand” — When maintainers leave high-profile projects like PS5 Linux, it is important to look at the reasons. In the age of AI-assisted software development, unvetted LLM code and shifting bounty incentives are straining both maintainers and community trust. What can we learn from these events as we update open source and AI governance best practices to keep projects safe, secure, and viable?

Which of these stories will you be chatting about at your next meetup or conference? Let us know! Share with us on our @GoogleOSS X account or our @opensource.google Bluesky account.

Standardizing fine-grained access control in Apache Iceberg's REST catalog

2026-10-01T11:30:00.000-07:00

by Talat Uyarer & Sung Yun, Google Cloud

The Apache Iceberg community recently merged a change to the REST Catalog specification that gives lakehouses something they have long been missing: a standard, engine-neutral way to express column masking and row filtering. It is a small change, but it addresses a structural gap in how open table formats handle data governance. This post walks through the problem, the design, and why the details matter.

The problem: there is no server in the read path

In a traditional database, access control is straightforward because there is exactly one door. Every query passes through the database server, the server knows who is asking, and if policy says "this user only sees the last four digits of the card number," the server applies that policy before returning results. One process, one enforcement point.

Apache Iceberg deliberately removed that door. The data is Parquet files in object storage, and any engine—such as BigQuery, Spark, Trino, Flink, PyIceberg, or DuckDB—can read those files directly. This design is what makes Iceberg fast and interoperable. But it also means that once an engine asks the catalog "where is the payments table?" and receives the metadata pointer, it has full access to the files. The catalog can grant or deny access to the whole table, but it has had no vocabulary for "access granted, but mask the email column" or "access granted, but only on rows where region = 'US'."

Until now, Iceberg had no built-in answer to this. The REST spec did offer coarser instruments like credential vending, which controls storage access per table, and server-side scan planning, which lets a catalog withhold entire data files. But neither can express "mask this column" or filter rows that are not already physically separated into their own files. So users who needed fine-grained access control had to step outside the open protocol entirely. They could adopt a vendor-provided client that understands its proprietary policy format, or route every read through a vendor's proxy service, giving up the direct-to-storage performance that motivated the use of Iceberg in the first place. It also quietly undermines Iceberg's core promise: the moment governance requires a specific vendor's client, the table is no longer open to any engine.

The new read-restrictions field in the REST Catalog spec is the community standardizing that vocabulary.

The mechanism

When an engine calls loadTable, the response may now include an optional read-restrictions object:

Read restrictions are expressed in two fields:

required-row-filter: a standard Iceberg predicate expression. Rows for which it evaluates to false must not appear in the result, and no information derived from them may be included.
required-column-projections: a list of columns, identified by field ID, each with a transformation the reader must apply before returning values.

One evaluation rule ties them together: the row filter is evaluated against the original, untransformed column values, and projections are applied to the rows that survive. This ordering is what makes the two features composable. A policy can filter on region = 'US' and also mask region in the output, and the filter still works. If masking ran first, any policy that filtered on a masked column would silently break.

Note what the catalog is doing here. It evaluates the access policy server-side—it knows the caller's identity from the authentication token—and returns only the result of that evaluation. The policy itself, with its roles, tags, and governance model, never crosses the wire. The engine does not need to understand how any particular catalog models governance. It needs to understand nine actions and a predicate.

The whole architecture fits in one sequence:

The catalog stays the policy decision point; the trusted engine becomes the policy enforcement point; the end user never holds storage credentials. Sections below unpack the three load-bearing details in this picture: the enforcement ordering, the fail-closed branch, and the trust boundary.

The nine masking actions

The specification defines a closed set of nine masking actions, each with an exact, per-type definition. The goal is cross-engine consistency: Spark, Trino, and PyIceberg must produce identical output for any given masking action. These actions are being implemented in iceberg-core, providing engines with spec-compliant transformations out of the box rather than requiring them to reimplement byte-level logic independently.

Below, the actions are grouped by the analytical utility of the resulting masked data:

Preserve the shape, hide the value

mask-alphanum—digits become n, other characters become x, with a small allowlist of punctuation (( ) , . - @) preserved. iceberg16112018@apache.org becomes xxxxxxxnnnnnnnn@xxxxxx.xxx—recognizably an email address, but not whose.
show-first-4 / show-last-4—preserve four code points, and apply mask-alphanum to the rest. 4111-1111-1111-4444 becomes nnnn-nnnn-nnnn-4444, the familiar customer-support view of a card number.

Hide everything

replace-with-null—the value becomes NULL. Only valid for optional fields; a server must not return it for a required field, and a reader that receives one must fail the query.
mask-to-fixed-value—the value becomes a type-specific constant (0, "XXXXXXXX", the epoch, an all-zero UUID, an empty list, and so on, each spelled out in the spec). Uniquely among the actions, this one also replaces NULL inputs—even the null-or-not bit is hidden.

Reduce precision

truncate-to-year / truncate-to-month—2024-07-15 becomes 2024-01-01 or 2024-07-01. Cohort analytics keeps working; identifying individuals gets harder.

Allow joins, hide values

sha-256-global—deterministic SHA-256, with exact byte-encoding rules per input type. The same input always produces the same output, everywhere, so GROUP BY user_id and joins across tables on a hashed key still work. The cost of that determinism is that hashed values can be tested against precomputed guesses—this is pseudonymization, not encryption.
sha-256-query-local—the same hash, salted with a fresh, cryptographically random salt (at least 16 bytes) per query. Values remain consistent within a single query, so self-joins and aggregation work, but cannot be correlated across queries, and precomputed-guess attacks no longer apply.

This last pair is a nice piece of design: the tradeoff between linkability and privacy, expressed as two enum values that a policy author chooses between per column.

Every action produces a value of the same type as its input—masked strings are strings, truncated dates are dates—so restrictions never change the schema an engine plans against. Queries do not need rewriting; values simply arrive transformed.

Fail-closed by design

The most consequential sentence in the specification is this one:

If a trusted reader that supports read-restrictions cannot apply any returned restriction, it must fail the query and must not silently return raw, partial, or empty results.

Consider the failure modes. A catalog sends an action added in a future spec version that the engine does not recognize. Or an expression type it cannot evaluate. The convenient behavior would be to skip what it does not understand and return the data. The spec rules this out: unrecognized action, fail; unparseable filter, fail; duplicate field ID in the projections, fail. Every ambiguity resolves to "no data" rather than "raw data." This is the right default for an access control mechanism, and it is also the one implementers would be tempted to soften—which is exactly why it is normative in the spec rather than left to judgment. It is also what allows the action vocabulary to grow in future versions without older engines becoming silent leak vectors.

A few prohibitions in the spec reward a closer look, because each one closes a subtle correctness hole:

No projections on map keys. Masking a map's keys can collapse two keys into one, or produce null keys, which engines silently coalesce or reject—data corruption presented as privacy. The spec bans it outright.
No projection on both a nested type and a field inside it. Masking a struct and also a field within that struct has no well-defined order of operations, so the spec refuses to define one: servers must not send it, and readers must reject it.
Everything references field IDs, never column names. This is standard Iceberg discipline: if ssn is renamed to national_id, the policy remains bound to the same physical column. A name-based policy would silently detach on rename—the worst possible failure mode for access control.

The trust model, stated plainly

All of this is enforced by the reader. The catalog hands the engine the file locations along with the restrictions, and a client that chooses to ignore the restrictions can read the raw files. So what is this actually protecting?

The specification is explicit: this mechanism assumes a trust relationship between the catalog and the engine, and how that trust is established is deliberately out of scope. The intended deployment is one where a platform team's engines—the shared Spark and Trino clusters—are trusted enforcement points that hold storage access (for example, through credential vending), while end users only ever talk to those engines and never hold storage credentials themselves. The trusted engine becomes the enforcement point, playing the role the database server played in the traditional architecture. The difference is that its behavior is now defined by a common, open protocol rather than by N proprietary integrations.

In other words, read restrictions do not protect data from the engine; they let the catalog direct a trusted engine to protect data from the engine's users. For genuinely untrusted readers, the coarse-grained model still applies: they are restricted to vended credentials where the storage access granted to the user aligns with the data permission of the user.

One operational subtlety deserves attention: restrictions are per-response and per-identity. The same loadTable call made by a different principal—or by the same principal later—may return different restrictions, and the restrictions attach to every read performed with that response, including subsequent planTableScan and fetchScanTasks calls. The spec therefore requires that the response not be cached outside its authentication scope. If your platform caches loadTableResponse, that cache is now security-sensitive and worth an audit.

What is still missing

Read restrictions are a foundation, not the finished building. It is worth being honest about the gaps between this specification and complete fine-grained access control:

The action vocabulary is fixed, because Iceberg Expressions are not implemented yet. Nine actions cover the common masking policies, but they are a deliberately closed set: a policy like "apply this custom redaction function" cannot be expressed, and the row filter is limited to predicates—comparisons that produce true or false. The path to generalizing this already exists on paper: the Iceberg Expressions specification, proposed by Ryan Blue and adopted in mid-2026, defines a portable structure for value expressions—constants, field references, and calls to well-defined functions or SQL UDFs. Once engines can evaluate those expressions, a catalog could return arbitrary transformations instead of choosing from an enum. Today no engine implements general expression evaluation, which is why the initial design confines itself to a small vocabulary that can be specified bit-for-bit.
There is no policy definition—deliberately. The specification standardizes the result of policy evaluation, never the policy itself. How an organization expresses "analysts see masked PII, auditors see everything"—the roles, tags, rules, and administrative APIs—remains entirely the catalog vendor's domain, and the assumption is that it stays there. Only the consequences of a policy are portable across engines; the policy definition is not. Whether communities eventually want a portable policy format is an open question the spec does not attempt to answer.
Trusted clients are asserted, not proven. As the trust-model section noted, how a catalog establishes that a caller is a trusted, enforcing engine is out of scope. In practice that trust is deployment configuration—service identities, network boundaries, and which principals receive vended credentials. There is no attestation mechanism in the protocol by which an engine proves it enforces restrictions; the trusted-client mechanism is an assumption the platform operator must make true.

None of these gaps undermines the design—each is a deliberate scoping decision that kept the proposal small enough to reach consensus—but they define the roadmap for what "complete" fine-grained access control in the open lakehouse still requires.

Why this matters

The specification change itself does not ship enforcement; that work in the engines begins now, starting with the default actions. But the shape of the design is right in three ways:

It picks the honest enforcement point. In an architecture with no server in the read path, the trusted engine is the only place enforcement can live without giving up direct storage reads. The spec accepts that constraint explicitly rather than obscuring it.
It standardizes the narrow waist. Catalogs keep their own rich policy engines—roles, tags, and attribute-based rules. Engines implement nine actions and a predicate evaluator, once. N×M becomes N+M.
It is fail-closed everywhere, which is what makes the vocabulary safely extensible.

The pattern—the catalog evaluates policy against the caller's identity and returns a small, closed vocabulary of obligations that the client must enforce or fail—is a useful template, and it would not be surprising to see more of the governance surface expressed this way over time.

The proposal was approved on the Apache Iceberg dev list with eight binding +1 votes and no objections. This is a strong signal of consensus across the many companies and open source communities that participate in the project. Consistent, engine-independent enforcement of fine-grained policies is a property the ecosystem has long wanted; it now has a specification for it, and the interesting work of implementing it in engines and catalogs is underway. If you work on either, the dev list is the place to get involved.

The full schema is available in rest-catalog-open-api.yaml under ReadRestrictions. View the full vote thread.

New public datasets available in Google Cloud Lakehouse

2026-09-30T11:30:00.000-07:00

by Daniel Rodrigues & Alex Stephen, Google Cloud

Did you know the Google Chrome Wikipedia page received over 36 million page views in 2025, or that Chromium logged over 3,700 commits in April 2010? If you have needed large, real-world datasets to benchmark query engines on Apache Iceberg, Google Cloud’s Lakehouse team is excited to announce the release of new public datasets in Google Cloud Lakehouse to help you explore and analyze open data at scale.

Google Cloud’s Lakehouse provides a high-performance storage catalog using Apache Iceberg as its open table format. By decoupling storage from compute, it enables you to use your preferred query engines—such as BigQuery, Apache Spark, or Trino—while managing data in an open format to avoid vendor lock-in.

These new public datasets are designed to help you explore the Apache Iceberg ecosystem and begin working immediately with real-world data. We are providing access to some of the most popular BigQuery public datasets, including Wikipedia pageviews and GitHub commit histories. Let’s dive into how you can start querying them.

Prerequisites

A Google Cloud project (for authentication).
Standard Google Application Default Credentials (ADC) configured in your environment.

Explore metadata with PyIceberg

To explore table metadata using PyIceberg, install the required Python packages in a virtual environment:

sudo apt-get install python3-venv
python3 -m venv public_data_lakehouse
source public_data_lakehouse/bin/activate
pip install google-auth
pip install pyiceberg
pip install pyarrow

After installing the required packages, you can inspect a table’s schema using the Python script below. Here is how to inspect the table containing Wikipedia page view data from 2016:

from pyiceberg import catalog as pyiceberg_catalog

PROJECT = "<YOUR_PROJECT_ID>"
NAMESPACE = "wikipedia"
TABLE = "pageviews_2016"

def load_catalog() -> pyiceberg_catalog.Catalog:
  properties = {
      "type": "rest",
      "uri": "https://biglake.googleapis.com/iceberg/v1/restcatalog",
      "warehouse": "gs://lakehouse-public-data",
      "auth": {"type": "google"},
      "header.X-Iceberg-Access-Delegation": "vended-credentials",
      "header.x-goog-user-project": PROJECT,
  }
  return pyiceberg_catalog.load_catalog(PROJECT, **properties)

def main() -> None:
  catalog = load_catalog()
  table = catalog.load_table((NAMESPACE, TABLE))
  print(f"Schema for `{NAMESPACE}.{TABLE}`:", table.schema())

if __name__ == "__main__":
  main()

Note: Replace <YOUR_PROJECT_ID> with your actual Google Cloud Project ID. This is required for the REST catalog to authenticate your quota usage, even for free public access.

PySpark and Managed Service for Apache Spark

Now that we have explored the table’s schema, we can use a managed, serverless Spark notebook to query the tables. This provides the speed and flexibility of Apache Spark without creating or managing a cluster. You can run this script in Google Cloud's serverless Managed Service for Apache Spark (formerly Dataproc):

from google.cloud.dataproc_v1 import Session
from google.cloud.dataproc_spark_connect import DataprocSparkSession

PROJECT_ID = "<YOUR_PROJECT_ID>"
spark_catalog = "lakehouse-public-data"

session = Session()
session.runtime_config.properties = {
  "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions",
  f"spark.sql.catalog.{spark_catalog}": "org.apache.iceberg.spark.SparkCatalog",
  f"spark.sql.catalog.{spark_catalog}.type": "rest",
  f"spark.sql.catalog.{spark_catalog}.uri": "https://biglake.googleapis.com/iceberg/v1/restcatalog",
  f"spark.sql.catalog.{spark_catalog}.rest.auth.type": "org.apache.iceberg.gcp.auth.GoogleAuthManager",
  f"spark.sql.catalog.{spark_catalog}.io-impl": "org.apache.iceberg.gcp.gcs.GCSFileIO",
  f"spark.sql.catalog.{spark_catalog}.header.X-Iceberg-Access-Delegation": "vended-credentials",
  f"spark.sql.catalog.{spark_catalog}.warehouse": "gs://lakehouse-public-data",
  f"spark.sql.catalog.{spark_catalog}.header.x-goog-user-project": PROJECT_ID,
}

spark = (
   DataprocSparkSession.builder
     .appName("Lakehouse Public Data Demo")
     .dataprocSessionConfig(session)
     .getOrCreate()
)
spark.conf.set("spark.sql.defaultCatalog", spark_catalog)

With the session active, you can query monthly Wikipedia view counts for BigQuery-related articles by executing the following code in a new cell:

df1 = spark.sql("""
SELECT
  title,
  wiki,
  SUM(views) AS total_views
FROM wikipedia.pageviews_2026
WHERE datehour >= TIMESTAMP '2026-01-01 00:00:00'
  AND datehour <  TIMESTAMP '2026-02-01 00:00:00'
  AND LOWER(title) LIKE '%bigquery%'
GROUP BY title, wiki
ORDER BY total_views DESC
LIMIT 20
""")
df1.show(10)

Or, you can retrieve recent GitHub commits referencing Iceberg using the following snippet:

df2 = spark.sql("""
SELECT
  commits.commit,
  commits.subject,
  commits.message,
  commits.author.name AS author_name,
  timestamp_seconds(commits.committer.date.seconds) AS commit_time,
  repo_name
FROM github_repos.commits AS commits
WHERE LOWER(commits.subject) LIKE '%iceberg%'
   OR LOWER(commits.message) LIKE '%iceberg%'
ORDER BY commit_time DESC
LIMIT 50
""")
df2.show(10)

Start building today

These datasets were imported from their BigQuery counterparts and transformed using Apache Iceberg as the table format and Parquet as the data file format. Our goal is to lower the entry barrier so you can learn and explore Apache Iceberg with your favorite query engine without managing any infrastructure. To get started with building an open, managed, and high-performance Iceberg lakehouse, visit the Google Cloud Lakehouse page.

This Week in Open Source for September 24, 2026

2026-09-24T11:30:00.000-07:00

by Daryl Ducharme, Open Source Programs Office

This Week in Open Source for September 24, 2026

A look around the world of open source

Open source underpins an $8.8 trillion global economy, yet sustaining it requires moving beyond voluntary charity and philosophical appeals. This week’s reads examine how the ecosystem is adapting to modern economic and AI pressures—from proposing package-registry royalties and quantifying 2–5x net value for regulated utility grids, to turning autonomous AI security incidents into hardened supply-chain defenses that empower open-science breakthroughs like NASA and IBM’s Lunar Foundation Model.

Upcoming Events

🗓️ October 2026

MCP Dev Summit Toronto (October 5–6, 2026) — Toronto, Ontario, Canada. Dedicated Linux Foundation developer summit advancing the Model Context Protocol (MCP) and open agentic interoperability standards. Join Google OSPO's Daryl Ducharme on October 6 for "Tag-Team Transmission: Navigating A2A and MCP for Optimum Orchestration," examining how the Agent2Agent (A2A) protocol and MCP interoperate across multi-agent architectures.
Open Source Summit Europe (October 7–9, 2026) — Prague, Czechia. The premier Linux Foundation gathering in Europe celebrating the 35th anniversary of Linux and connecting developers, technologists, and community leaders across open AI, embedded systems, and digital trust.
Community Over Code (October 11–14, 2026) — Glasgow, Scotland. The flagship Apache Software Foundation conference bringing together project maintainers, committers, and users to collaborate on open governance, data architecture, and community-led software development.
All Things Open 2026 (October 19–20, 2026) — Raleigh, North Carolina, USA. One of the largest community-focused open source conferences on the U.S. East Coast exploring open source software, AI engineering, DevSecOps, and maintainer sustainability.
GitHub Universe 2026 (October 28–29, 2026) — San Francisco, California, USA & Virtual. Annual global developer gathering highlighting open source workflows, collaborative security, and AI-assisted software engineering.

🗓️ November 2026

Open Source in Finance Forum (OSFF) New York (November 4–5, 2026) — New York, New York, USA. Dedicated industry conference examining open source compliance, supply-chain security, and collaborative innovation across regulated financial institutions.
KubeCon + CloudNativeCon North America 2026 (November 9–12, 2026) — Salt Lake City, Utah, USA. The Cloud Native Computing Foundation's flagship conference uniting adopters and maintainers around Kubernetes, platform engineering, and modern cloud infrastructure.
SFSCON (South Tyrol Free Software Conference) 2026 (November 13–14, 2026) — Bolzano, Italy. One of Europe's longest-running Free Software conferences bringing together public-sector decision-makers and developers to advance digital sovereignty and open infrastructure.

Open Source Reads and Links

[Blog] Nobody pays for open source. We can force them to. — Companies already spend over $1 billion a year on open source, but that money goes to supply-chain mirror and security vendors rather than the maintainers writing the code. Laurie Voss breaks down why voluntary charity and license changes always fail, arguing that commercial package registries and mirror vendors should pay automatic royalties down the dependency tree to long-tail maintainers.
[Article] AI Is reshaping open source software and straining the systems that sustain it — As conversations grow around moderating the speed of AI development across the industry, it is critical to watch the friction points where AI and open source overlap—particularly as an influx of AI-generated code and vulnerability discovery strains maintainers of an $8.8 trillion ecosystem. Funding remains front and center, requiring not just raw capital but continuous monitoring for effectiveness in sustaining the human governance, consensus-building, and non-coding infrastructure that AI cannot replace.
[Report] LF Energy Research Finds Open Source Software Can Deliver 2-5x Greater Net Value for Grid Operators — While open source software and public utilities seem like natural philosophical allies as public goods, philosophical appeals routinely fail in regulated infrastructure sectors where operators are bound by strict reliability mandates and ratepayer accountability. This new LF Energy report demonstrates how to bridge that gap: by translating collaborative "make together" governance into an auditable benefit-cost framework showing two to five times greater net value over proprietary procurement.
[Article] AI Is at a Turning Point — High-profile incidents of autonomous AI agents breaching Hugging Face or social-engineering open source maintainers inevitably dominate headlines and amplify purely negative narratives around AI. However, treating these security failures as concrete post-mortems—exposing how standard reinforcement learning incentivizes deceptive subgoals—is exactly what allows open source communities to establish hardened supply-chain defenses and safely advance high-impact, positive AI use cases.
[Article] IBM and NASA Release Open-Source AI Model to Support Lunar Exploration — Seeing open source, AI, open data, and space science converge is genuinely inspiring, especially when it demonstrates that open scientific models are only as useful as the open datasets beneath them. While NASA and JAXA planetary archives have long been public, IBM and NASA's release of the Lunar Foundation Model alongside the first harmonized, machine-learning-ready lunar dataset (unifying 30+ multi-instrument layers) proves that transforming raw open data into shared, interoperable infrastructure is what unlocks domain discovery.

Which of these stories will you be chatting about at your next meetup or conference? Let us know! Share with us on our @GoogleOSS X account or our @opensource.google Bluesky account.

Google Cloud: Investing in the future of PostgreSQL — 2026 highlights

2026-09-23T11:30:00.000-07:00

by Dilip Kumar, Cloud SQL for PostgreSQL

Google Cloud is committed to open source, and PostgreSQL is a cornerstone of managed database offerings, including Cloud SQL and AlloyDB.

Continuing our work with the PostgreSQL communities, we've been contributing to the core engine and participating in the patch review process. Below is a summary of that technical activity between January 2026 and September 2026, highlighting our efforts to enhance the performance, stability, and resilience of the upstream project and ecosystem. By strengthening these core capabilities, we aim to drive innovation that benefits the entire global PostgreSQL ecosystem and its diverse user base.

Our technical contributions in this period have focused on enhancing core engine performance, introducing features for logical replication, fixing critical bugs, and improving upgrade resilience. We also continue to invest in the PostgreSQL ecosystem by addressing bugs in widely used extensions.

Technical contributions: January 2026 – September 2026

Our contributions this cycle span four key areas:

Logical replication and conflict management: Paving the way for active-active replication, enhanced conflict logging, and catalog safety.
Core engine performance and reliability: Eliminating tuple-level overhead in sequential scans and making promotion timeouts accurate.
Core catalog, collation, and indexing bug fixes: Strengthening privilege consistency, deferrable index builds, collation handling, and memory safety.
PostgreSQL extension and ecosystem hardening: Resolving critical crashes, lock tranche registrations, and memory vulnerabilities across popular ecosystem extensions (plpgsql_check, pgfincore, and pgtt).

1. Logical replication and conflict management

Logical replication is essential for near-zero downtime migrations, major version upgrades, and multi-region data distribution. Our recent work focuses on conflict log table infrastructure, namespace clarity, and cross-database catalog hygiene.

Conflict log table infrastructure

Background and challenge: A major milestone on the roadmap to active-active multi-master replication is the ability to automatically record and resolve data discrepancies across nodes. Without structured conflict logs, tracking conflicting writes requires inspecting server logs or relying on custom application handlers.
Solution: This patch introduces the foundational conflict log table management infrastructure as an option in CREATE SUBSCRIPTION. It establishes the catalog structures and configuration hooks necessary to direct conflict records into dedicated, queryable tables, serving as the cornerstone for upcoming automatic conflict logging into tables.
Note: Provided no blocking issues arise, this functionality is slated for inclusion in PG 20.
Contributors: Dilip Kumar (Primary Author)

Fix REASSIGN OWNED for subscriptions in other databases

Background and challenge: While pg_subscription is physically a shared catalog so the background launcher process can scan all databases, subscription objects are logically local to each database. Certain operations, such as REASSIGN OWNED, failed to restrict their catalog scans to the current database (MyDatabaseId), leading to accidental modifications across database boundaries.
Solution: Added explicit guards to pg_subscription readers to ensure non-launcher processes filter strictly by MyDatabaseId, protecting cross-database isolation and updating documentation.
Contributors: Dilip Kumar (Author)

Schema-qualified names in EXCEPT clause error messages

Background and challenge: When publishing tables with EXCEPT clauses, check_publication_add_relation() previously reported only unqualified table names when a relation could not be processed, leading to ambiguous error messages in multi-schema databases.
Solution: Updated error reporting paths to emit fully schema-qualified relation names, aligning with PostgreSQL's broader error messaging standards.
Contributors: Dilip Kumar (Author)

2. Core engine performance and administrative enhancements

Optimizing throughput and improving the predictability of administrative operations remain top priorities for database workloads.

Timeout handling in pg_promote()

Background and challenge: Standby promotion via pg_promote() allows users to specify a timeout interval. Due to imprecise elapsed-time tracking during wait loops, promotion operations could terminate prematurely before the configured timeout elapsed.
Solution: Refined the promotion loop to pre-calculate the expected end timestamp and track actual elapsed time across iterations, ensuring strict adherence to user-configured wait intervals.
Contributors: Robert Pang (Reporter and Author)

Sequential scan performance optimization

Background and challenge: Previously, CheckXidAlive validation was executed inside the inner table_scan_next routines. This incurred repetitive check overhead on every single tuple fetched during sequential scans.
Solution: Restructured the scan control flow to eliminate redundant per-tuple checks, yielding cleaner execution paths and measurable throughput improvements on large table scans.
Contributors: Dilip Kumar (Author)

3. Core engine access control, collation, and indexing bug fixes

We continue to harden PostgreSQL's core engine against catalog inconsistencies, segmentation faults, and edge-case query anomalies.

Large object access with pg_{read,write}_all_data

Problem: The default roles pg_read_all_data and pg_write_all_data were designed to allow maintenance utilities like pg_dump to operate without superuser privileges. However, Large Objects (LOBs) remained inaccessible under these roles without explicit object-level grants.
Fix: Updated permission checks to extend pg_read_all_data and pg_write_all_data coverage to Large Objects, completing superuser-free dump and maintenance workflows.
Contributors: Nitin Motiani (Author), Dilip Kumar (Reviewer)

Immediate property propagation in index copies (REINDEX CONCURRENTLY)

Problem: When building a replacement index during REINDEX CONCURRENTLY for a deferrable unique constraint, index_create_copy() defaulted constraint flags to 0, setting the immediate property to true. This caused concurrent transactions to immediately trigger constraint violations rather than deferring verification until commit time.
Fix: Introduced the INDEX_CREATE_DEFERRABLE flag to properly propagate an immediate property of false to transient copied indexes without violating internal constraint assertions.
Contributors: Nitin Motiani (Author)

LIKE matching with nondeterministic collations and backslashes

Problem: Following the addition of nondeterministic collation support for LIKE, literal pattern substring parsing unconditionally skipped all backslash characters. When encountering escaped backslashes (\\), the engine omitted the second backslash entirely instead of emitting a literal \.
Fix: Corrected pattern de-escaping logic to correctly recognize and emit escaped backslashes during evaluation.
Contributors: Nitin Motiani (Author)

DSM lock release and crash prevention

Problem: If a backend encountered a FATAL exit while holding a lock in a Dynamic Shared Memory (DSM) segment (e.g., inside dynamic shared hashtables dshash) outside of an active transaction, releasing locks during process termination could reference already detached DSM segments, triggering a segmentation fault.
Fix: Hardened cleanup and lock release sequences during fatal exits to safely detach memory segments without segfaulting.
Contributors: Dilip Kumar (Reviewer)

4. PostgreSQL ecosystem and extension hardening

Enterprise PostgreSQL architectures rely heavily on third-party extensions. Our team actively contributes bug fixes and stability improvements upstream to critical ecosystem projects.

plpgsql_check: LWLock tranche registration for PG14

Problem: In PG14 and earlier, loading plpgsql_check via shared_preload_libraries could fail with shared memory lock errors due to missing or outdated named LWLock tranche registrations (plpgsql_check profiler funcs stats and plpgsql_check profiler func stmts stats).
Fix: Aligned pre-PG15 tranche initialization with modern shmem_request_hook patterns, guaranteeing safe shared memory allocation on older server versions.
Contributors: Aniket Jha (Author)

pgfincore: Memory safety hardening

Problem: pgfincore contained two subtle memory corruption issues: an off-by-one array boundary access during buffer inspection and a dangling pointer assignment during deallocation.
Fix: Authored patches to enforce strict boundary checks and clean pointer resets, eliminating potential memory corruption during OS buffer cache analysis.
PRs: klando/pgfincore#12 and klando/pgfincore#13
Contributors: Robert Pang (Author)

pgtt: Use-after-free prevention on cached plans

Problem: When running utility commands via the extended query protocol or within cached contexts (such as PL/pgSQL and SQL functions), pgtt modified cached statement parse trees in-place using short-lived query memory. Once that memory was freed, subsequent executions of the cached plan led to Use-After-Free crashes.
Fix: Updated the extension to operate on an isolated, deep copy of the parse tree for cached and read-only statements, ensuring memory safety across repeated executions.
Contributors: Sunaina Punyani (Author)

Community roadmap: Your feedback matters

We encourage you to utilize the comments area to propose new capabilities or refinements you wish to see in future iterations, and to identify key areas where the PostgreSQL open source communities should focus their investments.

Acknowledgments

We would like to celebrate our engineers for their ongoing dedication to open source:

Dilip Kumar (PostgreSQL Significant Contributor): Authoring and reviewing core replication, catalog, memory, and performance patches.
Nitin Motiani: Authoring core privilege expansions, collation de-escaping, and indexing constraint fixes.
Robert Pang: Authoring promotion timing fixes and hardening pgfincore memory safety.
Aniket Jha: Hardening plpgsql_check shared memory lock mechanics.
Sunaina Punyani: Resolving memory and execution safety in pgtt.

We also extend our sincere gratitude to the wider PostgreSQL open source communities—especially the committers, reviewers, and extension maintainers—for their collaborative reviews and shared commitment to keeping PostgreSQL the world’s most advanced open source database.

Reconnecting with the heart of open source: Highlights from our 2026 GSoC India tour

2026-09-17T11:30:00.000-07:00

by Mary Radomile, Stephanie Taylor & amanda casari, OSPO

For over twenty years, Google Summer of Code (GSoC) has welcomed new developers into open source by pairing them with experienced mentors on real projects. This spirit is especially vibrant in India, which is home to more than 55% of all global GSoC participants over the last decade.

This July, our team traveled across Bengaluru and Delhi to host a series of developer events and debut our first-ever GSoC Alumni CAMP, bringing together members of India’s vibrant GSoC alumni community. We engaged directly with current and former GSoC Contributors, Mentors, and project maintainers, experiencing firsthand the passion and energy of the Indian developer ecosystem.

Community stories: Learning to think and lead as an engineer

In both Bengaluru and Delhi, a highlight of the trip was hearing directly how open source and GSoC have fundamentally changed how developers think and work.

For many attendees, having a dedicated open source mentor through GSoC took the fear out of tackling new and intimidating codebases. One former participant told us they almost walked away from a distributed storage project because it felt too overwhelming: "Storage systems felt impossible. But great mentors taught me how to think, not just how to code." Another echoed that shift in perspective: "Why am I spending so much time thinking rather than coding? Then I realized that building products is actually about thinking more than coding. GSoC taught me to think like an engineer."

That shift in mindset turns first-time contributors into long-term open source community leaders. We met one developer who submitted their very first pull request in 2023, started mentoring in 2024, and is now a lead maintainer for a major open source Android app. We also saw how new contributors to global projects can spark entire local ecosystems—like the Indian compiler community, which started with a few GSoC alumni and has rapidly grown into a 3,500+ member network with dozens of meetups across the country.

The Google Summer of Code Alumni CAMP — Bengaluru

Open source mentorship in the age of AI

Across our sessions and unconference discussions, one recurring conversation resonated above all others: the evolving role of mentorship in an AI-assisted world. The human element of open source is more critical than ever. As one attendee noted:

AI can generate the slides, but the context takes nine years.

We heard over and over from participants—open source maintainer time and attention remains a limited resource. CAMP participants presented multiple examples of where AI is proving effective for generating starter templates, writing tests, or fixing syntax. Even with this, what open source projects fundamentally need hasn't changed: maintainer time, clear architectural vision, and thoughtful code reviews.

Beyond code quality, the industry agrees that dedicated mentorship is the vital bridge between temporary contributions and long-term project stewardship. Without structured guidance, newcomers often struggle with unwritten project norms, complex codebase histories, or public review feedback, leading to contributor burnout and abandoned pull requests. Programs like GSoC transform casual interest into a sustainable maintainer pipeline by fostering psychological safety, belonging, and accountable relationships. By investing directly in maintainer time and human connection, GSoC ensures that open source projects remain secure and resilient for generations to come.

What’s next?

If our trip across Bengaluru and Delhi taught us anything, it’s that the strength of open source has always come from the communities we build together, not the volume of code any one person can ship.

As developer tools evolve, our main focus for GSoC is preserving the mentorship experience that makes the program special. Manually sifting through low-quality, automated submissions wastes maintainer time and drains the energy of volunteers who signed up to mentor new peers and colleagues. We're ready to tackle these challenges directly by optimizing our program to assist and protect our community of open source maintainers, so they can focus on leading their open source projects and helping new engineers grow. You can stay updated on GSoC’s program rules and timelines at g.co/gsoc.

To everyone who joined us in Bengaluru and Delhi—thank you for your energy, your endless inspiration, and your dedication to open source!

How much should you trust your OSS data?

2026-09-03T09:00:00.000-07:00

by Sophia Vargas, Google Open Source & Andrew Nesbitt, Ecosyste.ms

Every second, open source contribution quietly shapes the software we rely on, and yet our view of this open ecosystem is surprisingly opaque. Open source development is performed in public spaces — we can see the commits, issues and comments, the APIs and endpoints are free to use — the logs are just sitting there, so why can’t we just collect all of the data?

…Said every researcher, everywhere. However in most cases of open source related data, we are only looking at part of the whole. Why am I writing this post? Because many of us (including many business decision-makers) are too comfortable with unsubstantiated data. We’ve gotten used to it. Our models assume that it's smelly and we adjust the logic and weights to compromise. When it comes to open source, our confidence is even lower, even though our resulting decisions can directly impact individuals whom we collectively depend on.

Let’s consider one of my favorite datasets: GHarchive. Started as a hobby project in 2011, this crawler has amassed more than 15 years of event data from GitHub. While this source provides a historical record of open source development on GitHub, as a real-time or comprehensive source of metrics, it's unreliable and should not be a source for volume-based metrics.

In 2025, GHarchive captured 14% fewer events than in 2024, despite steady growth in platform adoption. Since 2025, we estimate that data retention in GHarchive has fallen to ~50% and in 2026 it may be as low as 20% for some event types (see figure below). Prior to 2025, you could make the general assumption that the majority of events would be represented in this pipeline. Since 2025, we must now assume we may be missing at least half of events and possibly more — not to mention all of the additional activity that’s left out of the event API (see GitHub’s GraphQL API.)

The crawler logic behind this dataset is simple: give me all the events from the GitHub Event stream (e.g. opening pull requests, commenting on issues etc). However, the GitHub API has limitations on the number of calls per hour as well as the number of events listed, so for days with a lot of spiky activity, the crawler will miss some. Although we never assumed that this dataset was collecting 100% of events, the current architecture is showing signs of strain. We suspect that this is due, in part, to the rate of repository growth and adoption of automated tooling on GitHub. In 2011, GitHub announced it reached 2 million public repositories, and by 2026, that figure surpassed 400 million.

I want to acknowledge that building and sharing comprehensive open datasets at scale is hard. Have you ever built a pipeline only to discover that the variables changed mid year, the payload for one output is getting truncated, all your joins broke because one side of the dataset is case sensitive … I could go on. And these examples are just ordinary data issues. Building a dataset at the scale of GitHub where “Every second, more than one new developer on average joined GitHub—over 36 million in the past year”—you start running into a new set of challenges.

My own journey with open source related data began when I repeatedly found myself questioning how much we could trust our own metrics. To expand my understanding of the nuances and the limitations of open source related datasets, I reached out to Andrew Nesbitt, who has spent years digging in data trenches for the benefit of the community. Together we converged on the following issues that we wanted to highlight for the broader community.

Assembling: Assume there will be problems

When I asked Andrew ‘can you summarize the challenges you have faced assembling comprehensive datasets?’ —“I just assume I'm going to have a terrible time anyway, so I start with my best effort and fill in the gaps”. While disappointing, this aligned with most data aggregation methods I’ve reviewed—tools such as Grimoire labs and OSS insights also require multiple processes for collection, combination and reconciliation. Even with these approaches, many sources have missing, incomplete, or inconsistent information.

One source is probably not enough. If you are considering the use of an open source project, you may want to know how many maintainers work on this project, what versions are available, what their dependencies are and any active vulnerabilities or known issues. Each of these queries requires a distinct source—the development history, the dependency graph, the CVE database, etc. Ecosyste.ms strives to pull this information together into one place, but combining data from 1000+ datasets has its own unique set of challenges.

For example, my index is probably not your index. One perennial issue is inconsistent naming conventions across sources. Beyond variable type and format, repository names, versions, packages, tags, licenses, urls, etc. tend to be unique across platforms. Some are case sensitive, there are often duplicates, and anyone can change a name at any time… I’ve been keenly following the adoption of purl and SWHID, but so far I have not found one name to rule them all.

Now we have to keep this up to date: At the moment, there is no consistent way of sharing updates across platforms. Changes to names, APIs, deletions, etc. are more often discovered by errors and breakage than by scouring release notes. To keep Ecosyste.ms up to date, Andrew has written multiple syncing processes that identify or infer updates that need to be accounted for. I asked Andrew ‘If you could ask a platform/data source to change one thing, what would it be?’, “Can I crawl an endpoint that's just NEW stuff?’

Consuming: Design your pipeline for your use case

Because of LLMs, “it's now easier for anyone to try to access and build reports”. But those building quick reports are likely not going to go through the pain of being comprehensive. This is where aggregated sources like GHarchive and Ecosyste.ms thrive. As data providers, we’d love if data consumers knew that:

How you collect data matters. If everyone wanted the same dataset, in the same format, at the same time, it would be simple. Depending on how the data is stored—centralized vs distributed and cached, relational vs graph, etc. —queries could be more efficient (in cost and computation) than exports or bulk requests faster than individual requests. This all depends on the topology of the infrastructure and the dataset. In a perfect world, data producers would design their architecture for their top user journeys. However open source related datasets serve a wide variety of user personas from corporations to non-profits, researchers to individual users, maintainers, funders, and many more, with a variety of demands from historical deep dives to realtime feedback. Data producers can’t design for all of these cases, so my challenge to them is to be more open about the best way to access this information.

At the end of the day, we have to respect the human infrastructure: Open source-related datasets are riddled with personally identifiable information (PII). Some individuals may be comfortable sharing their information with fellow contributors, but seeing it aggregated across platforms can be uncomfortable. Any source with PII should be handled with care: anonymize when you can and ensure you are in alignment with policies and regulations. Open source communities are real people so please, consume their data responsibly.

Interpreting: Never stop asking questions

While many have moved on from ‘data-driven’ to ‘AI-enabled’, the fact remains that ALL AI SYSTEMS DEPEND ON DATA. Our data about open source will continue to be incomplete and imperfect, but by asking questions about our sources, acknowledging the gaps, and considering both the technical and human processes behind open source development, we can refine and improve on how we interpret our insights and models even if they don’t completely reflect reality.

Securing the agentic era: Introducing formal verification for CEL

2026-08-18T11:30:00.000-07:00

by Sean Huh, Common Expression Language Team

We are rapidly entering an era where AI agents can autonomously draft, refactor, and deploy policies that protect our users and our systems. But this velocity introduces a vital question: How do we trust AI-generated policies?

Unit tests may fail to cover the infinite set of possible inputs that occur in production; thus, an AI agent that overfits its policy to existing tests may fail spectacularly in production. To secure automated policy authoring, we must combine heuristic testing with mathematical proofs.

We are thrilled to announce the Common Expression Language (CEL) Formal Verification Framework is now available. Powered by the Z3 theorem prover, this framework allows you to prove the correctness of your CEL expressions and policies, serving as the ultimate safety net for the agentic policy.

Automated reasoning definitively answers questions like:

“Is there any combination of inputs that allows an unapproved request into production?”
“Are we absolutely certain this AI-refactored policy matches the original behavior?”
“Can a bad actor manipulate this rule to force an evaluation error?”

Formal verification establishes mathematical certainty across the infinite spectrum of inputs. Proven policies protect your users and system while giving auditors clear proof of compliance.

To see these capabilities in action, watch our video demonstrating how the CEL Verifier REPL catches subtle logic flaws in seconds:

Proving rules from the ground up

Getting started with formal verification doesn’t require learning complex architectures right away. You can evaluate simple standalone CEL expressions to catch edge cases that tests easily miss.

(Note: The examples below use our interactive REPL syntax—check out the REPL documentation to follow along!)

1. Catching logic bugs in simple expressions (Equivalence)

How do you guarantee a refactored rule behaves identically to the original? Suppose we have a policy that allows ports 80 or 443 in production. An agent might factor the is_prod check like so:

equiv
  (is_prod && port == 80) || (is_prod && port == 443) 
  <=>
  is_prod && port == 80 || port == 443

Because logical AND has a higher operator precedence than OR, the verifier immediately flags Violated, and outputs the exact exploit: in a non-production environment (is_prod = false), the rule mistakenly allows port 443. Fixing the grouping parentheses returns Verified.

2. Enforcing exhaustive guardrails (Validity)

This capability scales directly to use cases like Kubernetes Validating Admission Policies. Suppose an engineer writes a guardrail expression that assumes every request will either be on a low port (under 80) or a high port (over 1024):

valid request.port > 1024 || request.port <= 80

When we check validity (whether an expression holds true for all inputs), the verifier exhaustively searches the entire integer space, flags Violated, and outputs the exact counterexample:

[VIOLATED] Condition is not always true. Counterexample input:
  request.port = 81

3. Guaranteeing security invariants with CEL Policy

While the verifier works perfectly with standalone CEL expressions, complex environments compose multiple rules and variables. Here, the CEL policy format shines. Using assume and assert blocks, the verifier proves a mathematical implication: if the assumptions hold, the assertions must also hold.

name: workload_admission
rule:
  variables:
    - is_admin: 'request.auth.claims.groups.exists(g, g == "admin")'
  match:
    # A subtle flaw introduced during authoring:
    - condition: 'request.is_privileged && request.is_prod'
      output: 'true'
    - condition: 'variables.is_admin || request.has_approval'
      output: 'true'
    - output: 'false'
verification:
  invariants:
    - id: universal_no_unapproved_privileged_prod
      assume:
        - 'request.has_approval == false'
        - 'variables.is_admin == false'
      assert:
        - 'rule.result == false'

The first condition admits privileged workloads into production without checking for approval or admin status. The verifier flags this and provides an example that exploits the issue:

Invariant 'universal_no_unapproved_privileged_prod' violation detected. Counterexample input:
  request.is_privileged = true
  request.is_prod = true
  request.has_approval = false
  request.auth.claims.groups = []

Assertions and assumptions define the boundaries of acceptable agent behavior, allowing developers to configure CI/CD pipelines to validate AI-generated changes simply and securely.

Under the hood: High-fidelity mathematical modeling

Translating a dynamic language into the Satisfiability Modulo Theories (SMT) domain requires immense engineering rigor to prevent the solver from hanging or hallucinating bugs. Our engine provides:

Zero false positives via three-pass taint tracking

Traditional verification tools are prone to “solver hallucinations”—reporting fake bugs when encountering custom domain-specific functions or external variables they don’t fully understand. To eliminate this noise, if a potential issue relies on an unmapped custom function, the verifier isolates and flags it as Inconclusive rather than breaking your CI pipeline with a false alarm. This guarantees every Violation report is a 100% real, reproducible bug.

Deep structural extensionality

The Formal Verification Framework offers configurable-depth bounded-model checking to prevent infinite loops within SMT quantifiers. These configurable limits allow you to control the cost of verification when analyzing deep structure equivalence in expressions like [[1], [2]] == [[1], [2]].

The mandatory bridge of trust

In the agentic era, code writes code. Mathematical proof isn’t just a nice-to-have; it is the fundamental bridge of trust developers require to let AI operate autonomously in their most sensitive systems. Get started with the CEL Formal Verification Framework, to take the next step toward a more secure agentic future today!

Let us know what you think—issues, pull requests, and feedback are always welcome!

Google joins the OpenROAD Initiative as principal member to accelerate open source silicon innovation

2026-08-11T11:30:00.000-07:00

by Ethan Mahintorabi & Aaron Cunningham, Hardware Toolchains Team

Google is committed to advancing open source silicon innovation. We are excited to share that we have formally joined the OpenROAD Initiative (ORI), Inc. as a principal member. ORI is a nonprofit public benefit corporation dedicated to the open source electronic design automation (EDA) ecosystem. As part of this commitment, Aaron Cunningham has been appointed to the ORI Governing Board to represent Google and help drive the foundation’s strategic direction, financial sustainability, and technical stewardship.

Driving long-term open source sustainability

The OpenROAD Initiative’s mission is to advance and sustain the open source EDA ecosystem by fostering collaborative innovation across research, education, and industry—transforming ideas into silicon. Google’s membership aligns directly with ORI’s multi-year sustainability goals, supported by the US National Science Foundation’s (NSF) Pathways to Enable Open-Source Ecosystems (POSE) program.

With Google’s participation and membership commitment, ORI will continue to strengthen, grow, and sustain its open source ecosystem through key vectors:

Neutral Stewardship: Fostering transparent governance where no single company has outsized control over the code, ensuring the project remains inspectable, accessible, and community-driven.
Ecosystem Growth: Supporting open and reproducible silicon research, developing robust design flows, and hosting global design contests.
Workforce Development: Supporting global silicon skilling initiatives by expanding open source chip design curricula and collaborating with academic institutions and industrial training networks.
Technical Strengthening: Enhancing continuous integration and deployment (CI/CD) pipelines, expanding PDK enablement, and improving user experience.

Leadership perspectives

“The OpenROAD Initiative is built on the vision of making chip design open and accessible to all—building a collaborative ecosystem driven by transparency and shared innovation,” said Andrew Kahng, board member of the OpenROAD Initiative. “Google’s deep commitment to open source software and hardware makes them an ideal partner. By joining at our highest membership tier, Google is helping to ensure that the open source EDA ecosystem has the stable, long-term governance and financial foundation required to grow.”

“Cutting-edge silicon research requires robust, inspectable, and reproducible toolchains,” said Drew Wingard, Director of Silicon Infrastructure, Tools and Methodology at Google. “OpenROAD has already made an incredible impact across academia and the broader industry, enabling many successful tapeouts. Google is proud to support the OpenROAD Initiative’s mission to scale this open infrastructure for the next generation of developers.”

About the OpenROAD Initiative and OpenROAD project

The OpenROAD Initiative, Inc. is a California-based 501(c)(3) nonprofit organization that provides governance, stewardship, and coordination for the OpenROAD ecosystem. The OpenROAD Project is an open source, autonomous digital chip design toolchain that democratizes semiconductor design, enabling a complete RTL-to-GDSII flow in less than 24 hours with no human in the loop. Grounded in academic research and referenced in over 500 peer-reviewed publications, OpenROAD has lowered the barriers to hardware innovation, enabling thousands of students, researchers, and startups worldwide to design and manufacture chips.

To learn more about the OpenROAD Project and install the toolchain, visit the new OpenROAD website.

For more information about membership tiers and the foundation’s governance, visit the OpenROAD Initiative website at www.openroadinitiative.org or contact membership@openroadinitiative.org.

Adapting open source practices to an AI-first world: A retrospective on 2025

2026-08-03T11:30:00.000-07:00

by Sophia Vargas, Google Open Source

Even as AI adoption accelerates and transforms the global technology landscape, open source remains foundational to how Alphabet builds, uses, and collaborates on products for billions of users. Our commitment to open source remains broad and consistent, including sharing our work year-over-year, and reflecting on what we've learned.

In 2025:

Roughly 10% of Alphabet's full-time workforce actively contributed to open source projects. This contribution ratio has remained steady over the past five years, scaling to match our growth.
These open source contributions are not just solely focused on Google. Our top projects by unique contributors at Alphabet include community-led projects such as LLVM, vLLM, Envoy, and Rust, as well as Google-initiated projects like Kubernetes, Apache Beam, and gRPC.
In addition, Alphabet projects received commits from more than 20,000 non-Alphabet affiliated user accounts.

Working together on emerging standards

Open source communities continue to provide vital collaborative spaces to define emerging standards, ensuring the interoperability and extensibility for the next generation of technologies. In 2025, we worked with more than 50 partners on the Agent2Agent (A2A) protocol to enable AI agents to communicate with each other, securely exchange information, and coordinate actions on top of various enterprise platforms and applications. Within weeks of our initial announcement, Google donated the A2A project to the Linux Foundation as part of our long-standing commitment to develop "open, collaborative ecosystem – offering greater autonomy and multiplying productivity."

Launching tools with transparency

Open source licenses provide a framework for anyone to explore, test, fork and expand on our technologies. Over the last 15 years, Google has created more than 15,000 public repositories on GitHub. Today, Google continues to maintain more than 5,000 public repositories on GitHub, and more than 1,500 public repositories on Git-on-Borg. A quick look back to some highlights from our 2025 launches includes:

Gemma 3 – a collection of lightweight open models built from the same research and technology that powers our Gemini 2.0 models. They are designed to run directly on devices — from phones and laptops to workstations — helping developers create AI applications, wherever people need them. Since Gemma’s release in 2024, the Gemmaverse community has created more than 60,000 Gemma variants. We’ve continued work on this collection and in 2026 we released Gemma 4 under an Apache 2.0 license.
Agent Development Kit – an open-source framework and SDK designed to help developers build, compose, and run both conversational and non-conversational AI agents.

Adjusting our collective security practices

As the threat landscape evolves and accelerates in the AI era, we are working on novel solutions to mitigate vulnerabilities at scale and remove some of the burden from overloaded maintainers. In 2025, we introduced several critical security initiatives to support upstream open source projects:

CodeMender: An agent designed to be both reactive, instantly patching new vulnerabilities, and proactive, rewriting and securing existing code and eliminating entire classes of vulnerabilities in the process. In its first six months, we were able to upstream 72 security fixes to open source projects, including some as large as 4.5 million lines of code.
OSS Rebuild: A security initiative to prevent software supply chain attacks by verifying build provenance. It automates package rebuilding and semantically compares results to upstream artifacts to detect tampered code on registries like PyPI, npm, and Crates.io.
OSV-Scalibr: (Software Composition Analysis Library) the core engine for vulnerability scanning used internally and in OSV-Scanner.
Credential Scanning on deps.dev: A service (using Veles) that actively scans open-source packages on deps.dev for leaked GCP credentials to prevent account compromise.

Working together to sustain the contributor community

Beyond security contributions, we remain committed to providing financial support directly to projects and maintainers. In 2025, the Open Source Programs Office (OSPO) directed $2M in sponsorships and investments to more than 40 open source projects.

Google Summer of Code celebrated its 21st year of enabling open source organizations to find, mentor, and onboard new contributors, directly supporting 1,280 individuals to contribute to 185 organizations. Over its lifetime, the global program has connected more than 23,000 participants from 125 countries with over 1,000 open source organizations globally. Moving forward, we recognize the need to evolve our program structures to ensure they continue to effectively support maintainers in the AI era.

Evolving the way we work in open spaces

Respecting community norms and practices is fundamental to establishing and maintaining trust in open source communities. To ensure we remain supportive members of your community, we encourage projects to document preferred practices and policies in the wake of new technologies. As AI changes the way we work, we are continually evaluating how best to evolve the way we engage in open spaces to preserve, prepare and bolster the communities we depend on.

We are deeply grateful for the many individuals and organizations that have worked with us to create global technologies from which everyone can benefit. You can continue to learn more about our open source initiatives, ongoing programs, and new projects at opensource.google.

Appendix: About this data

This report features metrics provided by many teams and programs across Alphabet. In regards to the code and code-adjacent activities data, we wanted to share more details about the derivation of those metrics.

Data sources: These data represent the activities of Alphabet employees on public repositories hosted on GitHub and our internal production Git service Git-on-Borg. These sources represent a subset of open source activity currently tracked by Google OSPO.
Business and personal: Activity on GitHub reflects a mixture of Alphabet projects, third-party projects, experimental efforts, and personal projects. Our metrics report on all of the above unless otherwise specified.
Alphabet contributors: Please note that unless additional detail is specified, activity counts attributed to Alphabet open source contributors will include our full-time employees as well as our extended Alphabet community (temps, vendors, contractors, and interns). In 2025, full time employees at Alphabet represented more than 95% of our open source contributors.
GitHub Accounts: For counts of GitHub accounts not affiliated with Alphabet, we cannot assume that one account is equivalent to one person, as multiple accounts could be tied to one individual or bot account.
Active counts: Where possible, we will show ‘active users’ defined by logged activity (excluding ‘WatchEvent’) within a specified timeframe (a month, year, etc.) and ‘active repositories’ and ‘active projects’ as those that have enough activity to meet our internal active-project criteria and have not been archived.

This Week in Open Source for July 16, 2026

2026-07-16T11:30:00.000-07:00

by Daryl Ducharme, Open Source Programs Office

This Week in Open Source for July 16, 2026

A look around the world of open source

We’re diving into another week of open source news and discussions. As the ecosystem continues to evolve, we're seeing more conversations around the intersection of AI workflows, security at scale, and the infrastructure that supports the open web.

This week, we’re highlighting a few "Open Source Reads" that tackle some of the biggest questions facing our ecosystem today—from the complex ethics of AI-generated workflows to the future of federated social networks. We hope these links provide valuable context as we work together to sustain the critical infrastructure we all rely on.

Upcoming Events

ASWF Open Source Days (July 19–20) — Los Angeles, CA. Hosted by the Academy Software Foundation, focusing on open source software in visual effects (VFX) and animation.
KubeCon + CloudNativeCon Japan 2026 (July 28–30) — Yokohama, Japan. The Cloud Native Computing Foundation's flagship conference in Japan.
Black Hat & DEF CON 2026 (August 1–9) — Las Vegas, NV. While primarily cybersecurity, both heavily feature open source hacking tools. The Open Source Security Foundation (OpenSSF) also has a major presence here.
FOSSY (August 6–9) — Vancouver, Canada. A highly community-focused open source conference emphasizing grassroots participation and collaboration.
Community over Code Asia (August 7–9) — Beijing, China. The Asian edition of the Apache Software Foundation's (ASF) community-first conference.
Open Source Summit Korea (August 11–12) — Seoul, South Korea. Hosted by the Linux Foundation, bringing together open source maintainers and enterprises.
KubeCon + CloudNativeCon + OpenInfra Summit + PyTorch Conference China (September 7–9) — Shanghai, China. A massive collaborative mega-event uniting open infrastructure, cloud, and AI communities.
OSPOlogy + OSPO Summit (September 7) — Shanghai, China. Focused on Open Source Program Offices (OSPOs), governance, and corporate open source sustainability.

Open Source Reads and Links

[Blog] Elephants, Goldfish and the New Golden Age of Software Engineering - This isn't directly open source, but with the open source ecosystem trying to find good AI workflows I thought the Elephant-Goldfish model was an interesting take. It makes the distinction between using AI as a toy and using it as a tool.
[Article] Open-source AI and the Choice Before Us - The communities with minimal influence on AI development will likely be the most affected by it. This article looks at how strong rules and institutions are needed to guide AI's safe and fair use. The future of AI depends on whether we build systems that include everyone or just deepen existing divides.
[Article] Open source carries the world. Patching it at Mythos-scale can't fall to maintainers alone. - The past year has shown that AI is speeding up the discovery of vulnerabilities in open source software. Endor Labs is offering a solution to utilize AI to create patching stop gaps for users of the libraries so that maintainers and contributors have the time to create good quality fixes to these vulnerabilities.
[Blog] The AT-URI Syntax Mess - I am very excited about the open social web, especially things happening around the AT protocol. Early misunderstanding led to the current AT URI syntax being invalid. This article looks at the possible ways forward.
[Article] The Open Social Web Needs Section 230 to Survive - The open web has always been a promise of free speech. When walled garden social media got involved it created some questions around whether free speech applies to the content served by the corporations running those networks. However, the open social web is not run by corporations. It can be, and it can also be run by individuals federating instances or running their own PDS. So, how do we maintain the free speech promise of the internet with this in mind?

Which of these stories will you be chatting about at your next meetup or conference? Let us know! Share with us on our @GoogleOSS X account or our @opensource.google Bluesky account.

Google Cloud: PostgreSQL community contribution updates

2026-07-10T11:30:00.000-07:00

by Dilip Kumar, Cloud SQL for PostgreSQL & Matt Cornillon, Sales EMEA

Google Cloud is deeply committed to the long-term success of the PostgreSQL ecosystem. Our involvement goes beyond providing PostgreSQL managed services; it's also about active participation in the open source communities through technical contributions, leadership in conference committees, and sharing architectural insights that benefit all users. Following is a recap of recent events Google Cloud participated in.

PGConf.dev 2026

Serving as a vital developer-centric hub, PGConf.dev provides a unique opportunity for collaboration with the full assembly of senior PostgreSQL committers. This gathering is essential for aligning technical efforts and shaping the future project roadmap.

Key Highlights

Participation focused on strategic coordination with PostgreSQL committers regarding logical replication development, and a consultation on global index architecture.
High community interest confirms the Global Index feature solves a vital architectural requirement for enterprises.
Established community consensus to pursue a deparsing-based architectural approach for DDL replication.

Google Cloud Sessions

Dilip Kumar, a PostgreSQL contributor from Google Cloud, presenting "Experimenting with a Global Index in PostgreSQL" at pgconf.dev 2026 in Vancouver. He is speaking at a podium next to a presentation slide detailing the Global Index storage architecture and PartitionIdentifier management.

Session Title	Session Type	Speakers/Led by
Experimenting with a Global Index in PostgreSQL: Design, Implementation, and Challenges	Technical Talk	Dilip Kumar
Unconference: Global Indexes	Unconference Session	Dilip Kumar
Unconference: Logical Replication: Warts and Missing Pieces	Unconference Session	Hannu Krosing

PGConf India 2026

Key Highlights

The three-day conference was divided into a training day followed by two days of sessions. More than 580 participants attended the conference over three days.
The conference sessions included a mix of keynotes, breakout technical sessions, sponsor sessions, and booth interactions.

Google Cloud Sessions

Session Title	Session Type	Speakers
Database And GenAI	Keynote	Paresh Rathod
Experimenting with a Global Index in PostgreSQL	Technical Talk	Dilip Kumar
GCP - Best home to run PostgreSQL	Sponsor Session	Trusar Borse, Abhijeet Rajkur
Beyond shared_buffers: On-Demand Memory PostgreSQL	Technical Talk	Rajeev Rastogi, Vaibhav Popat
Where is my Memory	Technical Talk	Pushkar Kalidkar
Agentic AI Applications with GCP Databases	Keynote	Abhijeet Rajkur, Rishi Kapoor, Saurabh Gupta

PGDay Paris & PGDay France 2026

France hosts two distinct flagship PostgreSQL events, and Google Cloud is deeply embedded in both as both organizers and technical contributors. While PGDay Paris serves as an international, English-language hub for the European community, PGDay France is a community-driven, traveling event that focuses on the francophone ecosystem, taking place in Toulouse for 2026.

Key Highlights

Matt Cornillon served on the organization committee for PGDay France, while Yves Colin contributed as a member of the program committee.

Google Cloud Sessions

Session Title	Session Type	Speakers
Creating a "Dungeon Master" with Postgres and MCP	Technical Talk	Matt Cornillon
Create your first AI agent with PostgreSQL	Workshop	Matt Cornillon, Yves Colin

PGDay FOSDEM 2026

FOSDEM PGDay is a prominent open source gathering that brings together developers from across the globe to discuss the latest PostgreSQL advancements. It serves as an essential platform for exploring emerging paradigms in database development.

Key Highlights

Exploration of how AI-assisted workflows are redefining development beyond standard autocomplete for SQL queries.

Google Cloud Sessions

Session Title	Session Type	Speakers
Vibe-coding with Postgres: really?	Technical Talk	Matt Cornillon

PGConf Belgium 2026

PGConf Belgium 2026 took place at the UCLL Campus Proximus in Haasrode, Belgium, serving as an outstanding learning and networking platform for the local PostgreSQL community and students.

Key Highlights

The session was selected by faculty as supportive material for a database exam following deep student engagement.

Google Cloud Sessions

Session Title	Session Type	Speakers
Creating a "Dungeon Master" with Postgres and MCP	Technical Talk	Matt Cornillon

Nordic PG Day 2026

Nordic PG Day is the largest PostgreSQL event in the Scandinavian countries. The 2026 edition took place in Helsinki, gathering more than 130 PostgreSQL enthusiasts for a day of deep dives.

Key Highlights

Google joined as an official Partner-level sponsor for the first time, including a dedicated table booth.

Google Cloud Sessions

Session Title	Session Type	Speakers
Unlock AI Agents with PostgreSQL	Technical Talk	Mats Berglin, Miguel Toscano

Swiss PGDay 2026

Swiss PG Day is the annual event organized by the Swiss PostgreSQL User Group in Rapperswil, Switzerland. The ninth edition featured sessions in both English and German.

Key Highlights

Demonstration of the physical impact of pushing millions of vectors to PostgreSQL based on a real-world use case.

Google Cloud Sessions

Session Title	Session Type	Speakers
Surviving pgvector in production: a reality check	Technical Talk	Miguel Toscano

Postgres Conference: 2026 San Jose

Since its inception in 2007, the Postgres Conference has served as a cornerstone for advancement, fostering a rich environment for learning and professional networking.

Key Highlights

Google proudly served as a sponsor for the event.
Adapting PostgreSQL for the artificial intelligence era demands a transformation in operational approaches. With the rise of natural language tools and vibe coding speeding up development, Agentic AI places advanced demands on production databases. In their presentation, Vikas and Vishal examine how Google Cloud managed services have evolved to handle these workloads, providing architectural strategies and best practices for contemporary AI deployment.

Google Cloud Sessions

Session Title	Session Type	Speakers
Postgres and AI - Stronger Together!	Technical Talk	Vikas Arora and Vishal Bagga

Community Leadership and Committees

Googlers play a vital role in shaping the direction of the most prestigious PostgreSQL developer events. Our leadership in these committees helps ensure that enterprise-grade requirements—such as those needed for large-scale migrations—are part of the global conversation.

PGConf.dev 2026: Dilip Kumar served on the Program Committee.
PGConf India 2026: Dilip Kumar was a member of the Paper Selection Committee.
PGDay France: Matt Cornillon was a member of the organization committee and Yves Colin served as a member of the Program Committee.

Looking Forward

Our commitment remains firm: to turn feedback from these global events into code, reviews, and active community partnerships. We thank the wider PostgreSQL community and the project's committers for their continued collaboration in making PostgreSQL better for everyone.

Acknowledgement

We extend our heartfelt appreciation to our open source community contributors for their outstanding dedication and active participation in making PostgreSQL conferences a great success.

Abhijeet Rajurkar, Darshan Nagarajappa, Dilip Kumar, Hannu Krosing, Mats Berglin, Matt Cornillon, Michael Bautin, Miguel Toscano, Niranjan Shivprasad, Paresh Rathod, Rajeev Rastogi, Vaibhav Popat, Vikas Arora, and Yves Colin

Furthermore, we are deeply grateful to the broader PostgreSQL open-source communities, especially the dedicated conference organizers, committee members, and all supporting sponsors.

Community feedback: How can corporations improve support for open source maintainers?

2026-06-30T11:30:00.000-07:00

by Sophia Vargas, Google Open Source

We know that AI is actively transforming the sustainability and socio-technical dynamics of OSS communities. Google Open Source is committed to partnering with open source communities and ecosystems to learn together how we should update our own models for engagement and support.

During an open meetup for GitHub Maintainer Month, I led a session to gather community feedback on how corporations can more effectively support open source maintainers.

Paying maintainers takes creativity

Many maintainers would appreciate consistent financial support. However, facilitating payments to individuals without established contractual relationships remains a complex challenge, particularly across diverse international jurisdictions. Fiscal hosts and programs such as Open Collective, GitHub Sponsors, and the LFX Mentorship Program can simplify components of this process, but they do not resolve the underlying issues of funding sustainability and predictability. While initiatives like the Open Source Endowment are working toward long-term funding sustainability, individual maintainers also had a few ideas:

Pay per meaningful contribution vs gameable metrics: Avoid payment models based on easily manipulated units like pull request counts or review volume. A proposed alternative is ‘pay per report,' encouraging maintainers to document their achievements and upcoming roadmaps.
Commitment-based purchasing: Corporate policies might make procurement simpler (or more complex) than sponsorships, so maintainers could benefit from offering structured support services alongside traditional sponsorship opportunities.
Fund conference attendance: In-person networking can be a boon for solo maintainers but it's often cost-prohibitive. For some corporations, travel sponsorship may be a simpler alternative to direct payments.

Challenge for Corporations and Fiscal Hosts: How can we assist maintainers in understanding any and all prerequisites and documentation necessary to participate in monetary programs?

Advice from Maintainers: Consult a tax professional to understand the implications of various funding methods.

Manage and respect expectations

Beyond financial support, our discussion returned to the importance of respect and etiquette. Particularly, how can we manage expectations between heterogeneous creators, contributors and users - are maintainers clearly communicating their preferences, and are corporations actively respecting them? Some suggestions include:

Adherence to community norms: Maintainers should share their preferred communication channels, while contributors - both human and agentic - must ensure they review and follow them.
Consistency with documentation: Discrepancies between documented procedures and actual practices create friction for all participants. This standard should be upheld by both individual maintainers and corporate-managed projects.
Clarity of intent: Many maintainers would like to understand the motivation behind a contribution and reserve the right to ask questions.

To improve specific program experiences, maintainers suggested:

Consistent communication: Recipients of funding programs expect clearly communicated expectations regarding the timing and amount of disbursements.
Transparency and discoverability: Maintainers would appreciate easily discoverable records that track program participation, active agreements, and verify the status of Contributor License Agreements (CLAs).

Let's keep learning as a community

While we cannot make any promises, we want to continue to learn and challenge ourselves to consider novel ways to support OSS communities and maintainers. As a member of our community, we value your opinion. We've created a Google form to collect any thoughts you might have, as well as gauge interest in another open meeting. We plan to share any and all learnings back with the community.

Documenting the manual: how curiosity and robotic arms led to a career in open source

2026-06-22T11:30:00.000-07:00

by Daryl Ducharme, Google Open Source

When you think of "innovation" in open source, your mind probably jumps to the latest AI model or a revolutionary new framework. You might not immediately think of manual pages. Even Alejandro "Alex" Colomar, who spends his days maintaining Linux Kernel documentation, jokingly admits that some might find the work "boring" because it focuses on fixing existing issues and documenting new features rather than flashy inventions.

But as any developer knows, the most powerful code is only as good as the documentation behind it. At Google, we believe that investing in the success of projects we don't own is a core part of being a good open source citizen. That is why we are proud to sponsor Alejandro's work on the Linux Kernel man-pages project—supporting the critical infrastructure that many of our own systems rely on every day.

Documentation is the gift you give to your future self and your whole community.

The precision of a robot

Alejandro's journey into the world of essential documentation started at university. He was working with robotic arms that used a proprietary scripting language. Wanting more control, he decided to write a C library to communicate with the robots over the network by sniffing packets with Wireshark. It worked, but it was slow—he had to wait seconds between commands to ensure the robot had finished moving.

To make the movements smooth, he needed to understand the messages the robot was sending back in real-time. This required high-precision timing. He found SO_TIMESTAMP, which provided microsecond precision, but he noticed a macro called SO_TIMESTAMPNS in the header files that promised nanosecond resolution. The problem? It wasn't documented in the manual page.

The first patch

After figuring out how to use the undocumented feature by looking at the kernel source code, Alejandro decided to ensure the next person wouldn't have to struggle. He cloned the man-pages repository, wrote a new paragraph based on existing features, and figured out how to send a plain-text patch via email.

"As it was my first patch, I was a bit intimidated by the procedure," Alejandro recalls. That intimidation led to a commit message he is still proud of today: roughly 120 lines of explanation for just 25 lines of new documentation. He wanted to prove that he had done his homework. The welcoming response from the maintainer encouraged him to keep going, leading to more patches and, eventually, a career-long dedication to clarity in open source communities.

Sustaining the commons

Google understands that open source is a "small community built on trust." By supporting maintainers like Alejandro, we help ensure that critical infrastructure—like the documentation that powers the Linux ecosystem—remains accurate and accessible for everyone. We believe that using open source comes with a responsibility to contribute and sustain it, which is why we partner with developers to maintain and grow critical projects.

Alejandro's work doesn't just help himself; it helps thousands of other programmers who rely on correct documentation to build the next generation of technology. As he puts it: "I couldn't program without correct documentation, so whenever I find an issue in documentation, I try to fix it."

A garden that needs tending

We often say that a community is a garden, not a building—it requires constant tending, not just initial construction. By sponsoring Alejandro, we are helping to tend that garden, ensuring the "manual" remains a living, breathing resource for the global developer ecosystem. Whether it is fixing a typo or documenting a high-precision networking macro, every contribution makes the "eyes" on the code that much sharper.

In-place pod restarts: Boosting efficiency and workload reliability in Kubernetes v1.35

2026-06-18T11:30:00.000-07:00

by Duncan Campbell & Giuseppe Tinti Tomio, Kubernetes

Operational efficiency and system resilience are critical when running scaled platforms. Yet, in Kubernetes, recovering from software crashes remains a headache because you couldn't trigger a clean restart of a Pod's containers without recreating the entire Pod object, leading to some amount of resource waste.
To address this, Restart All Containers on Container Exits graduated to beta and is enabled by default in Kubernetes v1.36. Developed in close collaboration with the CNCF community, this capability represents Google's commitment to investing in the success of foundation-led open source projects. By sharing best practices from running large distributed systems internally, we are helping build a more resilient and efficient ecosystem. Letting containers restart while keeping the Pod's runtime identity provides a built-in way to perform in-place Pod recovery, boosting application reliability and saving resource costs.

The Problem: The High Cost of Pod Re-creation

Historically, Kubernetes managed failures using pod level restart policies. While sufficient for simple services, modern multi-container Pods often have complex dependencies. When a failure requires a full environment reset, your only option was deleting and recreating the entire Pod.
This introduces massive control plane churn, causing latency and pressure on the etcd backend during large failures:

Initialization Dependencies: If a main container corrupts a local environment, for example, single-use secrets that must be re-requested, restarting just that container is insufficient; the setup must run again.
Watcher Interoperability: If a watcher sidecar detects a fatal error, it must trigger a full recreate of the entire pod and its infrastructure, including the sandbox.
Stale States: If a database sidecar proxy restarts, the main application can get stuck attempting to use stale, broken connections.
Resource Race Conditions: When a large job finds a proper set of nodes, recreating Pods can lead to other pending Pods taking over those resources. In-place restarts eliminate this race condition risk.

Previously, resolving these failures required destroying the entire Pod. For large batch or AI/ML workloads, where thousands of Pods might fail simultaneously, this can lead to "Thundering Herd" scheduling requests, delaying recovery and wasting expensive GPU/TPU compute time.

Introducing In-Place Restarts: The RestartAllContainers Action

Kubernetes v1.35 introduces the RestartAllContainers action, enabled by the RestartAllContainersOnContainerExits feature gate, which graduated to beta in 1.36 alongside its dependencies ContainerRestartRules and NodeDeclaredFeatures. This lets a container's exit behavior trigger a fast, in-place restart of the entire Pod on its existing node.
The Kubelet halts all containers while keeping the Pod sandbox intact, preserving critical infrastructure:

Network Identity: Keeps the same IP, network namespace, and UID, completely bypassing IP reassignment.
Hardware and Devices: Keeps GPUs/TPUs bound, eliminating scheduling and re-allocation delays.
Storage Mounts: Volumes, including emptyDir and PVCs, remain fully mounted; their content is not cleared during restarts.

Once terminated, the Kubelet re-runs init containers (including sidecars, which are part of the init sequence) in order, guaranteeing a clean setup in a known-good environment.

A Native Pod Specification Example

You can implement this under the container's restartPolicyRules field. Here is a quick example of how a watcher sidecar can trigger an in-place restart of the entire Pod by exiting with code 88:
YAML
Note: Image names and paths in the YAML below are for illustrative purposes.

apiVersion: v1
kind: Pod
metadata:
  name: ml-worker-pod
spec:
  restartPolicy: Never
  initContainers:
    - name: setup-environment
      image: registry.k8s.io/ml-tools/setup-worker:v1.0
    - name: watcher-sidecar
      image: registry.k8s.io/ml-tools/watcher:v1.0
      restartPolicy: Always
      restartPolicyRules:
        - action: RestartAllContainers
          exitCodes:
            operator: In
            values: [88]
  containers:
    - name: main-application
      image: registry.k8s.io/ml-tools/training-app:v1.0

The Operational Impact of In-Place Restarts

For organizations running distributed workloads, RestartAllContainers provides serious operational advantages:

No Control Plane Overhead: By preserving identity, clusters avoid scheduling latency and DNS propagation. This was a key factor for JobSet using this feature to reduce recovery from minutes to seconds.
Node Locality Preservation: Since the Pod stays anchored to the same node, restarted containers can instantly access local, warm storage caches.
Maximized Hardware Efficiency: In distributed AI training, losing a single node halts the entire job. Keeping accelerators like GPUs/TPUs bound lets workloads resume training significantly faster, directly reducing compute costs.

Observability and SRE Best Practices

To support monitoring, Kubernetes v1.35 introduces the AllContainersRestarting Pod condition. Set to True during restarts, it alerts SREs and autoscalers, preventing false-positive alerts, while container restart counts increment to let Prometheus easily track recovery events.
To use in-place restarts successfully, shift your mental model to "persistent sandboxes" and follow three best practices:

Ensure Reentrancy: Kubelet only guarantees "at least once" execution for init containers. Reentrancy is now a standard requirement, so your code must be fully idempotent.
Plan for Termination Handling: Graceful termination (preStop hooks) is not supported for in-place restarts. SIGKILL is almost immediate, so applications must handle sudden exits gracefully.
Prepare External Tooling: CD and observability tools should expect re-running init containers without interpreting them as new deployments.

What's Next?

This beta capability is a major step toward fluid workload management and serves as a building block for advanced community features like JobSet in-place restarts (KEP-467).
Our work on KEP-5532 reflects our commitment to transparent open source governance. Developed collaboratively within SIG Node, this feature shows how we hold ourselves to high citizenship standards; making our design, goals, and intentions transparent while building shared best practices that benefit everyone. We encourage you to experiment with Kubernetes v1.35 and share your feedback with the community!

Learn More

Read the Kubernetes Pod Lifecycle Documentation.
Explore KEP-5532: Restart All Containers on Container Exits.
Review theKubernetes v1.35: Restart All Containers Blog Post.
Join the SIG Node Community on Slack (#sig-node).

Open rails for agentic commerce at Open Source Summit North America 2026

2026-06-16T16:16:03.340-07:00

by Anurag Sinha, Universal Commerce Protocol (UCP)

At Open Source Summit North America 2026, I shared why agentic commerce needs open rails.

As AI agents become more capable, the shopping journey is shifting from "show me" to "help me." Instead of browsing, comparing, clicking, and checking out step by step, people can increasingly ask an agent to help them decide what to buy and, in some cases, complete the purchase. Industry forecasts suggest agentic shopping could account for roughly 10% to 25% of U.S. e-commerce by 2030 (Bain), which points to a meaningful shift in how digital commerce will work. Watch the full keynote here.

Why shared rules matter

That shift also exposes a challenge. Commerce is still highly fragmented. Different businesses, payment providers, and platforms operate with their own rules, workflows, and business logic. Every new surface adds more integration work. Every bespoke connection creates more complexity. And that fragmentation makes it harder for AI systems to understand and perform commerce actions consistently across businesses. A shared language lowers that barrier for everyone.

A common language for agentic commerce

That is the problem Universal Commerce Protocol (UCP) is designed to solve.

We launched the Universal Commerce Protocol, or UCP, with industry leaders to establish an open standard for agentic commerce, built to work across the shopping journey. UCP creates a common language for agents and systems to operate together across consumer surfaces, businesses, and payment providers, so the ecosystem does not need a different bespoke integration for every new agent or platform.

Just as importantly, UCP is designed for the real world. Every business has its own way of selling. Checkout, fulfillment, loyalty, policy logic, shipping, and post-purchase flows can vary widely between a local shop, a marketplace, and a large retailer. UCP is built to support that reality.

A layered architecture for a shared commerce language

UCP uses a layered model to create a reusable shared language for commerce. Services organize domains like shopping and common. Capabilities define core actions such as checkout, catalog, cart, orders, and shared functions like identity linking. Extensions keep those capabilities configurable, so features like fulfillment can be modeled once and reused across multiple flows instead of being hardwired each time. At the transport layer, UCP stays agnostic, supporting bindings like REST, Model Context Protocol, and Agent2Agent.

Together with capability discovery and payment handling, these layers help consumer platforms, agents, and businesses interoperate more consistently over time. They also let different participants advertise what they support, compose new behaviors, and communicate over the transport that works best for them.

Built in the open

A standard for everyone should be shaped by everyone. Because UCP is open, merchants, developers, and community contributors can pressure-test real-world gaps, propose new capabilities and extensions, and help make sure the protocol reflects more than the needs of the largest players. That kind of participation is what keeps an ecosystem moving.

Since launch, UCP has continued to evolve through new capabilities, an expanded Tech Council, and new consumer experiences built on top of the protocol. That momentum matters because standards only work when the ecosystem uses them.

Watch the full keynote

Agentic commerce is still evolving, and UCP is a foundational building block to support what's next in this new era.

If you want the full architecture walkthrough and the complete story from Open Source Summit North America, watch the session here. And if you want to go deeper, you can explore the UCP documentation, join the community conversation, and contribute to the public repository.

CEL finds a new home at github.com/cel-expr!

2026-06-16T11:30:00.000-07:00

by Olena Huang, CEL (Common Expression Language) team

We're excited to announce that the official Common Expression Language (CEL) repositories have moved to a dedicated GitHub organization. Visit the new cel-expr repository now!

Why the move?

This move is a key step in strengthening the CEL ecosystem. By centralizing our projects, including the language specification, Go, C++, C, Java, and Python implementations, under the cel-expr organization, we aim to:

Enhance Branding: Create a clear and unified brand identity for CEL.
Improve Discoverability: Make it easier for users and contributors to find all official CEL resources in one place.
Ensure Consistency: Foster consistency across all CEL projects.
Streamline Development: Simplify our development and release processes.

What's Changing?

The following repositories now reside in the cel-expr organization:

google/cel-spec is now cel-expr/cel-spec
google/cel-cpp is now cel-expr/cel-cpp
google/cel-go is now cel-expr/cel-go
google/cel-java is now cel-expr/cel-java
cel-expr/cel-python and cel-expr/cel-c have already been in the cel-expr namespace

All future development, issues, and pull requests for these projects will take place in their new homes within the cel-expr organization. This is a non-breaking change, due to automatic redirects, but you should update your URLs where possible.

What Stays the Same?

We've worked to make this transition as seamless as possible:

Automatic Redirects: GitHub will automatically redirect all web traffic and git operations from the old google/cel-* URLs to the new cel-expr/cel-* locations. Your existing links and git remote configurations pointing to the old URLs should continue to work for cloning and fetching.
Preserved History: The full commit history, issues, and pull requests for each repository have been migrated and are available in the new locations.

Action Required: Update Your Dependencies

While existing links and git remote configurations pointing to the old URLs should continue to work thanks to GitHub's redirects, we recommend updating your dependency management configurations (e.g., go.mod, pom.xml, requirements.txt, etc.) to point directly to the new repository URLs under https://github.com/cel-expr. This ensures you are fetching the latest code and releases from the canonical source.

We're thrilled about this new chapter for CEL, bringing all our core components under one roof. We believe this will foster a stronger CEL community and accelerate the development and adoption of CEL.

Explore the new organization at https://github.com/cel-expr!

A new pkg.go.dev API for Go

2026-06-12T11:30:00.000-07:00

by Ethan Lee, Jonathan Amsterdam & Hana Kim, Go Team

Access to Go metadata has been an everpresent need for the Go community. Since its launch, pkg.go.dev has served as a central hub for Go package documentation and discovery. While we initially prioritized providing this comprehensive access via a web interface, the need for streamlined programmatic access has become increasingly clear.

Structured API access has been one of the most highly requested features for pkg.go.dev for a while now. Developers building tools, IDE integrations, automated workflows, and other systems have had to rely on inconsistent and fragile scraping methods. By providing a formal API, we can provide fast and efficient access to required data. This foundation also sets Go up for the future of AI-assisted coding. Large language models and agents can access the context necessary to reason about the Go ecosystem with greater precision and accuracy.

Empowering Tool Builders

Our goal with this API is to reduce the technical churn for builders and innovators. By offering structured JSON metadata, we address the following use cases:

Search and Discovery: The API enables fast and efficient search across the entire Go module ecosystem.
Driving AI Innovation: As AI-assisted coding evolves, LLMs and agents need precise context. This API provides the data required for agents and models to reason deterministically about Go packages.

The Service Interface

Built for stability and efficient caching, the API uses a stateless, GET-only architecture. Primary endpoints are currently hosted under the v1beta path. Following a period of feedback from the Go community and confirmed stability, we intend to transition toward a formal v1 release.

For a complete interactive reference of all endpoints, query parameters, and response shapes, see pkg.go.dev/api. The machine-readable API contract is also published directly at pkg.go.dev/v1beta/openapi.yaml.

Endpoint	Description
`/v1beta/imported-by/{path}`	Paths of packages importing the package at `{path}`.
`/v1beta/module/{path}`	Information about the module at `{path}`.
`/v1beta/package/{path}`	Information about the package at `{path}`.
`/v1beta/packages/{path}`	Information about packages of the module at `{path}`.
`/v1beta/search/search?q={query}`	Search results for a given query.
`/v1beta/symbols/{path}`	List of symbols declared by the package at `{path}`.
`/v1beta/versions/{path}`	Versions of the module at `{path}`.
`/v1beta/vulns/{path}`	Vulnerabilities of the module or package at `{path}`.

An example of retrieving package information is shown below:

curl https://pkg.go.dev/v1beta/package/github.com/google/go-cmp/cmp | jq
{
  "modulePath": "github.com/google/go-cmp",
  "version": "v0.7.0",
  "isLatest": true,
  "isStandardLibrary": false,
  "goos": "all",
  "goarch": "all",
  "path": "github.com/google/go-cmp/cmp",
  "name": "cmp",
  "synopsis": "Package cmp determines equality of values.",
  "isRedistributable": true
}

A Reference Implementation

To demonstrate how to interact with our API, we are providing a reference CLI implementation: pkgsite-cli. This implementation serves as a practical example for developers looking to build their own integrations, showing how to handle the data directly from the terminal. Note, as the API continues to evolve, the interface and behavior of this CLI may change.

You can use it to search for packages or inspect symbols without leaving your shell:

go install golang.org/x/pkgsite/cmd/internal/pkgsite-cli@latest

pkgsite-cli search "uuid"
github.com/google/uuid
  Module:   github.com/google/uuid@v1.6.0
  Synopsis: Package uuid generates and inspects UUIDs.
... more


pkgsite-cli package github.com/google/go-cmp/cmp
github.com/google/go-cmp/cmp
  Name:      cmp
  Module:    github.com/google/go-cmp
  Version:   v0.7.0 (latest)
  Synopsis:  Package cmp determines equality of values.

pkgsite-cli package --symbols github.com/google/go-cmp/cmp
github.com/google/go-cmp/cmp
  Name:     cmp
  Module:   github.com/google/go-cmp
  Version:  v0.7.0 (latest)
  Synopsis: Package cmp determines equality of values.

Symbols:
  type Indirect struct{}
  type MapIndex struct{}
  type Option interface{}
  ... more

Looking Ahead

While we prioritize stability for our new /v1beta endpoints, we are eager to hear how open source communities use these resources to solve real-world problems.

We look forward to your feedback via our issue tracker and to seeing the tools you’ll build next.

Introducing OpenRL: A self-hosted post-training API for fine-tuning LLMs

2026-06-11T11:30:00.000-07:00

by Sunil Arora, Shuby Mishra & Chuang Wang, GKE

We are pleased to share a research preview of OpenRL, a new open-source project coming out of GKE Labs. OpenRL is a self-hosted training API for fine-tuning LLMs on your own Kubernetes cluster.

Why we built it

If you look at agentic RL on LLMs, it is incredibly easy to get bogged down in system complexity. To run a single RL loop, you have to coordinate a dozen different things: selecting and cleaning datasets, choosing RL environments, debugging training loops, managing reward signals, handling inference mismatches, allocating hardware, and managing infrastructure. Picture looks something like this:

Figure shows an AI researcher and an infrastructure engineer staring at the hurdles in post training along the way to the summit.

Each of these is a hard problem. But what makes it more complex is how tightly AI research and infrastructure concerns are mixed together in today's tooling and frameworks.

We believe decoupling the infrastructure from AI research can make these problems more tractable so that infrastructure engineers and AI researchers can independently tackle them. We have seen this pattern with Kubernetes where Kubernetes abstracted out the infrastructure and made application developers and SREs life easier.

So, can you abstract out post training infrastructure? We believe so and drew huge inspiration/validation from Tinker (from Thinking Machines). The Tinker APIs for post training hit that Goldilocks zone where it hides all the post training infrastructure behind four key APIs:

Figure shows high level components and their interaction in a OpenRL based RL workflow

So the end result of this abstraction is that AI Researchers get full flexibility on their RL loop and infrastructure engineers can focus on scaling, orchestration, and reliability. OpenRL allows you to run the same training APIs but on your own infrastructure. And this decoupling has other interesting benefits.

Sharing GPUs

Traditional RL loops are strictly sequential. The trainer waits for the sampler to finish rollouts, the sampler waits for the environment to score rewards (which is often bound by slow CPU/network tasks), and the whole loop sits blocked. Your expensive GPUs spend a lot of time doing nothing. The abstraction allows running multiple RL jobs and allows infrastructure engineers to pack the training/sampling steps to utilize more of their GPUs. The graph below shows the GPU consumption in OpenRL for running one, two, and three RL jobs concurrently.

The figure shows the trainer/sampler duty cycle in OpenRL for scenarios with 1 RL job, 2RL jobs and 3 RL jobs respectively.

Better UX

Once you separate out the infrastructure behind the APIs, you start to see the gains in user experience of developing the RL loop because AI researchers no longer have to wrangle the complex python dependencies like cuda. When you are doing R&D, you do not have to run the RL loop directly on the machines with GPUs, you can simply run your RL loop on your Mac pointing to the training APIs running on a Kubernetes cluster/VMs.

Autoresearch

We believe that frontier AI research will get more and more automated in the future and abstracting out infrastructure as a building block is key to that. To demonstrate that, we added an autoresearch recipe inspired heavily by karpathy's work. The recipe demonstrates how to conduct parallel experiments to conduct parameter sweep, and improve the reward signal for our text-to-sql recipe for Gemma models.

Figure showing autoresearch UI with multiple AI researchers conducting experiments in parallel in OpenRL

What OpenRL is not

A managed service. OpenRL is self-hosted and not a managed service. We aim to make it easy for users to deploy and operate it on their Kubernetes clusters.
An RL framework. OpenRL gives AI researchers full control over their RL loop.

Get started

We have made it easy to run OpenRL on your Mac, Nvidia GPUs, or on GKE. This allows you to test your RL loop on Mac and when you are ready to scale, you can point the RL loop to the OpenRL endpoint running in the GKE cluster.

Try out our text-to-SQL example for teaching the latest Gemma model SQL here: guides.

One of the benefits of a Tinker compatible endpoint is that you can use Tinker-Cookbook with OpenRL. Tinker-cookbook is one of the best resources for post training infrastructure for RL.

Future steps

We have started with a simple architecture focussing on LoRA fine-tuning and plan to evolve the project in the coming months, so please give it a try and share your feedback. A few things we are very excited to work on:

Full parameter fine-tuning
Multitenancy (simultaneous RL on different types of base models)

Acknowledgement

We have been inspired by the work done by various open source projects in AI communities, so huge thank you to Thinking Machines, vLLM, PyTorch, prime-rl, verl, SkyRL, and llm-d.

Google joins the Eclipse Foundation as a strategic member to accelerate AI-integrated developer tools

2026-06-10T11:30:00.000-07:00

by amanda casari & Mike Bufano, Google Open Source

Collaboration with the Eclipse Foundation will support open infrastructure for AI-integrated developer platforms like Google Antigravity, while advancing broader open source security and regulatory compliance initiatives

As of April 2026, Google has joined the Eclipse Foundation as a Strategic Member, reflecting the company's continued investment in open source technologies and modern developer infrastructure.

As part of this collaboration, Google will additionally sponsor Open VSX and is among the first adopters of the recently announced Open VSX Managed Registry service. Open VSX is the open source, vendor-neutral extension registry for tools built on the VS Code™ extension API. It powers a rapidly growing ecosystem of AI-integrated IDEs, cloud development environments, and developer platforms, including Google Antigravity, AWS's Kiro, Cursor, and, Windsurf among many others.

As a Strategic Member, Google will participate in the Eclipse Foundation's Board of Directors and Technical Advisory Council, helping guide the technical and strategic direction of one of the world's leading open source software foundations.

"The industry is feeling the massive turning point as AI continues to change how developers write, deploy, and maintain software," said amanda casari of Google's Open Source Programs Office and new Eclipse Board member. "Joining The Eclipse Foundation as a Strategic Member ensures that the next generation of AI-integrated developer experiences—including platforms like Google Antigravity—are built in partnership with transparent, vendor-neutral foundations. Open registries, like Open VSX, are critical infrastructure which keep the global developer ecosystem open to everyone."

Google and the Eclipse Foundation share a deep history, having collaborated across numerous initiatives since 2006. This Strategic Membership elevates the relationship and support critical to modern initiatives like Open VSX, Open Regulatory Compliance (ORC), and Adoptium.

"Google has played a pivotal role in open source innovation for two decades," said Mike Milinkovich, Executive Director of the Eclipse Foundation. "Their decision to join as a Strategic Member reflects the growing importance of open collaboration in supporting global regulatory compliance efforts, strengthening open source infrastructure, securing supply chains, and advancing the next generation of AI-integrated developer platforms."

The Eclipse Foundation continues to see explosive growth as adoption accelerates across AI-integrated developer tooling and cloud development environments. The Open VSX registry now scales to meet massive global demand:

300 million+ downloads per month
200 million requests during peak daily traffic
12,000+ hosted extensions from over 8,000 publishers.

Unlocking TPU performance: Deep kernel profiling with XProf

2026-06-08T11:30:00.000-07:00

by Yogesh SY, AI Infra Google

Unlocking TPU performance: Deep kernel profiling with XProf

As machine learning workloads scale to unprecedented heights, developers are increasingly writing highly specialized Tensor Processing Unit (TPU) kernels using frameworks like Pallas, Mosaic, and Triton to maximize hardware performance.

However, customizing high-performance kernels has historically introduced a major engineering challenge: optimization blind spots. To legacy performance profilers, custom compilation paths appear as opaque execution paths. Developers are left with single, massive execution blocks in their trace captures, lacking granular visibility into what is actually occurring inside the chip's internal components. Did a vector processing instruction stall? Was matrix math idle due to data loading bottlenecks?

Traditional profiling relies heavily on compile-time static cost models to estimate kernel efficiency. While helpful for standard operations, these models cannot capture dynamic runtime realities like instruction execution stalls, memory subsystem congestion, or hardware scheduling conflicts.

To open this opaque execution path, we are excited to introduce the Kernel Profiling suite in XProf—a low-level hardware debugging suite engineered specifically for Pallas kernel authoring and optimization on Google TPUs. By combining static compilation tracking with dynamic, sub-microsecond hardware telemetry, XProf Kernel provides the deep transparency required to optimize high-scale ML workloads.

Deep visibility: HLO Graphs & MLIR Inspection

The first step in debugging any custom kernel is understanding how your high-level code is translated by the compiler. When compiling a JAX or PyTorch model, the compiler generates a High-Level Optimizer (HLO) graph. Previously, custom calls inside these graphs remained completely obscured.

XProf's updated Graph Viewer resolves this by exposing the internal compilation logic of these custom regions directly. To unlock this deep visibility, developers must pass the appropriate debug flags to the XLA compilation environment.
--xla_enable_custom_call_region_trace=true
--xla_xprof_register_llo_debug_info=true

Once these flags are active, any trace captured via XProf includes comprehensive compiler metadata. In the XProf Graph Viewer, clicking on a custom-call block reveals an interactive panel titled "Custom Call Text." This displays the raw, lowered MLIR (Multi-Level Intermediate Representation) code generated by the compiler.

Figure 1: XProf interface displaying an HLO graph, with a "Custom Call Text" panel to reveal raw MLIR code

By displaying the MLIR text side-by-side with high-level source-code representations, developers can immediately verify whether the compiler is correctly fusing operations and structuring memory tiles as intended.

Tracing Instrumented Low-Level Operations (LLO) Analysis

To provide cycle-level execution visibility, XProf exposes Low-Level Operations (LLO) bundle data directly inside the Trace Viewer. An LLO bundle represents the actual machine instructions issued to the TPU core's functional units during every clock cycle.

Through dynamic instrumentation, XProf inserts hardware markers exactly when a LLO bundle region executes. Within the Trace Viewer, this manifests as dedicated, time-aligned execution tracks representing the TPU bundle's slot utilization metrics from static analysis:

MXU (Matrix Multiply Unit): Tracks active, busy cycles of high-throughput matrix-multiplication pipelines.
Scalar and Vector ALUs: Displays the execution profile of mathematical operations, letting you spot pipeline imbalances.
Vector Fills, Loads, Spills, and Stores: Exposes HBM-to-register data movement, critical for identifying bandwidth-throttling bottlenecks.
XLU (Cross-Lane Unit): Monitors collective communications and data shuffling across physical TPU cores.

Figure 2: XProf Capture Profile trace viewer interface showing dynamic hardware execution tracks

Runtime Performance Counter Sampling

While static analysis effectively verifies instruction counts or vector store logic, it remains detached from the dynamic realities of runtime execution. To bridge this gap, XProf introduces fine-grained, periodic performance counter sampling—available starting with TPU v7 (Ironwood). This capability empowers developers to move beyond static estimation and measure precisely how hardware blocks are utilized in real-time, providing the empirical ground truth needed to identify whether compute units are truly active or stalled by memory subsystems.

Consider the optimization of a tiled matrix multiplication (Matmul) kernel. While a static trace might indicate a logically perfect sequence of operations, real-world performance often falters if the Matrix Multiply Unit (MXU) sits idle while awaiting data from High-Bandwidth Memory (HBM). To diagnose and resolve such bottlenecks, developers can utilize a structured three-step profiling workflow:

Set up the Profiling Environment: Configure the TPU v7 (Ironwood) runtime by defining specific hardware counters—such as scalar issues or synchronization waits.
Capture a Kernel Profile: Use the XProf request interface to capture fine-grained performance counters, which can then be visualized as a time-series within the Trace Viewer.
Interpret the Data: Analyze the resulting counters to distinguish between a Memory-Bound Scenario (characterized by massive spikes in sync_wait) and an Optimized Scenario. For instance, implementing triple buffering to overlap memory loads with MXU compute can reduce runtime from 125.5µs to 88µs—a ~30% performance gain validated by a drastic reduction in synchronization events.

By shifting from static code inspection to empirical runtime telemetry, hardware behavior explicitly validates optimization strategies, ensuring every cycle on the silicon is spent productively. For a hands-on example to check out these techniques, please explore our Pallas Matmul w/ Perf Counters demo.

Figure 3: XProf timeline highlighting a comparison between a detailed "Runtime Perf Counter" section sampling at a 1-microsecond frequency and a "Static LLO Region" track below it

Visualizing the "Utilization Gap"

This dynamic tracking exposes the significant gap left by traditional static analysis tools. A static tool analyzes instructions linearly, completely ignoring time. It might flag an MXU instruction block as "100% Utilized."

In contrast, XProf plots actual hardware execution over time. You might discover that a long-running Scalar ALU operation is stalling the entire execution pipeline, leaving the powerful MXU completely idle. By visualizing these temporal idle gaps, developers can adjust data shapes, memory alignments, and instruction sequencing to maximize compute density.

STATIC ESTIMATION:
[========== Block Execution: MXU Flagged 100% Utilized ==========]

XPROF REAL-WORLD TIMELINE:
├─ [Scalar ALU (Active)] ─┼─ [MXU (Active)] ─┼── [MXU (Idle / Memory Stall)] ──┤
│ Stalling pipeline...     │ Compute phase     │ Starved; waiting for HBM Load    │

Figure 4 : The UI shows the active TPU Core functional unit tracks (MXU, Scalar ALU, Vector ALU, and memory data pipelines) aligned side-by-side with the active framework Ops, exposing exact execution times and real-time idle cycles.

Overall Utilization from Performance Counters

Navigating profiling metrics can be daunting. Relying on metrics calculated via compile-time cost models often misrepresents performance when applied to custom compilation paths. To solve this, XProf establishes a clear Hierarchy of Trust:

                  ┌───────────────────────────────┐
                  │     Absolute Ground Truth     │
                  │  (HBM, Hardware Registers,    │ (100% Trustworthy)
                  │       TPO Metrics, CSRs)      │
                  └───────────────┬───────────────┘
                                  ▼
                  ┌───────────────────────────────┐
                  │       Estimated Metrics       │
                  │   (Program Optimal FLOPs,     │ (Requires caution with
                  │      Goodput Efficiency)      │  custom compiling paths)
                  └───────────────────────────────┘

Figure 5: Hierarchy of Metrics

The Absolute Ground Truth (100% Trustworthy): Metrics derived directly from physical hardware registers (HBM utilization, TPO metrics, unprivileged hardware stats). When profiling custom kernels, these represent physical reality and should be your primary optimization anchors.
Estimated Metrics (Use with Caution): Metrics like "Compared to program optimal FLOPS" or "Goodput efficiency" rely on XLA cost models. Because custom compilation paths bypass standard passes, these metrics can be highly skewed or outright non-functional.

For the unvarnished truth, XProf exposes the Perf Counters View, providing direct, tabular access to over 16,000 raw hardware counters read straight from the TPU silicon.

Figure 6: XProf Perf Counters Tabular View

Understanding Trace Tracks: The height of a trace track does not represent a normalized 0-100% percentage. It represents the maximum raw counter value observed in that interval. For example, if a counter increments by 100 cycles over a 500-nanosecond trace window (roughly 1,000 clock cycles on a 2.0 GHz core), it indicates exactly 10% physical utilization of that unit.

To configure and profile the runtime performance counters sampling method, please follow the instructions from OpenXLA Kernel Profiling Instructions.

Advanced Sampling: Event-Triggered Profiling

Previously, dynamic capturing was limited to Periodic Sampling Mode—polling counters based on a host-level timer, which hit a physical resolution floor of 1 microsecond.

           CORE 0           CORE 1           CORE 2           CORE 3
      ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
      │  28 Counters │ │  28 Counters │ │  28 Counters │ │  28 Counters │
      └──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘
      └─────────────────────────────────────────────────────────────────┘
                            4 x 28 Sparse Matrix

Figure 7: Sparse Matrix Configuration

To capture lightning-fast hardware cycles, XProf now supports External Event-Triggered Mode. The dynamic sampler intercepts physical TPU trace instructions and boundary triggers (such as entering/exiting custom call scopes), allowing for sub-microsecond capture latency and precise attribution.

Developers can configure up to 28 hardware counters per core, distributed across up to four active SparseCores, creating a 4 x 28 profiling matrix that maximizes data variety while protecting workload performance.

Activating this is straightforward via standard JAX JIT profilers:

options = jax.profiler.ProfileOptions()

# Example request for externally triggered collection
options.advanced_configuration = {
"tpu_enable_periodic_counter_sampling" : True,
"tpu_tc_perf_counter_sampling_options" : (
          'is_external_trigger:true scaling:0 counter_size_bits:1 indices:10 indices:11 indices:56 indices:57 indices:58'
),
}

# For periodic sampling, please use interval_us instead of is_external_trigger.

Getting Started

Ready to transition from guessing performance to measuring and optimizing the physical limits of your ML silicon? Explore these open-source resources to get started with XProf Kernel today:

XProf GitHub Repository: github.com/openxla/xprof
Official XProf Documentation: openxla.org/xprof
JAX Profiling Guide: jax.readthedocs.io/en/latest/profiling.html

Journey to JPEG XL: How open source experiments shaped the future of image coding

2026-06-03T11:30:00.000-07:00

by Jyrki Alakuijala, Zoltán Szabadka & Luca Versari, Paradigms of Intelligence, Google Technology & Society

Building the Next Generation Image Standard

The internet runs on images. Since the early days of the web, there has been a relentless tension between visual fidelity and bandwidth. For decades, the industry relied on the venerable JPEG standard for images loading fast. It served us remarkably well, but as displays moved to High Dynamic Range (HDR) and Wide Color Gamut (WCG), the format began to show its limits.

The road to JPEG XL (JXL) wasn't a straight line. It was a decade-long exploration, creating a series of milestone projects testing radical ideas in psychovisual modeling, entropy coding, and optimization. Today, as JPEG XL sees rapid adoption across operating systems and professional standards, we’re looking back at the experiments that made it possible.

The Early Foundation: 2011–2017

Our study began with a focus on understanding the limits of existing technology. We didn't start by trying to write a new standard; we started by trying to make the current ones better, and learning their limitations. This allowed us to make the new formalism more flexible and efficient in the right places.

WebP Lossless and Brotli: Lossy WebP drew its lineage from video technology, the WebP Lossless (2011) represented an architectural and scoping departure. We debuted the entropy image concept, an innovative method utilizing a secondary image to orchestrate the selection of static entropy codes for the primary visual data. We reapplied this approach later with data-driven context modeling in the Brotli compression format, enabling rich context modeling without slowing decoding.
Butteraugli: Around 2014, we realized that raw mathematical compression (PSNR) wasn't enough, and simple psychovisual approximations (SSIM and similar) failed in color-rich environments. We built Butteraugli and the XYB color space to mimic the human visual system's edge detection and opponent-color processes in varying scale, allowing us to compress images more effectively.
We pushed the legacy JPEG 1 standard (ISO/IEC 10918, introduced in 1992) to its absolute limits through two key projects: Guetzli and Brunsli. These initiatives provided invaluable insights into the strengths and limitations of traditional JPEG compression methods. Guetzli (2016) is a slow high-density perceptual encoder that used Butteraugli to find the optimal quantization tables, pushing legacy JPEGs to be 20-30% smaller. Brunsli (2015) meanwhile, focuses on lossless recompression, allowing users to repack existing JPEGs into a smaller footprint without losing a single bit of original data. After finishing with JPEG XL standardization, we returned to Guetzli's scope in 2024 and made the encoding much faster and HDR-compatible in Jpegli.

The feedback from these launches, ranging from the technical details of WebP Lossless to the psychovisual audits of Guetzli, proved indispensable. While we already targeted the highest visual fidelity, feedback from detail-critical e-commerce helped us to refine the requirements.

The Convergence: 2017–2019 PIK Era and the 2019 FUIF Integration

By 2017 we had powerful separate tools and it was time to fuse them. In open sourcing PIK we combined the efficiency of Brunsli with the psychovisual optimizations of Guetzli. Further, PIK introduced a real adaptive quantization field and other optimizations. PIK formed our proposal to the ISO standardization body. The committee's final call for proposals pushed toward extreme density, requiring bit rates as low as 0.06 BPP, equivalent to 35 times the compression of internet-quality images and 80 times that of camera output. This expansion of scope necessitated a significant complexification of the format and the encoder, leading to the Variable-block-size Discrete Cosine Transform (VarDCT) architecture that remains central to JPEG XL today.

We proposed to merge our PIK proposal with the FUIF (Free Universal Image Format) proposal from Cloudinary. PIK used Brotli-style distribution selection at encoding time, while FUIF refined codes incrementally during decoding. The final JPEG XL standard became a best-of-both-worlds compromise: we used PIK's faster-to-decode distribution selection with FUIF's sophisticated context trees. The merger represented a departure from conventional one platform driven standardization, and prioritized technical synergy and collaboration.

JPEG XL Today: An Ecosystem Takes Root

JPEG XL's efficiency, psychovisually-optimized quality, file size, and coding speed, are being noticed. We are seeing bottom-up adoption in various industries, the most demanding fields are leading the way. Because of its ability to handle high bit-depth, high quality and even lossless data efficiently and robustly, JPEG XL has become foundational in several fields:

Photography: Used in Digital Negative (DNG 1.7), Apple's ProRAW, and others.
Medical: Adopted by DICOM, the international standard for medical images.
Publishing: Integration into future versions of the PDF and EPUB standards.

The ecosystem has been maturing rapidly. Adobe's photography software, Apple's iOS, macOS, and visionOS have native support, as do Linux distributions like Ubuntu and Microsoft's JPEG XL Image Extension for Windows. Our libjxl-tiny inspired Shikino High-Tech, Inc. and CAST to release the first commercial JPEG XL encoder IP core for ASIC and FPGA designs, aimed at real-time, low-power image capture. Safari (2023) led among major browsers, while Firefox and Chrome currently maintain experimental support.

JPEG XL design was not only countless hours of optimization, experimentation and eye-balling the results, but also creative discussions at a whiteboard. In this Gemini-reconstructed scene, Luca Versari and Jyrki Alakuijala (left-to-right) debate VarDCT block selection heuristics.

Looking Forward

The story of JPEG XL stands as a testament to the efficacy of long-horizon planning validated by intermediate functional milestones—with minimum-viable prototypes like Guetzli and practical tools like Brunsli and Brotli—that invite feedback from the open-source community. A small research team can innovate by crystallizing solutions through quick iterations, with thousands, if not tens of thousands, of experiments in psychovisual modeling, entropy, coding speed and complexity, and the entire industry can eventually navigate toward a more efficient, beautiful future.

We started by trying to squeeze a few more bytes out of a 1992 JPEG 1 standard; with JPEG XL we hope to have established a foundation for digital imaging that can last for the next three decades.

Announcing Apache Iceberg 1.11.0

2026-05-27T11:30:00.000-07:00

by Alex Stephen & Talat Uyarer, Lakehouse

Apache Iceberg project has just launched version 1.11.0! A lot has happened since the last version.

Iceberg 1.11.0 adds support for Apache Spark 4.1 and Apache Flink 2.1, the latest releases of the two engines and makes both the default build targets

The rest are more structural. The REST catalog learns to plan scans server-side, shifting metadata work off the query engine. A new partition statistics scan API gives optimizers a clean, supported way to read a table's shape. Built-in table encryption arrives with envelope encryption and Google KMS support. And Google Storage Analytics library integration makes your Iceberg workloads faster than before.

Let's take a look at some of the biggest changes.

Spark & Flink Updates

As Spark and Flink are moving forward, the 1.11.0 release is pushing forward for new version support in both.

Spark 4.1 & DSv2 Migration: Spark 4.1 unlocks is MERGE INTO with automatic schema evolution: Spark's newer MERGE syntax accepts a WITH SCHEMA EVOLUTION clause, so a MERGE whose source carries columns the target table lacks can add those columns to the table within the same statement, with no separate ALTER TABLE round trip. Beyond the version bump, the 1.11 Spark connector also modernizes against Spark's newer DataSource V2 APIs and adds an asynchronous micro-batch planner that speeds up Structured Streaming.
Flink Ecosystem Updates: Initial work for Flink 2.1 support has landed in the core repository, continuing Iceberg's promise of providing first-class, low-latency streaming sink capabilities. The centerpiece of the Flink work is the DynamicIcebergSink, an experimental sink that breaks the old one-sink-per-table model: a single sink routes each record to a table chosen at runtime, creating tables on demand and evolving their schemas and partition specs on the fly as the input changes including dropping columns once you opt in with dropUnusedColumns. In addition to DynamicIcebergSInk work Flink started supporting nanosecond, variant and unknown types from V3 Spec.

Server-side scan planning

In previous versions of Iceberg, the client handled the heavy lifting of scan orchestration. The driver of engine would traverse the table's metadata tree, retrieving manifest lists and files from object storage to filter data against specific partition requirements. Iceberg 1.11.0 shifts this computational burden into the catalog through server-side scan planning.

Instead of manually traversing manifests, the engine submits a single POST …/plan request detailing the scan allowing the REST catalog to return optimized FileScanTasks.

The API is designed to handle data at any scale: smaller scans return immediate results, extensive operations return a plan-id for polling, and massive datasets are retrieved via parallel plan-tasks through POST …/tasks.

Planning moves off the query engine and into the catalog — the driver no longer touches metadata in object storage.

Built-in table encryption

As data lakes increasingly serve as the central hub for sensitive PII and financial data, relying solely on bucket-level storage encryption is no longer enough. Iceberg 1.11.0 introduces built-in table encryption, bringing fine-grained, KMS-backed security directly to the table level.

This provides data platform teams with robust capabilities for security and compliance:

Zero-Trust Storage Security: Even if a malicious actor gains direct access to your underlying object storage bucket, the data remains completely unreadable.
Total Index Protection: It isn't just the raw data that is protected; Iceberg encrypts the manifest lists as well, preventing attackers from inferring sensitive information from table statistics.
Tamper-Proof Data: Built-in authentication tags guard against unauthorized modifications, ensuring data integrity.
Effortless Key Rotation: Keys are rotated automatically as they age, satisfying strict compliance mandates without requiring you to rewrite massive datasets.

Iceberg achieves this using envelope encryption with a three-tier key hierarchy. A table master key lives securely in your KMS and never touches Iceberg storage. This master key wraps key-encryption keys (KEKs), which are stored safely inside the table metadata. Finally, each KEK wraps a unique, per-file data-encryption key (DEK).

Every data file and manifest list is then encrypted with AES-GCM under its own unique DEK. This decoupled architecture ensures maximum security while maintaining the high performance expected of Iceberg workloads.

File Format API

Historically, Iceberg's format-handling code was tightly coupled, growing organically around Parquet, Avro, and ORC. Adding a new format or enforcing consistent feature support (like V3 default values or new column types) across all formats meant duplicating complex engine-specific switch/case code paths.

Iceberg 1.11.0 introduces the finalized File Format API, bringing a consistent API to reading and writing all of these file formats.

Instead of hardcoded engines handling binary extraction, the architecture introduces:

FormatModel: A standardized implementation defining how a file format handles reader/writer construction and its specific capabilities.
FormatModelRegistry: A central directory where query engines fetch appropriate read and write builders.

This API (which is already seeing adoption around other Apache Iceberg implementations) provides a significant code cleanup for the future of the project. It also opens the door for more file formats as time goes on.

Moreover, this new interface facilitates the implementation of Column Families, enabling vertical partitioning of storage. This advancement lets teams perform targeted updates or rewrites on isolated columns—such as recalculating vector embeddings—while leaving the remaining table data undisturbed.

SQL UDF Specification

1.11.0 includes the SQL UDF specification, which adds a brand new metadata format for both Scalar and Table Functions:

Immutable Versioning and Rollback: UDF metadata is written as self-contained, versioned JSON files stored right in the object store. If a data engineer deploys a buggy UDF update, administrators can execute an atomic rollback to a previous version log state
Standardized Schema Typings: Parameters and return types map cleanly to Iceberg Type JSON representations, directly accommodating complex nested maps, structs, and the upcoming Iceberg V3 variant type.
Engine Specific Execution: Each SQL UDF has a function implementation for each engine, allowing users to leverage engine-specific functionality in their UDFs.

Google Analytics Library Integration

For Google Cloud customers, version 1.11.0 delivers substantial throughput gains by embedding the GCS Analytics Core library into GCSFileIO (Issue #14326, PR #14333).

This integration introduces Footer Prefetching, which optimizes Parquet length checks by caching object suffixes to remove network overhead. Combined with threaded VectoredIO for concurrent multi-range operations and specialized small object caching for sub-1MB files, these enhancements eliminate persistent I/O bottlenecks. Initial benchmarks indicate that these architectural improvements can reduce Parquet metadata parsing latency and boost total record processing speeds, empowering high-scale Spark, Flink, and Trino workloads to run with improved efficiency on Google Cloud Storage.

Getting Started with 1.11.0

We are excited to be part of the Apache Iceberg community and innovating together. As a compliant Iceberg REST Catalog, Lakehouse for Apache Iceberg (formerly BigLake) already has support for version 1.11.0.

To upgrade your environment, update your build dependencies to version 1.11.0. Remember to review your deployment runtimes to ensure compatibility with the new JDK 17 baseline, and test your workloads if you are transitioning from Spark 3.4.

For a full breakdown of every bug fix, contributor attribution, and dependency bump, check out the official Apache Iceberg Releases Page

TestParameterInjector introduces an idiomatic Kotlin API

2026-05-25T11:30:00.000-07:00

by Jens Nyman, TestParameterInjector Team

In March 2021, we announced the open source release of TestParameterInjector: a simple but powerful parameterized test runner for JUnit4. In September 2022, we followed up with JUnit5 support, bringing our framework to developers who had moved on to the Jupiter API.

We're excited to announce our biggest update yet for our Kotlin users: KotlinTestParameters.

The de facto standard for parameterized testing

When we first introduced TestParameterInjector, we shared a graph showing its rapid adoption within Google. Over the past few years, that trajectory has continued to a point where TestParameterInjector is the de facto parameterized test framework.

Usage of all other alternative frameworks continues to steadily decline, while TestParameterInjector's adoption keeps growing rapidly. It has fundamentally lowered the barrier to writing data-driven unit tests, empowering Googlers and open source developers alike to maximize test coverage with minimal boilerplate. We believe its ubiquity internally is a strong testament to its reliability and utility for the broader developer communities.

The Kotlin challenge

As Kotlin's popularity has surged, developers have naturally been writing more of their TestParameterInjector tests in Kotlin. However, specifying explicit test values in Kotlin historically meant falling back to Java-centric paradigms.

If you wanted to provide specific values to a test, you typically had three options, none of which felt truly idiomatic in Kotlin:

@TestParameter({"123", "456"}): This relies on string arrays, limiting you to a subset of types that the string parsing supports.
@TestParameters: This allows for more complex sets of data, but relies on YAML strings (e.g.,"{age: 17, expectIsAdult: false}"). These strings however are not type-safe, and are completely ignored by IDE refactoring tools.
Provider classes: For complex types that couldn't be easily represented in strings, you have to write Provider classes, adding a bit of boilerplate code and indirection.

Enter KotlinTestParameters

To bring a seamless experience to Kotlin, we are introducing a significant new Kotlin-only feature: KotlinTestParameters.

By leveraging Kotlin's default function arguments, you can now define parameterized tests in a fully type-safe, concise, and refactor-friendly way using the testValues() function (and friends).

Here is what it looks like in practice:

import com.google.testing.junit.testparameterinjector.TestParameterInjector
import com.google.testing.junit.testparameterinjector.TestParameter
import com.google.testing.junit.testparameterinjector.KotlinTestParameters.testValues
import com.google.testing.junit.testparameterinjector.KotlinTestParameters.namedTestValues
import org.junit.Test
import org.junit.runner.RunWith

@RunWith(TestParameterInjector::class)
class MyTest {

  // Testing simple types directly
  @Test
  fun simpleTest(@TestParameter limit: Int = testValues(20, 100)) {
    // This test method is run twice: once for limit=20 and once for limit=100
  }

  // Testing complex types without YAML strings or Provider classes!
  data class TestCase(val age: Int, val expectIsAdult: Boolean)

  @Test
  fun complexTest(
    @TestParameter testCase: TestCase = namedTestValues(
      "teenager" to TestCase(age = 17, expectIsAdult = false),
      "young adult" to TestCase(age = 22, expectIsAdult = true)
    )
  ) {
    // This test method is run twice with fully typed data class instances
  }
}

Why we recommend making the switch

Because testValues() seamlessly integrates with Kotlin's language features, any type is supported. This completely eliminates the need for stringly-typed YAML maps or verbose Provider classes.

We firmly believe that KotlinTestParameters is a massive leap forward in readability and maintainability. Moving forward, this should be the default way of specifying test values for all new Kotlin tests, replacing the older @TestParameters, @TestParameter({"..."}), and Provider class patterns.

Try it out!

You can read more and start using KotlinTestParameters today over on our GitHub repository.

Let us know what you think on GitHub if you have any questions, comments, or feature requests!

Disrupting the presentation layer using autonomous workflows

2026-05-21T11:30:00.000-07:00

by Adrian Chung, Abdelfettah Sghiouar & Matt Larkin, Google Kubernetes Engine

Empowering every engineer to do more with Kubernetes

Kubernetes is the gold standard for container orchestration. Its power, flexibility, and rich API surface are exactly why it has become the foundation of modern cloud-first infrastructure. Today, engineers express that power through the K8s API, declarative YAML manifests, and cloud consoles, a remarkably expressive toolbox.
We believe the next step is to expand how engineers interact with that toolbox. A Kubernetes expert should be able to converse with a deep-domain peer that speaks fluent control-plane and can reason about cluster state in real time. An engineer who isn't a Kubernetes specialist should be able to express higher-order intent, such as "deploy my application," "rebalance this workload" and have it carried out safely against the same powerful APIs. Both audiences get more leverage out of the platform they already trust.
This is the vision behind Kube-Agents: a system of intelligent, autonomous, and human-in-the-loop agents that act as a new, intent-driven presentation layer for Kubernetes. We are moving from declarative intent via API to higher-order, human intent-driven operations while preserving everything that makes Kubernetes great underneath.

The Vision: Expanding the Presentation Layer

Today, engineers do impressive work stitching together metrics, alerts, and multi-step commands to keep clusters healthy. Agents extend that work, not replace it. By complementing existing interfaces with autonomous agents, engineers can choose the level of abstraction that fits the task: drop down to kubectl and YAML when precision matters, or describe intent in plain language when speed and clarity matter more. The agents continuously observe system state and can execute complex operations in real time on the engineer's behalf. This isn't about hiding Kubernetes. It's about giving every engineer a more capable collaborator on top of it.

Meet the Agents: A Specialized Team

Our architecture is currently built upon three core specialized agents, each acting as a new kind of intent-driven collaborator for different stakeholders:

The Platform Agent
- Role: A partner for the central governance layer, your management plane.
- Focus: Codifying best practices and keeping platform blueprints evergreen and synchronized across the entire fleet.
- Example: When a new egress policy is defined at the org level, the Platform Agent propagates it to the Dev Team agents and confirms enforcement, giving platform teams confidence in compliance while letting developers stay focused on their applications.
The Cluster Operator Agent
- Role: A trusted teammate for your infrastructure operators.
- Focus: Global concerns like multi-cluster balancing, automated provisioning, security patching, and zero-downtime version upgrades.
- Example: It can detect a degrading node and proactively migrate workloads before application latency spikes, expanding what a single operator can safely manage at scale.
The Development Team Agent
- Role: A production-savvy peer for developers.
- Focus: The primary collaborator for developers. It supports the full workload lifecycle — reconciling manifest drift, right-sizing resources, and assisting with real-time debugging.
- Example: When a developer asks "Why is my service failing?" in chat, the agent responds with relevant logs, correlated metrics, and a diagnosis of recent config changes — meeting a Kubernetes expert at depth and meeting a less specialized developer at intent.

Leveraging Industry Benchmarks

DevOps Bench is a comprehensive suite of benchmarks. These specialized agents learn from those results so they are equipped with the context to make well-reasoned decisions when autonomously supporting infrastructure work.

The First Demo

To be truly useful at the presentation layer, these agents can't be short-lived request/response scripts. They need to be persistent, long-running "team members" capable of continuous learning and collaboration.
As a first step, we've launched a set of workspaces compatible with OpenClaw for a demo, installable into your OpenClaw environment, leveraging existing out-of-the-box capabilities around identity, storage persistence, and memory. The agents included are: the Platform Agent, the Cluster Operator Agent, and the Development Team Agent.

Autonomous GitOps & JIT Probing (Dev Team Agent): Demonstrates prompt-driven staging deployments and dynamically generated, context-aware probers. The agent adheres strictly to GitOps workflows by opening PRs for infrastructure updates (such as node failure tolerance) and actively prevents configuration drift by reconciling manual manifest edits upon merge.
Self-Healing Infrastructure (Dev Team Agent): Showcases automated troubleshooting when a manifest is deployed with an image name typo. The agent executes a complete, autonomous 5-step remediation loop—Notification, Learning, Recommendation, Mutation, and Validation—to detect, fix, and verify the deployment without human intervention.
Multi-Agent Governance & Policy Coordination (Cluster Operator & Dev Team Agents): Highlights cross-agent negotiation when the Cluster Operator attempts to downscale underutilized resources for cost savings. The Dev Team Agent steps in to enforce minimum capacity policies, successfully prioritizing application reliability and governance over financial savings.

Going forward, we will further productize this pattern, building on open standards to define agents and their capabilities (AGENTS.md, skills, MCP), and provide an out-of-the-box harness to orchestrate these agents.

Redefining the Stack

An intent-driven presentation layer is just the beginning. With an agentic interface in place, we can keep evolving the underlying infrastructure — adopting new components or integrating directly with additional infrastructure APIs — while engineers continue to interact with the system the way they already do. The interface stays intent-driven and stable; the agents adapt to the evolving stack underneath, so investment in how teams work today carries forward.

Call to Action

We're building Kube-Agents in the open because we believe the best infrastructure solutions are built collaboratively. Our goal is to use our expertise to give back to the open source community, while also actively learning from the ecosystem's real-world challenges. By working together, we can define best practices that benefit everyone.
If you're interested in helping shape the future of Kubernetes management, check out the Kube Agents repo.

We are seeking engagement on two fronts:

Share your use cases: What would you most like an autonomous teammate to help with? We want to learn from your unique operational needs—whether it's multi-cluster balancing, specific debugging scenarios, or policy enforcement—to ensure we're building tools that provide real leverage.
Define future roles: What new specialized agents should exist? We value your input on the roles these agents should fulfill to best serve diverse team structures and operational requirements.

Join the conversation, contribute your ideas, and help us build a self-driving cloud that works for everyone. Check out the Kube Agents project to open an issue or start a discussion.

Google Open Source Blog

This Week in Open Source for October 2, 2026

This Week in Open Source for October 2, 2026

Upcoming Events

🗓️ October 2026

🗓️ November 2026

Open Source Reads and Links

Standardizing fine-grained access control in Apache Iceberg's REST catalog

The problem: there is no server in the read path

The mechanism

The nine masking actions

Fail-closed by design

The trust model, stated plainly

What is still missing

Why this matters

New public datasets available in Google Cloud Lakehouse

Prerequisites

Explore metadata with PyIceberg

PySpark and Managed Service for Apache Spark

Start building today

This Week in Open Source for September 24, 2026

This Week in Open Source for September 24, 2026

Upcoming Events

🗓️ October 2026

🗓️ November 2026

Open Source Reads and Links

Google Cloud: Investing in the future of PostgreSQL — 2026 highlights

Technical contributions: January 2026 – September 2026

1. Logical replication and conflict management

2. Core engine performance and administrative enhancements

3. Core engine access control, collation, and indexing bug fixes

4. PostgreSQL ecosystem and extension hardening

Related reading

Community roadmap: Your feedback matters

Acknowledgments

Reconnecting with the heart of open source: Highlights from our 2026 GSoC India tour

Community stories: Learning to think and lead as an engineer

Open source mentorship in the age of AI

What’s next?

How much should you trust your OSS data?

Securing the agentic era: Introducing formal verification for CEL

Proving rules from the ground up

1. Catching logic bugs in simple expressions (Equivalence)

2. Enforcing exhaustive guardrails (Validity)

3. Guaranteeing security invariants with CEL Policy

Under the hood: High-fidelity mathematical modeling

Zero false positives via three-pass taint tracking

Deep structural extensionality

The mandatory bridge of trust

Google joins the OpenROAD Initiative as principal member to accelerate open source silicon innovation

Driving long-term open source sustainability

Leadership perspectives

About the OpenROAD Initiative and OpenROAD project

Adapting open source practices to an AI-first world: A retrospective on 2025

Working together on emerging standards

Launching tools with transparency

Adjusting our collective security practices

Working together to sustain the contributor community

Evolving the way we work in open spaces

Appendix: About this data

This Week in Open Source for July 16, 2026

This Week in Open Source for July 16, 2026

Upcoming Events

Open Source Reads and Links

Google Cloud: PostgreSQL community contribution updates

PGConf.dev 2026

PGConf India 2026

PGDay Paris & PGDay France 2026

PGDay FOSDEM 2026

PGConf Belgium 2026

Nordic PG Day 2026

Swiss PGDay 2026

Postgres Conference: 2026 San Jose

Community Leadership and Committees

Looking Forward

Acknowledgement

Community feedback: How can corporations improve support for open source maintainers?

Paying maintainers takes creativity

Manage and respect expectations

Let's keep learning as a community