Target audience: Data engineers migrating Azure cloud data platforms; technical architects evaluating ADF-to-Databricks strategies
Estimated read time: 12 minutes

The Problem

Your organization runs dozens of Azure Data Factory (ADF) factories containing hundreds of pipelines. Many of those pipelines read from or write to Azure Synapse Analytics (formerly SQL Data Warehouse) — and the decision has been made to migrate that workload to Databricks Unity Catalog.

A manual approach would require teams to open each pipeline in the ADF portal, replace activities, test, repeat. At 500+ impacted activities across 37 factories, that’s months of tedious, error-prone work.

We took a different approach: treat ADF pipeline JSON as a parseable AST, and build a compiler-like engine that rewrites it programmatically.

This post walks through the architecture, the key technical decisions, and the patterns that made it work.

Architecture Overview: A Two-Phase Pipeline Compiler

The framework operates in two distinct phases, orchestrated by a single parameterized notebook:

Phase 1: Normalize (“Load Clean”)

ADF’s REST API export format is not the same as the ADF Git/authoring format that the portal and CI/CD tooling expect. Before migration, we normalize every pipeline JSON through four transformations:

1. Strip ARM envelope fields — id, type, etag are Azure Resource Manager metadata, not pipeline logic
2. Convert snake_case keys to camelCase — the REST API returns linked_service_name; ADF authoring expects linkedServiceName
3. Nest activity-specific fields under typeProperties — the API export flattens them; authoring format nests them
4. Wrap top-level fields under properties — activities, variables, annotations belong inside a properties block

The critical subtlety: Not all keys should be camelCased. ADF pipeline variables, parameters, and stored procedure parameters are user-defined identifiers referenced literally in expressions like @{variables(‘Config_Schema’)}. Converting Config_Schema to configSchema would silently break every expression in the pipeline.

The solution: maintain a set of “user-defined containers” whose immediate child keys are exempt from case conversion:

USER_DEFINED_CONTAINERS = {

"variables",

"parameters",

"globalParameters",

"storedProcedureParameters",

}




def convert_keys_camel(obj, parent_key=None):

if isinstance(obj, dict):

skip = parent_key in USER_DEFINED_CONTAINERS

return {

(k if skip else snake_to_camel(k)): convert_keys_camel(v, parent_key=k)

for k, v in obj.items()

}

elif isinstance(obj, list):

return [convert_keys_camel(item, parent_key=parent_key) for item in obj]

return obj

Phase 2: Migrate

Once normalized, the engine scans every activity in every pipeline, determines whether it references a Synapse resource, classifies the migration pattern, and generates a replacement activity.

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐

│  ADF REST API   │    │   Normalized    │    │   Migrated       │

│  Export (raw)   │──▶│   Pipeline JSON  │──▶│   Pipeline JSON   │

│  snake_case     │    │   camelCase      │    │   Databricks     │

└─────────────────┘    └─────────────────┘    └─────────────────┘

     Phase 1: Load Clean          Phase 2: Activity Migration

Zero-Config Migration: Auto-Discovering the Blast Radius

The first hard question in any migration is: what actually needs to change?

Manually cataloging Synapse-dependent resources across 37 factories, 2,200+ JSON files, and dozens of linked services is a recipe for missed dependencies. Instead, the framework auto-discovers its own migration scope by scanning the ADF export artifacts.

The Discovery Chain

linked_services/*.json          datasets/*.json              pipelines/*.json

        │                            │                           │

        ▼                            ▼                           ▼

  Filter by type:              Match dataset → LS:         Match activity → dataset:

  AzureSqlDW or                "Which datasets point       "Which activities reference

  AzureSynapseAnalytics         to a Synapse LS?"           a Synapse dataset?"

        │                            │                           │

        ▼                            ▼                           ▼

    SYNAPSE_LS[ ]              SYNAPSE_DATASETS[ ]         Migration targets

    LS_TO_SECRET{ }            DS_TO_LS{ }

Step 1: Identify Synapse linked services. Scan linked_services/ for JSON files where properties.type is AzureSqlDW or AzureSynapseAnalytics. While scanning, extract the Key Vault secret name from the connection string definition — this is needed later when generating JDBC parameters for the replacement notebook.

Step 2: Identify Synapse datasets. Scan datasets/ and keep any dataset whose linkedServiceName.referenceName points to a LS from Step 1.

Step 3: Classify pipeline activities. For each activity in each pipeline, check five reference patterns:

This three-step chain means the framework needs zero manual configuration for a new factory — just point it at the ADF export folder and run.

Handling Nested Control Flow: Recursive Activity Tree Walking

ADF pipelines are not flat lists of activities. They contain container activities that nest child activities arbitrarily deep:

• ForEach — typeProperties.activities[]
• IfCondition — typeProperties.ifTrueActivities[] and typeProperties.ifFalseActivities[]
• Until — typeProperties.activities[]
• Switch — typeProperties.cases[].activities[]

A Synapse-dependent Lookup might be three levels deep inside a ForEach → IfCondition → activities array. A flat scan would miss it entirely.

The solution is a recursive descent function that walks every branch of the activity tree:

def migrate_activities(activities, migration_log):

    migrated = []

    for activity in activities:

        is_target, pattern, ref = is_synapse_activity(activity)




        if is_target:

            new_act = apply_migration_pattern(activity, pattern, ref)

            migration_log.append(build_log_entry(activity, pattern, ref))

            migrated.append(new_act)

        else:

            new_act = deepcopy(activity)

            tp = new_act.get('typeProperties', {})




            if 'activities' in tp:          # ForEach / Until

                tp['activities'] = migrate_activities(tp['activities'], log)

            if 'ifTrueActivities' in tp:    # IfCondition

                tp['ifTrueActivities'] = migrate_activities(tp['ifTrueActivities'], log)

            if 'ifFalseActivities' in tp:

                tp['ifFalseActivities'] = migrate_activities(tp['ifFalseActivities'], log)

            if 'cases' in tp:               # Switch

                for case in tp['cases']:

                    if 'activities' in case:

                        case['activities'] = migrate_activities(case['activities'], log)


            migrated.append(new_act)

    return migrated

Key design decision: Non-Synapse activities are deepcopy’ed and passed through unchanged. The engine only touches what it needs to. This means the output JSON is a perfect copy of the input, except for the specific activities that were replaced — a property we validate later.

The Pattern Catalog: Three Migration Strategies for Three Activity Types

Not every Synapse-dependent activity can be replaced the same way. The framework defines a pattern catalog — a mapping from source activity type to target Databricks activity, based on what the activity actually does.

Pattern A: Copy Activity → DatabricksNotebook

Source: ADF Copy activity reading from Synapse via SqlDWSource or AzureSqlSource

Target: A DatabricksNotebook activity that calls a parameterized Generic Ingestion Framework notebook

The replacement notebook uses JDBC to connect to the source database (credentials fetched from Key Vault at runtime), executes the original SQL query, and writes the result using the original sink format.

The sink-type-aware design is the critical detail here. The original Copy activities don’t all write to the same destination format — some write Parquet to ADLS, some write CSV, some write to SQL tables. Simply replacing everything with Delta writes would break downstream consumers.

The framework inspects the original activity’s sink.type and maps it:

SINK_TYPE_MAP = {

    "ParquetSink":       "parquet",

    "DelimitedTextSink": "csv",

    "AvroSink":          "avro",

    "AzureSqlSink":      "delta",

    "SqlDWSink":         "delta",

}

For file-based sinks, it also extracts the output dataset’s container, path, and filename parameters (which may be static strings or ADF dynamic expressions) and passes them through to the notebook.

Pattern B: Lookup Activity → WebActivity (SQL Statement Execution API)

Lookup activities query config tables, watermarks, or CDC flags. They’re lightweight — spinning up a notebook cluster for a single SELECT MAX(watermark) is overkill.

Instead, the framework replaces these with a WebActivity that calls the Databricks SQL Statement Execution API directly:

{

  "type": "WebActivity",

  "typeProperties": {

    "url": "https://<workspace>/api/2.0/sql/statements",

    "method": "POST",

    "body": {

      "warehouse_id": "<sql_warehouse_id>",

      "statement": "<original_query>",

      "catalog": "<target_catalog>",

      "wait_timeout": "30s"

    },

    "authentication": {

      "type": "MSI",

      "resource": "2ff814a6-3304-4ab8-85cb-cd0e6f879c1d"

    }

  }

}

This is cheaper, faster, and doesn’t require a running cluster — the SQL warehouse handles it.

But not all Lookups are read-only. Some contain DML (INSERT, EXEC stored_proc). The framework includes a query classifier that inspects the SQL text — including dynamically generated ADF expressions — to route:

• Read-only SELECT / WITH → WebActivity (Pattern B)
• DML / stored procedure calls → DatabricksNotebook (Pattern C)

Pattern C: Stored Procedure → DatabricksNotebook

Synapse stored procedures have no direct equivalent in Databricks. These are replaced with a DatabricksNotebook activity, and the stored procedure logic itself must be rewritten as SparkSQL or PySpark in a dedicated notebook.

The framework generates the activity scaffolding; the actual logic translation is a separate workstream.

Preserving ADF Expressions Across Migration

ADF pipelines are full of dynamic expressions: @pipeline().parameters.SourceSchema, @activity(‘Lookup1’).output.firstRow.watermark, @concat(…). These are evaluated at runtime by the ADF engine.

The migration framework must handle three scenarios:

1. Static values — pass through as {“value”: “dbo”, “type”: “String”}
2. ADF expressions that still work — pipeline parameters and variable references don’t change when the target activity changes. These are preserved verbatim:

{"value": "@pipeline().parameters.SourceSchema", "type": "Expression"}

3. Expressions that break — The biggest one: Lookup result access.

In native ADF:

@activity('MyLookup').output.firstRow.watermark_value

When the Lookup is replaced by a WebActivity calling the SQL Statement Execution API, the output shape changes. ADF reads notebook output via runOutput, and we wrap the result in a JSON envelope:

@json(activity('MyLookup').output.runOutput).firstRow.watermark_value

For WebActivity replacements, the output is the raw API response, requiring a different access pattern:

@activity('MyLookup').output.result.data_array[0][0]

The framework detects which downstream activities reference the migrated Lookup and annotates the migration report with the required expression changes. This is one area where full automation gives way to guided manual review — the combinatorial space of downstream expression patterns is too large to safely auto-rewrite.

Trust But Verify: A Five-Point Validation Suite

Automated migration is only valuable if you can trust the output. The framework runs five validation checks after every migration:

1. Passthrough integrity. For pipelines with zero Synapse dependencies, the output JSON must be byte-identical to the input. Any difference means the normalization or deepcopy logic introduced a bug.
2. File count verification. The all/ output folder must contain exactly as many files as the source. The impacted/ folder must match the number of pipelines that had at least one activity migrated.
3. Linked service reference check. Every migrated activity (identifiable by the [MIGRATED from Synapse] description tag) must reference the Databricks linked service. If any still point to the old Synapse LS, the replacement logic has a gap.
4. Synapse dataset removal. Scan all migrated pipelines for any remaining references to Synapse datasets. A DatabricksNotebook activity should never reference a Synapse dataset in its inputs/outputs.
5. False positive detection. Verify that only Synapse-linked activities were migrated. If an activity referencing an ADLS linked service or a REST API was converted, the detection logic is too aggressive.

These five checks give us a green/red signal per factory, and they run in seconds. In practice, checks 3-5 caught real bugs during development — edge cases where the LS reference was at the activity root instead of inside typeProperties, or where a dataset name appeared in both Synapse and non-Synapse contexts.

One Notebook, 37 Factories: Parameterization at Scale

The framework is a single notebook with widget parameters:

Changing factory_name is all it takes to migrate a different factory. The auto-discovery chain rebuilds the Synapse LS list, dataset list, and secret mappings from scratch for each factory.

The two-phase architecture is composed via %run:

%run "./Load_Clean_Pipelines"  ← Phase 1 (normalize)

# ... then Phase 2 (migrate) runs in the same notebook

This means a single “Run All” executes the complete pipeline: normalize → discover → classify → migrate → generate templates → validate → report.

Output Artifacts

For each factory, the framework generates:

• migrated_pipelines/all/ — Complete set of pipeline JSONs, deployable to ADF via CI/CD. Non-impacted pipelines pass through unchanged.
• migrated_pipelines/impacted/ — Only the pipelines that had activity-level changes (suffixed _UC for disambiguation during parallel deployment).
• notebooks/ — Template notebooks (generic ingestion framework, delta lookup query) parameterized to work with any source table.
• migration_report.json — Machine-readable report: every activity migrated, its original type, the pattern applied, source dataset, linked service, and sink type.

Lessons Learned

1. ADF’s REST API export and Git format are NOT the same

This was the single biggest time sink. We initially assumed we could migrate directly from the API export. The snake_case keys, flat activity structure, and missing properties wrapper meant every downstream tool — ADF portal import, CI/CD pipelines, validation scripts — rejected the output. Building the normalization phase as a separate, testable step was the right call.

2. User-defined identifiers hide in plain sight

The camelCase conversion bug was subtle. Everything looked correct until we tested a pipeline that used a variable called Config_Schema. The expression @{variables(‘Config_Schema’)} silently failed because the variable had been renamed to configSchema in the JSON. The fix — exempting user-defined container keys from conversion — required understanding ADF’s evaluation model, not just its JSON schema.

3. Sink-type preservation matters more than you think

Our first version replaced everything with Delta table writes. It worked — until we discovered downstream SSIS packages, Power BI dataflows, and third-party tools that consumed Parquet and CSV files at specific ADLS paths. Preserving the original sink format and output path was non-negotiable for a drop-in replacement.

4. Auto-discovery beats manual configuration every time

For the first factory, we hand-built a JSON config listing every Synapse LS and dataset. For factory #2, we realized the linked service JSON files already contain everything we need — the type, the connection string, the Key Vault reference. Scanning them programmatically eliminated a class of errors (missed datasets, typos in LS names) and made the framework truly zero-config.

5. Validation is not optional when you’re rewriting pipelines

We caught real bugs in production-bound output through automated validation: a linked service reference at the activity root instead of inside typeProperties, a dataset that appeared in both Synapse and non-Synapse contexts, and a passthrough pipeline that was accidentally modified by the normalization step. Five automated checks run in seconds and saved hours of debugging in ADF.

6. AI pair programming accelerated the iteration cycle

The framework was developed iteratively with AI coding assistance. The pattern was: describe the desired behavior → generate code → run against real data → inspect edge cases → refine. This was especially effective for the recursive tree walker (where getting the JSON path names right for each container type was fiddly) and the query classifier (where the combinatorial space of ADF expression patterns benefited from rapid prototyping). The human’s role was domain judgment: deciding which patterns to support, validating business rules, and making architectural choices the AI couldn’t infer from code alone.

Conclusion

Migrating ADF pipelines from Synapse to Databricks is not a lift-and-shift — it’s a targeted rewrite of specific activities within a larger orchestration graph. By treating pipeline JSON as a parseable tree, auto-discovering the migration scope from artifact metadata, and applying a pattern catalog with recursive traversal, we turned a months-long manual effort into a parameterized, repeatable, validated process.

The framework processed 37 factories, identified 245 impacted pipelines, and migrated 526 activities with zero manual JSON editing. Every output was validated against five automated checks before deployment.

Key Takeaways:

• Treat pipeline JSON as an AST, not a config file
• Auto-discover scope from the artifacts themselves — don’t rely on manual inventories
• Define a pattern catalog that maps source activity types to specific target implementations
• Preserve sink formats and ADF expressions — migration is not the time to change data contracts
• Validate aggressively: passthrough integrity, reference checks, false positive detection
• Parameterize everything — the 37th factory should be as easy as the first

Have questions about migrating ADF pipelines to Databricks? Reach out — we’ve seen the edge cases so you don’t have to.

The post Building a Pattern-Based Engine to Migrate ADF Pipelines from Synapse to Databricks appeared first on Perficient Blogs.

NotebookLM is Google’s free AI research assistant, available at notebook.google. Unlike a general chatbot, it only answers from the sources you give it — PDFs, Google Docs, websites, YouTube videos, audio files, even Slides — which keeps its output grounded and reduces the made-up-fact problem that affect open-ended AI chat. Every answer comes with inline citations pointing back to the exact source.

It has grown well beyond note-taking. The current version runs on Google’s Gemini models and includes a genuine “Deep Research” mode that can browse the web on its own, pull in outside sources, and compile an annotated research report. From there, a built-in Studio can turn that research into slide decks, infographics, mind maps, flashcards, quizzes, and its well-known podcast-style “Audio Overviews,” plus AI-narrated video summaries.

Key Features

• Source-grounded answers: responses cite the specific document, page, or timestamp they come from, which matters for anything that needs to be fact-checked before it’s published.
• Deep Research mode: the tool browses the web, gathers relevant sources, and produces a structured report or explainer on its own.
• Studio outputs: one click turns a notebook into a slide deck, infographic, mind map, briefing doc, or an audio/video overview.
• Free tier: up to 100 notebooks and 50 sources each are available on a standard Google account, with higher limits on paid plans.
• Google Workspace integration: Drive documents can sync directly into a notebook, so it stays current as source files are updated.

Use Case in Practice

2. Wispr Flow — Voice-First Writing, Everywhere You Type

Wispr Flow (wisprflow.ai) is a system-wide AI dictation tool built by Wispr AI, a San Francisco startup founded by ex-Apple and ex-Meta engineers. It’s not a plain transcription app — a hotkey or wake word activates it in any text field, on any app, and it turns spoken speech into clean, formatted, context-aware text. That means the filler words, false starts, and “ums” of natural speech never make it into the final draft.

It runs on Mac, Windows, iPhone, and Android, and works inside virtually anything with a text box — Gmail, Slack, Notion, Google Docs, Word, and AI chat tools included. People who write a lot report roughly 3-4x their typing speed once they’re used to it, since natural speech runs at 150+ words per minute against a typing average closer to 40.

Key Features

• Context-aware cleanup: automatically removes filler words, fixes sentence structure, and adjusts tone and formatting for the app you’re dictating into.
• Universal app support: works as a system-level voice keyboard rather than a feature inside one app, so it follows you across your whole workflow.
• 100+ languages, including mixed-language dictation such as Hinglish.
• Command Mode and snippets: supports voice commands and a reusable snippet library for text you type often.
• Cross-device sync: one account carries your vocabulary, snippets, and preferences across Mac, Windows, iOS, and Android.

Use Case in Practice

3. AI Ads & Content Creation: Where the Industry Is Headed

Generative AI is no longer a novelty in advertising — it’s baseline infrastructure. Salesforce’s State of Marketing 2026 found that 87% of marketers now run generative AI in at least one recurring workflow, up from roughly half in 2024. The competitive edge has shifted: nearly everyone has access to the same tools now, so the gap that matters in 2026 is between teams turning AI output into real return and teams just producing more content.

Three shifts stand out this year. Generative video is now the default rather than the exception — most leading ad-creative platforms generate video first, not static images. Ad operations are becoming agentic: tools don’t just generate creative anymore, they pick winning variants, reallocate ad spend, and refresh fatigued creative automatically. And brand consistency has caught up with generation speed, with brand-kit and fine-tuning features now standard on major platforms so AI-generated variants stay on-brand at scale.

The Numbers

What This Means for Us

• Treat AI creative as production infrastructure, not an experiment — competitors are already using it to cut creative production time and test more variants per campaign.
• Keep a human in the loop on brand voice. AI-generated copy and video move fast, but the teams winning with it are pairing it with human strategy and review, not replacing that step.
• Start with A/B testing at scale. generating dozens of ad variants for a single campaign is now realistic; the bottleneck shifts to deciding which few to actually run.
• Budget for it explicitly. AI ad tools are increasingly a line item, not a side tool — worth scoping into next quarter’s marketing tooling budget.

Use Case in Practice

The post AI Tools Briefing – NotebookLM and Wispr Flow: AI-Powered Research and Content Creation appeared first on Perficient Blogs.

It can summarize customer histories, generate campaigns, prepare sellers for meetings, resolve service requests, and execute increasingly complex sequences of tasks. The more interesting question emerging from Dreamforce is what happens when those capabilities stop living inside isolated pilots and begin operating across the enterprise.

Rather than center Dreamforce on a single breakthrough model or another generation of copilots, Salesforce outlined pieces of a much broader agentic architecture: AIforce as a new interface layer, specialized reasoning through Koa, increasingly autonomous agents, and an enterprise framework intended to govern agents and models across ecosystems.

Taken together, these announcements point to a larger shift:

Enterprise AI is moving from a feature inside software to a layer connecting people, data, decisions, and actions across the business.

The challenge isn’t simply adopting more AI. It’s deciding how applications, interfaces, data, governance, and human judgment should work when AI becomes an active participant in the enterprise.

The Future of Enterprise Software May Be Less About the Interface

One of the biggest ideas shared at Dreamforce was also one of the simplest: What if users don’t have to open an application to benefit from everything behind it?

AIforce is Salesforce’s answer. Positioned above Data 360, Customer 360, and Agentforce, AIforce is intended to make Salesforce data, workflows, and permissions available through the environments where people already operate. Its initial surfaces illustrate the strategy: Claudeforce brings Salesforce into Claude, Slackforce brings Salesforce context to Slack, and Agentforce Coworker provides an AI teammate within Salesforce itself.

Enterprise applications have traditionally asked users to come to them. Users learn the interface, find the right record, navigate the right workflow, and move information from one application to another.

Agentic interfaces potentially reverse that relationship. The system comes to the user, bringing the appropriate data, permissions, and actions with it.

One Model Isn’t Going to Run the Enterprise

Another highlight is an enterprise AI environment where different models perform different jobs.

Koa, Salesforce’s new CRM-specific reasoning model, is one example. Rather than positioning it as a replacement for general purpose models, Salesforce designed Koa for specialized CRM reasoning while models like Claude or ChatGPT can continue handling more general tasks.

This reflects a maturation in enterprise AI strategy.

Early generative AI encouraged organizations to think in terms of model selection: Which model is best? Which provider should become our standard? Which one wins?

The more likely enterprise architecture is heterogeneous. A general-purpose model may be appropriate for one interaction; a domain-specific reasoning model may outperform it for another. Some processes may require deterministic logic alongside probabilistic reasoning. Different cloud environments or data constraints may introduce still more options.

Salesforce’s growing partnerships with Anthropic, AWS, Google Cloud, NVIDIA, and others reinforce that direction.

The strategic advantage may not come from picking a single model. It may come from building an architecture capable of choosing the right intelligence for the right task without forcing the enterprise to rebuild everything around it.

Agents Are Becoming More Specialized and More Autonomous

Dreamforce also revealed another notable change in how Salesforce talks about agents. They increasingly look less like generic AI features and more like specialized digital roles.

Salesforce introduced agents spanning customer service, IT and HR, commerce, supply chain, lead qualification, and outbound sales.

For example, Hunter, Salesforce’s outbound sales agent, is built on what Salesforce describes as a long-horizon runtime. Rather than handling a single interaction and stopping, it’s designed to maintain a goal across days or weeks, use memory, and adjust as circumstances change.

As agents become capable of operating across longer time horizons, leaders will need to define not only what agents can do, but what they should own. Which decisions can they make independently? Which actions require approval? What conditions should trigger human intervention? Who is accountable when an autonomous sequence produces an unexpected result?

The AI Stack is Getting More Complex Just as the Interface Gets Simpler

For the user, AI may make enterprise technology simpler. Ask a question in Claude, Slack, or an AI coworker and let the underlying systems determine what data, model, or agent should respond.

Behind that simplicity, however, the technology environment becomes considerably more complicated.

Organizations may have agents from Salesforce alongside agents from other platforms. Multiple models may reason over the same business context. An agent could call another agent or cross several enterprise systems before completing an action. This makes governance less of a policy exercise and more of an architectural requirement.

Salesforce’s Trusted Enterprise AI Harness reflects this. Its framework brings together context, agency, action, governance, security, and models, supported by an AI Control Plane intended to help organizations discover, register, and monitor agents, including third-party agents.

Organizations have historically governed access primarily around people and applications. An agentic enterprise introduces another participant: nonhuman actors capable of accessing information, making decisions, and taking action.

This requires organizations to rethink identity, permissions, observability, and accountability at the same time they’re expanding AI access. The companies that solve this well won’t simply have stronger guardrails – they’ll have greater freedom to experiment because they’ll understand where their agents are, what they can access, and what they’re doing.

AI Readiness is Really Enterprise Readiness

All of this makes the path from pilot to production more demanding than another round of technology deployment. AIforce can expose business functionality through new interfaces. Specialized models can improve reasoning in targeted contexts. Long-horizon agents can pursue objectives over time. An enterprise AI architecture can govern interactions among them.

An AI-powered interface can’t compensate for customer data employees themselves don’t trust. An autonomous agent can’t reliably execute a process that the organization can’t clearly define. Model orchestration does little good if systems can’t exchange context or actions consistently. And agent governance becomes difficult when permissions were never designed with nonhuman actors in mind.

That’s why adding another AI pilot can create the illusion of progress.

A controlled pilot asks whether a technology can succeed under favorable conditions. Scale exposes everything around the technology that isn’t ready.

Organizations preparing for this next phase should be asking:

• Is the data sufficiently trusted and contextualized for an agent to act on it?
• Can agents carry identity and permissions appropriately across systems?
• Which decisions require human judgment, and which can safely be delegated?
• Can actions move reliably across the applications required to complete an outcome?
• Can the enterprise monitor agents across platforms rather than one ecosystem at a time?
• What happens when an agent fails, encounters ambiguity, or behaves unexpectedly?
• How will users’ responsibilities change as agents take on more persistent roles?

The Real Opportunity Isn’t More AI. It’s a Different Enterprise.

Dreamforce provided plenty of individual technologies to evaluate. AIforce challenges where employees interact with enterprise systems. Claudeforce and Slackforce demonstrate how business context can escape the traditional application interface. Koa points toward a multi-model future. Longer-running agents expand the boundaries of autonomous action. And Salesforce’s emerging governance architecture acknowledges the complexity all of this creates.

Enterprise software itself is being reassembled. The interface is becoming more fluid. Intelligence is becoming more specialized. Agents are becoming more persistent. Platforms are becoming more interoperable. Governance is moving deeper into the architecture.

This should change how leaders approach AI investment.

The next step isn’t to accumulate as many agents, models, or pilots as possible. It’s to decide what kind of enterprise those technologies are entering.

Where should intelligence live? Which systems should remain sources of truth even when users no longer interact with them directly? Which decisions should AI influence versus own? What capabilities need to be shared across every model and agent? And where will human judgment become more important because automation has taken over everything around it?

The first phase of enterprise AI proved that the technology can do remarkable things. The next phase is about building an enterprise that can trust it to do them at scale.

The post Dreamforce 2026: AI Has Left the Pilot. Now Comes the Hard Part. appeared first on Perficient Blogs.

Claims processing, prior authorization, utilization management, risk adjustment, and member benefits operate under precise policies, effective dates, audit requirements, and legal accountability. Yet the AI technologies being introduced into these workflows, including large language models, generative AI, and machine learning, are probabilistic by design. 

The same clinical document processed twice may produce slightly different summaries. A fraud model retrained on newer claims data may assign a different score to an unchanged claim. A benefits assistant may interpret “Is PT covered?” differently from “Does the member have physical therapy benefits?” 

This variability is not simply a defect that better prompting will eliminate. It is a characteristic of this technology. 

The real challenge is not to make every AI component deterministic. It is to build a controlled, reproducible, and auditable system around non-deterministic components. 

Why Healthcare Payers Are Different 

In consumer applications, small variations in recommendations or generated language may be acceptable. In payer operations, the same variation can influence coverage, payment, provider review, coding, or a member’s financial decisions. 

In prior authorization, AI can extract clinical facts and summarize lengthy medical records. However, an omitted diagnosis, unsupported statement, or incorrect policy association can affect a consequential determination. 

In claims adjudication, identical claims evaluated with the same data, policy, rules, and effective date should produce the same result. AI may assist with classification, extraction, and exception routing, but deterministic services should continue to calculate the final outcome. 

In fraud, waste, and abuse, a high anomaly score is not a defensible explanation by itself. The model version, contributing features, threshold, supporting evidence, rules, and investigator actions must be traceable. 

The same principle applies to coordination of benefits. If AI identifies another insurer or suggests a coverage order, the system should show the member and policy information supporting that recommendation, the checks performed, and the final reviewer decision. 

These are not merely model-accuracy problems. They are system-accountability problems. 

More Instructions Do Not Guarantee More Control 

A common response to inconsistent AI behavior is to expand the prompt: 

• Follow every policy. 

• Check all eligibility conditions. 

• Never hallucinate. 

• Apply every exception. 

• Cite the source. 

• Return the required structure. 

• Escalate uncertain cases. 

More instructions may improve performance initially, but long prompts eventually introduce instruction saturation. Policies, examples, retrieved documents, conversation history, formatting constraints, and business rules compete for the model’s attention. 

This creates exposure to the “lost-in-the-middle” effect, where relevant information may be present in a long context but not used consistently, particularly when critical evidence or instructions are buried between other content. 

The architectural implication is straightforward:  

If a requirement must always be followed, it should not exist only as a prompt instruction. 

Critical requirements should be implemented as executable rules, schemas, effective-dated queries, validations, authorization controls, and workflow gates. 

Separate Language from Decisions 

A Benefit Assist use case for example illustrates the right architectural pattern. 

When a member asks whether physical therapy is covered, the answer may depend on the member’s plan, plan year, date of service, eligibility, network tier, benefit limits, accumulators, authorization requirements, riders, exclusions, and coordination of benefits. 

A language model should not infer these facts or calculate the benefit. 

The LLM should identify the intent, normalize the question, and explain the validated response. Deterministic services should retrieve the applicable data, and a rules engine should calculate the benefit. 

A controlled Benefit Assist flow should: 

1. Identify the member’s intent. 
2. Map the request to a defined benefit category. 
3. Retrieve eligibility, plan, network, accumulator, and policy data. 
4. Apply effective-dated business rules. 
5. Assemble the result using an approved response structure. 
6. Allow the LLM to simplify the language without changing the determination. 
7. Cite the source, effective date, and data timestamp. 
8. Escalate ambiguous, unsupported, or high-risk questions. 

This separation preserves conversational usability without allowing the model to become the source of truth. 

Engineering Deterministic Boundaries 

High-impact AI workflows should version the model, prompt, tools, retrieval configuration, embedding model, policy corpus, rules package, output schema, and post-processing logic. Model and prompt changes should be treated as controlled releases with regression testing and approval gates. 

Generated outputs should follow defined schemas. Required evidence, citations, rule identifiers, confidence measures, missing information, and escalation reasons should be explicit. Invalid or unsupported outputs should fail validation rather than flow silently downstream. 

Human oversight must also be designed into the architecture. AI may extract, classify, summarize, and recommend. Authorized people should approve consequential outcomes, particularly when policy ambiguity, incomplete evidence, clinical complexity, or member impact is high. 

Confidence alone should never determine automation. A confident response can still be ungrounded, based on an expired policy, or inconsistent with structured member data. 

Evaluate the System, Not Just the Model 

Traditional accuracy metrics are insufficient for agentic payer workflows. Evaluation should include: 

• Idempotency: Does the same request, data, model, tools, and rules produce the same business outcome? 

• Rules compliance: Were the correct rules, exceptions, effective dates, and precedence conditions applied? 

• Groundedness: Is every consequential claim supported by an authorized source? 

• Retrieval quality: Did the system retrieve the correct document version and relevant passage? 

• Consistency: Do equivalent questions produce the same intent, retrieval, rule execution, and material answer? 

• Regression: Do model, prompt, retrieval, or policy changes alter critical outcomes? 

Groundedness alone is not enough. An answer can be grounded in the wrong plan document and still be operationally incorrect. Reliability must therefore be measured across the complete execution path. 

That path must also be reconstructable. A regulator-ready audit trail should capture the request, idempotency key, source-data versions, model and prompt versions, tool calls, retrieved evidence, rules evaluated, raw output, validation results, human actions, and final outcome. 

The Mental Model That Changes Everything 

The AI model is not the system. 

It is one probabilistic component within a larger platform of authoritative data, deterministic retrieval, executable rules, validation gates, human accountability, monitoring, and audit controls. 

The objective is not identical wording across every model response. The objective is a workflow that produces consistent, evidence-based, policy-compliant, explainable, and reproducible business outcomes. 

The healthcare payers that succeed with AI will not necessarily be those that deploy it fastest. They will be those that build the governance and engineering infrastructure required to operate it safely at scale. 

Control is not the opposite of innovation. Control is what makes sustainable innovation possible. 

The post Determinism on a Non-Deterministic Platform: The AI Challenge for Healthcare Payers appeared first on Perficient Blogs.

Three tools that cover the full build path — Google Stitch for design, Google AI Studio for prototyping with AI, and Claude Code for shipping the actual code.

How to Read This

The three tools below sit at three different points of the same journey. Stitch turns an idea into screens you can look at. AI Studio turns an idea into a working AI prototype you can click. Claude Code turns a decision into code that’s committed to your repository. Used together they compress the distance from “what if we built…” to something real.

Each section shows what the tool looks like in use, lists what it does, and walks through two concrete scenarios. The screenshots are illustrative mockups of the workflow, not captures of anyone’s account — the intent is that someone who has never opened these tools can look at a page and understand what happens.

1. Google Stitch — Describe a Screen, Get the Design

Stitch is Google’s AI-powered UI design tool, built by Google Labs and powered by Gemini. You describe the interface you want in plain language — or upload a hand-drawn sketch or a screenshot — and it generates high-fidelity screens for web or mobile. From there you keep refining conversationally: ask for a darker theme, a different layout, an extra screen, and it updates the design.

The reason it matters isn’t just speed of mockups. Stitch closes the gap between design and development: the same screens can be exported to Figma for a designer to refine or exported directly as front-end code (HTML/CSS, Tailwind, React) for a developer to build on. That removes the usual handoff step where a design has to be rebuilt from scratch in code.

What It Looks Like in Practice

Key Features

• Text-to-design: describe the screen in normal language and get a complete, laid-out interface with colours, spacing, and components.
• Image-to-UI: upload a napkin sketch, a wireframe, or a competitor’s screenshot and get a clean, editable design in that direction.
• Theme controls: switch light and dark mode, set primary colours, corner radii, and fonts from a sidebar — changes cascade across every screen at once.
• Annotate: mark up a generated screen with comments and visual notes; the AI reads the annotations and applies the changes.
• Interactive prototyping: link screens into a clickable flow so a journey can be walked through rather than just described.
• Export both ways: push designs to Figma for design refinement, or take the generated front-end code straight into development.

Use Case in Practice

A practical note: Stitch is still a Labs-stage product, so results vary — it’s excellent for exploration, early concepts, and getting alignment quickly, but treat its output as a strong first draft rather than a finished design system. Teams that need generated UI constrained to their own production component library will still want a designer in the loop.

2. Google AI Studio — The Fastest Way to Prototype With AI

Google AI Studio is a free, browser-based workspace for building with Gemini, Google’s AI model family. At its simplest it’s a prompt playground: write a prompt, pick a model, adjust settings like temperature and max tokens, and see the result instantly — no setup, no billing, no installation. Sign in with a Google account and start.

What makes it more than a playground is what comes after the prompt works. “Get code” exports the exact API call as Python, JavaScript, or cURL so a developer can drop it into a real application. It issues free API keys. It handles text, images, audio, and video in the same interface. And its Build mode can generate and deploy whole working apps from a natural-language description, including full-stack web apps and native Android apps, with Google Workspace integrations wired up automatically.

Key Features

• Prompt playground: test and compare prompts across Gemini models, and tune parameters like temperature and top-P to control how creative or deterministic the output is.
• Multimodal input: work with text, images, audio, and video in one place rather than needing separate tools.
• Get code: export any working prompt as a ready-to-paste API call in Python, JavaScript, or cURL.
• Free API keys: generate a key from a single Google account — no cloud project or billing setup required to start.
• Build mode: describe an app in plain language and have it generated, previewed, and deployed; supports full-stack web apps and native Android apps.
• Large context window: Gemini models in AI Studio handle very long inputs, so entire documents or transcripts can go in at once.

Use Case in Practice

3. Claude Code — An AI Agent That Works in Your Codebase

Claude Code is Anthropic’s agentic coding tool. Rather than being a chat window you copy code out of, it runs where the work already happens — in the terminal, in your IDE, in the desktop app, or in the browser — with direct access to your project. You describe what you want in plain English and it plans, reads the relevant files, makes the edits, runs the tests, and tells you what it did.

It works in a loop: gather context, take action, verify results, and repeat until the task is done. You can interrupt and redirect at any point. Because it can run commands and edit files directly, it handles the full task rather than the snippet — finding the bug, fixing it, running the test suite, and offering to commit. It also connects through MCP to outside sources like Google Drive, Figma, Slack, or Jira, so it can read a design doc or update a ticket as part of the same job.

Key Features

• Build features from descriptions: explain what you want in plain English; it makes a plan, writes the code, and checks that it works.
• Debug and fix: paste an error message or describe the bug, and it traces the cause through the codebase and implements a fix.
• Understands the whole project: it keeps awareness of your project structure, so you can ask questions about an unfamiliar codebase and get real answers.
• Takes action directly: edits files, runs commands, runs tests, and creates commits, rather than handing back text for you to paste.
• Works everywhere: terminal, VS Code, JetBrains, desktop app, browser, Slack, and CI pipelines — the same agent across surfaces.
• Scriptable and automatable: it can be piped into and run non-interactively, so routine jobs can run automatically in CI.
• Extensible via MCP: connect it to Google Drive, Figma, Slack, Jira, or internal tooling so it can pull real context into a task.

Use Case in Practice

Where Each Tool Fits

A quick reference for deciding which tool a given task belongs to.

Closing Thought

What these three have in common is that they lower the cost of trying something. A design concept, an AI feature, a fix to a messy piece of code — each used to be expensive enough that it needed justifying in advance.

The post AI Tools Briefing: From Idea to Interface to Working Code appeared first on Perficient Blogs.

We’re proud to share that several Perficient experts were interviewed as part of the research process for Forrester’s The State of Agentic Commerce, Q2 2026 (Forrester Research, Inc., May 2026). To Perficient, their participation reflects more than where the market is heading. It reflects where Perficient is already delivering today, bringing AI, data, commerce platforms, customer experience, and operations together to create measurable outcomes.

What the Report Reveals

Forrester’s report examines how agentic commerce could reshape product discovery and purchasing across consumer and business markets — and the gap between the current hype and where the technology actually stands today.

That gap should not be mistaken for a reason to wait.

According to Forrester,

Currently, approximately one in 10 users who regularly use answer engines are authorizing answer engines to act semi-autonomously (i.e., without direct supervision) to complete tasks for them. Another 23% say they haven’t done so yet but would in the future.

Fully autonomous commerce may still be taking shape, but AI-powered interfaces are already changing how buyers find and evaluate products. Answer engines and AI assistants can assemble recommendations before a buyer reaches a website, contacts a sales representative, or requests a quote.

The buying journey is not disappearing.  Buyers are entering it through new channels.

For commerce leaders, that raises a more immediate question: Can AI systems find, understand, and accurately represent their products?

Product Data Determines Whether Agents Find You

Traditional commerce strategies have focused heavily on websites, search, marketplaces, and sales channels. Those experiences remain important. But an AI agent does not evaluate a product the way a person does.

The agent relies on available data to understand technical attributes, applications, pricing, availability, delivery options, and compatibility. When that information is incomplete or inconsistent, the product can leave the consideration set before the company receives any signal of buyer interest.

A person may contact sales for clarification. An AI agent can simply recommend a competitor.

This is especially critical in B2B commerce, where companies manage thousands of complex SKUs, account-specific catalogs, negotiated prices, and configuration-dependent products. Product information management cannot remain a one-time cleanup project. It must become an operating discipline supported by governance, structured attributes, consistent syndication, and clear ownership.

Agentic Commerce Readiness Goes Beyond Discovery

Product data is the foundation, but readiness does not end there.

Agent-driven transactions will place new demands on commerce platforms, order management systems, inventory services, and fulfillment operations. If an agent cannot confirm stock, validate account pricing, reserve inventory, or provide an accurate delivery date, it will route the transaction elsewhere.

That loss may happen quietly. No abandoned-cart alert. No support ticket. No opportunity for a sales team to intervene.

Measurement must evolve as well. A buyer may discover a product in an answer engine and complete the purchase through a distributor, marketplace, procurement platform, or sales representative. Traditional analytics may capture the transaction without recognizing where the decision began.

Commerce teams need to treat AI-mediated discovery as an influence channel. That means tracking answer-engine referrals, assisted conversions, agent interactions, and the product attributes that improve visibility.

How Perficient Helps Commerce Organizations Prepare

Our team connects commerce strategy, data, AI, customer experience, and delivery. We help organizations tie pilots to the data, platforms, and operating processes needed to measure results.

That work includes strengthening product data foundations, modernizing commerce and order management architectures, improving product findability, deploying practical AI use cases, and building measurement models for emerging customer journeys.

Our AI-first approach applies AI where it removes friction, improves decisions, and produces measurable results. That is what it means to be Different for real: AI-first solutions backed by disciplined delivery and measurable outcomes.

Preparing for What Comes Next

Commerce leaders do not need to predict exactly when agentic commerce will reach full autonomy. They need to prepare for a buying environment in which agents play a larger role in discovery, evaluation, and transactions.

Strengthen product data first. Then test real-time fulfillment capabilities and measure AI-driven discovery. Build a readiness roadmap using short execution cycles.

Agentic commerce is still maturing. The companies building the right foundations now will be ready when buyer behavior catches up.

Forrester clients can access The State of Agentic Commerce, Q2 2026 directly through Forrester.

The post Agentic Commerce Readiness Starts with Product Data appeared first on Perficient Blogs.

1. Smart Reconciliations

One of the most powerful applications of AI in the financial close process is automating reconciliations. Traditional methods require manual comparisons of transactions across systems—tedious and time-intensive work. AI algorithms can analyze vast amounts of transactional data, identify anomalies, and automatically match records with high accuracy. This not only accelerates the process but also flags exceptions for human review, ensuring both speed and accuracy.

2. Predictive Close and Error Detection

AI can learn from historical close data to predict how long each step in the process will take and where issues are likely to arise. This proactive insight helps accounting and finance leaders allocate resources more effectively and avoid surprises. Machine learning models can also detect irregular patterns or inconsistencies that might indicate errors or fraud, giving teams more confidence in the numbers.

3. Natural Language Processing (NLP) for Reporting

AI-driven NLP tools can generate narrative summaries of financial performance, explaining variances and trends in plain language. This reduces the time spent preparing management reports and makes financial insights more accessible to non-financial stakeholders.

4. Workflow Automation

AI can optimize the close checklist by tracking task dependencies, monitoring progress, and sending automated alerts to keep the process on schedule. It acts like a digital project manager—ensuring nothing falls through the cracks.

5. Continuous Close Capabilities

With AI, the concept of a continuous close becomes more feasible. Instead of waiting until month-end to reconcile and report, AI tools can process and analyze data in real time throughout the month. This creates a faster, more agile accounting and finance function with more timely insights. 

Final Thoughts 

AI doesn’t replace accounting or finance professionals—it empowers them. While this may sound cliché, AI provides a starting point and developed approach that can be reviewed and completed by the human team.  By automating routine tasks and providing smarter insights, AI allows accounting and finance teams to focus on strategic activities, reduce errors, and close the books faster. For organizations looking to modernize their finance operations, investing in AI-driven close solutions is no longer a futuristic idea—it’s becoming a competitive necessity. 

The post How AI is Transforming the Monthly Financial Close Process appeared first on Perficient Blogs.

Brickbuilder Specialization: Lakeflow

Brickbuilder Specialization: Security & Governance 

Data governance isn’t a compliance checkbox. It’s the foundation that determines whether your organization can scale AI, maintain trust, and operate across regulatory boundaries without breaking. Most enterprises run scattered governance models that fragment security, slow analytics, and block AI adoption before it starts. 

Perficient’s Databricks Brickbuilder Specialization for Security & Governance recognizes our deep expertise in helping organizations confidently secure, govern, and scale their data and AI on the Databricks Data Intelligence Platform. With Unity Catalog at the center, we build governance frameworks that enable security without sacrificing speed. 

We empower enterprises to: 

• Establish unified, enterprise-grade governance frameworks across data, analytics, and AI workloads 
• Strengthen security with fine-grained access controls and automated data lineage tracking 
• Improve data quality, trust, and discoverability with consistent governance patterns across domains 
• Enable secure, seamless data sharing across clouds and partners with Delta Sharing 
• Empower business teams with AI-driven insights through AI/BI Genie, governed analytics experiences, and Perficient’s AI/BI Genie workshop designed to help organizations identify high-value use cases and accelerate adoption
• Monitor and optimize platform health, performance, and cost with native Lakehouse Observability 

We deliver governance that scales with your business — not against it. 

Brickbuilder Specialization: Data Warehouse Migration

Legacy data warehouses trap data in silos, slow modernization efforts, and break when you try to scale. Most migration approaches either lift-and-shift technical debt into the cloud or drag timelines into multi-year efforts that never fully deliver. Neither model works. 

Perficient’s Databricks Brickbuilder Specialization for Data Warehouse Migrations recognizes our proven expertise in modernizing legacy data warehouses and migrating organizations to the Databricks Data Intelligence Platform. Through our Agentic Migration Factory, agent-assisted migration methodology, and specialized automation frameworks, we transform migration from a risk-heavy project into a repeatable, accelerated path to modern lakehouse architectures. Our approach combines AI-powered assessment, code conversion, validation, and optimization to reduce effort while improving migration quality and speed.

We empower enterprises to: 

• Migrate from legacy systems and cloud warehouses to modern lakehouse architectures that unify analytics and AI 
• Accelerate migration timelines and reduce costs using pre-built code and automated tools 
• Leverage Unity Catalog and Databricks SQL for unified analytics, AI workloads, and improved performance 
• Move beyond “lift and shift” to true data modernization with enhanced reliability and cost-efficiency 
• De-risk complex migrations with repeatable frameworks that reduce manual effort and technical complexity 
• Leverage Perficient’s Agentic Migration Factory, proprietary IP, and automation accelerators to streamline assessment, conversion, validation, and modernization

We deliver migrations that modernize your data infrastructure — transforming legacy constraints into competitive advantages with speed, precision, and proven methodology. 

Brickbuilder Specialization: AI

Most AI initiatives stall in pilot mode. The challenge isn’t model sophistication — it’s building AI systems that perform reliably in production, govern effectively, and scale across the enterprise. Fragmented tools, incomplete data foundations, and unclear operationalization paths keep AI stuck in experimentation. 

Perficient’s Databricks Brickbuilder Specialization for AI recognizes our deep expertise in deploying Generative AI, machine learning, and agentic solutions on the Databricks Data Intelligence Platform. Using Databricks, we move from AI experimentation to operationalized AI outcomes with proven accelerators designed for enterprise complexity, including Perficient’s AI Studio and AI AMP frameworks that help organizations rapidly design, deploy, govern, and scale AI solutions.

With Brickbuilder Accelerators and specialized frameworks, we empower enterprises to: 

• Build production-grade AI agents with a focus on creation, evaluation, governance, and deployment 
• Accelerate time to value with pre-built code and industry-specific AI solutions 
• Enrich models tailored to unique datasets for summarization, classification, and beyond 
• Deploy AI-driven analytics for healthcare, life sciences, financial services, and retail sectors 
• Accelerate AI development and governance with Perficient AI Studio and reusable enterprise AI frameworks
• Move from AI strategy to implementation with AI AMP workshops designed to identify, prioritize, and operationalize high-value AI opportunities
• Scale Generative AI and machine learning initiatives with confidence on trusted data foundations 

We deliver AI that moves beyond experimentation — transforming AI potential into measurable business impact with speed, precision, and industry expertise. 

What Brickbuilder Specializations Mean 

Databricks Brickbuilder Specializations validate more than technical capability. They confirm proven delivery, measurable outcomes, and deep expertise in the domains that determine whether data and AI initiatives succeed at scale. 

Our specializations reflect what we deliver every day: security and governance frameworks that enable scale without breaking compliance, migration expertise that modernizes legacy systems without multi-year timelines, and AI capabilities that move from pilots to production. These specializations are further strengthened by Perficient’s proprietary accelerators, including Assetflow, AI/BI Genie Workshops, Agentic Migration Factory, AI Studio, and AI AMP, which help clients reduce risk, accelerate delivery, and achieve measurable business outcomes faster.

These aren’t credentials we pursued for marketing. They’re recognition of the work we’ve already delivered — and validation of the outcomes our clients rely on. 

Building on a Foundation of Proven Expertise 

As a Databricks Gold Partner with over 200 certified professionals, 6 Databricks Champions, and now 4 Databricks specializations, Perficient delivers unified data, analytics, and AI systems that perform under operational pressure. 

We don’t chase certifications. We pursue outcomes — solutions that hold up in production, scale with your business, and deliver measurable results on the Databricks Data Intelligence Platform. 

Ready to transform your data and AI from potential to production? 

Learn more about our Databricks capabilities 

The post Perficient Achieves Databricks Brickbuilder Recognition Across the Full Platform appeared first on Perficient Blogs.

Why Testing a RAG System is Different

Testing a traditional API is straightforward: given input X, expect output Y. Testing a RAG system is categorically different. You are simultaneously validating:

• Data quality — are the right documents ingested correctly?
• Chunking quality — are chunks meaningful and complete?
• Embedding quality — do similar topics score highly together?
• Retrieval accuracy — are the most relevant chunks returned?
• LLM output quality — is the answer grounded, accurate, and complete?
• Security — is the system resistant to prompt injection and PII leakage?

Each layer can fail independently. A perfect LLM with bad retrieval still produces bad answers. This is why RAG testing requires a layered, systematic approach.

The most expensive RAG failure is silent degradation — retrieval quality slowly drifts as documents change, and nobody notices until users stop trusting the system.

The RAG Testing Pyramid — 4 Layers

Borrow the concept of the traditional test pyramid and apply it to RAG. Build from the foundation up:

Layer 1: Data Quality Testing

This is the cheapest, fastest layer — and the most overlooked. Bad data causes silent failures downstream.

• Are all expected source documents present in the vector DB?
• Are there any duplicate chunks (same content stored twice)?
• Are any chunks empty, too short (< 50 tokens), or garbled text?
• Is metadata complete — source filename, page number, and ingestion date?

def test_no_empty_chunks(chunks):
    for chunk in chunks:
        assert len(chunk.text.strip()) > 50, f'Empty chunk: {chunk.id}'
        assert chunk.metadata['source'] is not None

def test_no_duplicate_chunks(chunks):
    texts = [c.text for c in chunks]
    assert len(texts) == len(set(texts)), 'Duplicate chunks detected'

def test_document_count(vector_db, expected_count):
    actual = vector_db.get_document_count()
    assert actual >= expected_count, f'Expected {expected_count}, got {actual}'

Layer 2: Embedding Quality Testing

• Embedding generated for every chunk — no nulls or errors
• Correct vector dimensions: 1536 (Azure), 1024 (AWS/OCI)
• Semantically similar chunks score above 0.80 cosine similarity
• Unrelated chunks score below 0.30 — no false positives

def test_embedding_dimensions(embedding):

    assert len(embedding) == 1536  # For Azure text-embedding-3-small

def test_semantic_similarity():

    score = cosine_similarity(

        embed('annual leave entitlement'),

        embed('how many days holiday do I get?')

    )

    assert score > 0.80, f'Similarity too low: {score}'

def test_dissimilar_chunks_score_low():

    score = cosine_similarity(

        embed('leave policy'), embed('network firewall rules')

    )

    assert score < 0.30, f'False similarity detected: {score}'

Layer 3: Retrieval Testing with RAGAS

RAGAS (Retrieval Augmented Generation Assessment) is the industry-standard framework for evaluating RAG systems. It provides metric-driven, reproducible evaluation across your entire pipeline.

Context Precision — ‘Is what we retrieved actually relevant?’

Measures the proportion of retrieved chunks that are genuinely relevant to the question.

# Example calculation:
# Retrieved 5 chunks for: 'What is the expense claim limit?'
# Relevant: chunks 1, 3, 4  (about expenses)
# Irrelevant: chunks 2, 5  (about leave policy, office hours)
# Context Precision = 3/5 = 0.60  -->  FAIL (threshold: 0.80)

Context Recall — ‘Did we retrieve everything needed?’

Measures whether all facts required to answer correctly were present in the retrieved context.

# Example calculation:
# Ground truth answer requires 4 key facts
# Retrieved context contains 3 of those facts
# Context Recall = 3/4 = 0.75  -->  PASS (threshold: 0.75)

from ragas import evaluate

from ragas.metrics import context_precision, context_recall

results = evaluate(
    dataset=test_dataset,  # questions + ground truths + retrieved contexts
    metrics=[context_precision, context_recall]
)

assert results['context_precision'] >= 0.80, 'Precision below threshold'
assert results['context_recall']    >= 0.75, 'Recall below threshold'
print(f"Precision: {results['context_precision']:.2f}")
print(f"Recall:    {results['context_recall']:.2f}")

Layer 4: LLM Response Testing — Hallucination Detection

The Four Response Quality Metrics

Hallucination in Detail

Faithfulness measures whether every claim in the LLM’s answer can be traced back to the retrieved context. A claim that cannot be traced is a hallucination.

# Context says: 'Employees are entitled to 20 days annual leave per year'
#
# BAD answer: 'Employees get 25 days plus 5 bonus days for performance'
# --> '25 days' and 'bonus days' are NOT in context
# --> Faithfulness = 0.0  -->  HALLUCINATION  -->  FAIL
#
# GOOD answer: 'According to company policy, employees receive 20 days annual leave'
# --> Every claim traces back to the context
# --> Faithfulness = 1.0  -->  PASS

Additional hallucination detection tools:

• DeepEval HallucinationMetric — standalone hallucination scorer with explanation
• PromptFoo — test known Q&A pairs, flag when answers deviate from expected output
• LangSmith — full chain tracing to see exactly what context was passed and what was generated

The Complete Enterprise CI/CD Pipeline

Every RAG system change — whether code, documents, or configuration — must go through this automated quality gate before reaching production.

The most important insight: RAGAS evaluation is NOT just a one-time quality check. Run it continuously in production. Retrieval quality drifts as your document base evolves.

QA Automation Framework — The Complete Stack

As a QA Lead or SDET on a RAG project, this is your complete toolbox:

Check Type	Tool	What is Validated
API Testing	PyTest + Requests	REST endpoints, response codes, latency SLA
Vector DB Validation	PyTest	Chunk count, metadata integrity, index health
Retrieval Accuracy	RAGAS	Context Precision ≥ 0.80, Context Recall ≥ 0.75
Hallucination Detection	DeepEval / RAGAS Faithfulness	Faithfulness ≥ 0.85
Prompt Regression	PromptFoo	Golden Q&A dataset — no degradation vs baseline
LLM Tracing	LangSmith	Full chain traces, latency profiling, error diagnosis
Performance Testing	Locust / k6	Embedding API P95, vector search P95, LLM P95 ≤ 3s
Security Testing	Custom + OWASP	Prompt injection, PII leakage, OWASP LLM Top 10
UI / E2E Testing	Playwright	Chat interface, user workflows, accessibility (WCAG)

Recommended Test Folder Structure

tests/
├── data/
│   ├── test_ingestion.py        # All source docs ingested
│   ├── test_chunking.py         # No empty or duplicate chunks
│   └── test_metadata.py         # Source, page, date correct
├── embeddings/
│   ├── test_embedding_dim.py    # Vector shape correct
│   └── test_similarity.py       # Semantic similarity scores
├── retrieval/
│   ├── test_ragas_precision.py  # Context Precision >= 0.80
│   └── test_ragas_recall.py     # Context Recall >= 0.75
├── llm/
│   ├── test_faithfulness.py     # Hallucination detection
│   ├── test_relevance.py        # Answer relevance
│   └── test_promptfoo.py        # Golden Q&A regression
└── security/
    ├── test_prompt_injection.py  # Injection blocked
    └── test_pii_leakage.py       # No PII in responses

Series Summary — 9 Principles to Remember

1. RAG is the enterprise standard. Private data + LLM = grounded, accurate answers. No retraining needed.
2. Every pipeline stage needs testing. Ingest → Chunk → Embed → Store → Retrieve → Generate — each can fail independently.
3. Chunking quality determines retrieval quality. 512 tokens + 10% overlap is your safe default. Invest time tuning this.
4. Never switch embedding models without re-indexing. It will silently break your entire retrieval layer.
5. RAGAS is non-negotiable. Context Precision + Recall measure retrieval. Faithfulness + Relevance measure LLM output.
6. CI/CD for RAG is mandatory. Every document change can silently break retrieval. Automate your quality gate.
7. Choose cloud by ecosystem. Microsoft shop → Azure. Multi-model → AWS. Oracle DB customers → OCI.
8. Security testing for LLMs is different. OWASP LLM Top 10 is your checklist. Prompt injection is your highest risk.
9. Monitoring never stops. Set up continuous RAGAS evaluation in production. Drift is silent and inevitable.

References

• RAGAS — https://docs.ragas.io
• DeepEval — https://docs.confident-ai.com
• PromptFoo — https://promptfoo.dev
• LangSmith — https://smith.langchain.com
• Azure AI Search — https://learn.microsoft.com/en-us/azure/search/
• AWS Bedrock — https://docs.aws.amazon.com/bedrock
• Oracle AI Vector Search — https://docs.oracle.com/en/database/oracle/oracle-database/23/vecse/
• OWASP LLM Top 10 — https://owasp.org/www-project-top-10-for-large-language-model-applications/

The post How to Test a RAG System: RAGAS, Hallucination Detection, and the Complete GenAI CI/CD Pipeline appeared first on Perficient Blogs.

Snowflake Horizon Catalog is designed to help solve these challenges. By transforming metadata into an active layer of business intelligence, Horizon Catalog helps organizations improve data discovery, governance, analytics adoption, and AI readiness. 

Currently in private preview, Horizon Catalog represents Snowflake’s vision for creating a unified understanding of enterprise data across people, applications, analytics tools, and AI agents. 

Why Metadata Matters More Than Ever 

As data ecosystems continue to grow, enterprises often face a common challenge: critical business knowledge is scattered across databases, dashboards, data pipelines, and cloud platforms. 

This fragmentation creates obstacles for both human users and AI systems: 

• Data teams spend excessive time answering questions about data definitions and lineage. 

• Business users struggle to find trusted data assets. 

• Governance teams lack complete visibility into how information moves through the organization. 

• AI applications risk generating inaccurate outputs when business context is unclear. 

Snowflake Horizon Catalog addresses these challenges by creating a centralized intelligence layer that collects, enriches, and activates metadata across the enterprise. Rather than functioning as a passive inventory of assets, Horizon Catalog helps organizations create a living, governed representation of their data ecosystem. 

Horizon Catalog’s Three-Stage Approach 

At its core, Horizon Catalog follows a three-stage architecture designed to transform raw metadata into actionable business context: 

Collect → Enrich → Activate 

This framework enables organizations to move beyond simply documenting data and toward making it truly understandable and usable across analytics and AI workloads. 

Stage 1: Collect Metadata Across the Enterprise 

The first step is collecting metadata from across the modern data landscape. 

Organizations today operate across numerous systems, including cloud storage platforms, SaaS applications, databases, business intelligence tools, and transformation frameworks. Without a unified approach to metadata collection, visibility becomes fragmented and governance becomes increasingly difficult. 

Horizon Catalog connects to a broad ecosystem of technologies, including: 

Data Lakes and Cloud Storage 

• AWS S3 

• Azure Data Lake Storage 

• Google Cloud Storage 

• Iceberg, Delta, and Parquet-based environments 

Enterprise Applications 

• SAP 

• Salesforce 

• Workday 

Databases 

• PostgreSQL 

• SQL Server 

• Oracle 

• MySQL 

• Amazon Redshift 

Analytics and Data Integration Tools 

• Tableau 

• Power BI 

• dbt 

• Fivetran 

• Sigma 

By bringing metadata into a centralized framework, organizations gain a consistent view of data assets regardless of where they reside. For enterprise leaders pursuing AI initiatives, this unified visibility becomes increasingly important as data sources continue to multiply. 

Stage 2: Enrich Metadata With Business and Technical Intelligence 

Collecting metadata is only the beginning. To make data useful at scale, organizations need additional context that helps explain how information is structured, governed, consumed, and connected. 

Horizon Catalog enriches metadata across six key dimensions: 

Schema Intelligence 

Schema enrichment captures structural details such as column types, relationships, and dependencies, providing foundational understanding of data assets. 

Lineage Visibility 

Cross-platform, column-level lineage helps teams understand how data moves from source systems through transformations and ultimately into dashboards, reports, and applications. This visibility supports governance, troubleshooting, compliance, and impact analysis efforts across the organization. 

AI-Generated Descriptions 

Natural language descriptions help make technical assets easier to understand for both business and technical stakeholders. This capability can significantly improve self-service adoption by reducing reliance on tribal knowledge. 

Tags and Classifications 

Business and technical metadata can be categorized using governance tags and classifications, helping organizations improve discovery, compliance, and policy enforcement. 

Popularity and Usage Insights 

Understanding how datasets are used provides valuable context around adoption, trust, and business value. Usage signals help teams identify high-value assets and prioritize governance efforts accordingly. 

Semantic Views 

Semantic views create business-friendly abstractions on top of physical data structures. By establishing shared business definitions, organizations can reduce reporting inconsistencies and improve alignment across analytics initiatives. 

Stage 3: Activate Context Across Analytics and AI 

The true value of Horizon Catalog emerges when enriched metadata is activated across the organization. Many traditional catalogs stop at documentation. Horizon Catalog extends beyond that model by making context available directly within analytics workflows and AI-powered experiences. This activation layer is enabled through Horizon Context, which helps ensure that data consumers and AI systems have access to trusted enterprise knowledge when making decisions. 

Potential downstream applications include: 

• Snowflake CoCo 

• Snowflake CoWork 

• Cortex Agents 

• Business intelligence platforms 

• Custom applications and experiences 

As enterprises move toward AI-assisted decision-making, the ability to ground AI interactions in governed business context becomes increasingly critical. Organizations that can provide consistent definitions, lineage, and governance information to AI models will be better positioned to generate reliable outcomes while maintaining trust in AI-generated insights. 

Horizon Context: The Foundation for Enterprise AI 

One of the most significant aspects of Horizon Catalog is Horizon Context, which serves as an active context layer for analytics and AI solutions. 

Key capabilities include: 

Level-of-Detail Expressions 

Organizations can define metrics at multiple levels of granularity without creating redundant data structures. 

Composable Semantics 

Reusable semantic components allow teams to build business logic efficiently while maintaining consistency across reporting and analytics experiences. 

User-Defined Materializations 

Performance optimization techniques allow expensive calculations to be cached while maintaining freshness requirements. 

Semantic Studio 

A visual workspace enables teams to build, manage, and govern semantic views across the organization. 

Semantic View Autopilot 

AI-assisted tools help accelerate semantic model development by generating business context directly from table structures. Together, these capabilities help bridge the longstanding gap between technical data models and business understanding. 

Improving Data Discovery Through Intelligent Search 

Finding the right data remains one of the most persistent challenges facing modern organizations. Horizon Catalog addresses this challenge through a hybrid search approach that combines multiple discovery mechanisms: 

Keyword Search 

Traditional search capabilities support users who know exactly what they are looking for. 

Semantic Search 

Vector-based search enables discovery based on meaning rather than exact terminology, helping users uncover related assets they may not have known existed. 

Intelligent Re-Ranking 

Machine learning-powered relevance scoring evaluates factors such as user context, popularity, and asset freshness to surface the most useful results. For organizations seeking to increase self-service analytics adoption, improved discoverability can significantly reduce the time required to locate trusted data assets. 

Expanding Metadata Intelligence Through Select Star 

Snowflake’s partnership with Select Star highlights a growing enterprise need for metadata management that extends beyond a single platform. 

The collaboration introduces capabilities such as: 

• Unified metadata visibility across multiple technologies 

• Cross-platform lineage tracking 

• Column-level impact analysis 

• Enhanced user experiences for both business and technical users 

• Integration with Snowflake AI workflows 

As organizations continue adopting multi-platform architectures, the ability to establish consistent governance and visibility across ecosystems becomes increasingly important. 

What This Means for Enterprise Data Leaders 

From a strategic perspective, Horizon Catalog represents more than another data governance tool. It reflects a broader shift in how organizations think about metadata. Historically, metadata was viewed primarily as documentation. Today, metadata is becoming a foundational asset for AI readiness, business intelligence, data governance, and self-service analytics. 

Organizations that can effectively collect, govern, and activate metadata will be better positioned to: 

• Scale AI initiatives with confidence 

• Improve data quality and trust 

• Accelerate analytics adoption 

• Strengthen governance and compliance efforts 

• Reduce time spent searching for and validating data 

• Create consistent business definitions across teams 

Perficient’s Perspective 

At Perficient, we see Horizon Catalog as an important evolution in the enterprise data management landscape. As organizations invest in generative AI, data products, and modern analytics platforms, trusted context becomes just as important as the data itself. Business leaders need confidence that AI systems are grounded in governed, accurate, and explainable information. Snowflake Horizon Catalog helps address this challenge by turning metadata into an active intelligence layer that supports both human decision-making and AI-powered experiences. 

For organizations pursuing enterprise-scale AI, this shift from passive governance to active context management may become a critical component of building a trusted, scalable data foundation. 

Ready to Build an AI-Ready Data Foundation? 

Perficient helps organizations maximize their Snowflake investments through modern data architecture, governance, analytics, and AI strategy services. Whether you’re exploring Horizon Catalog, strengthening your data governance framework, or scaling enterprise AI initiatives, our experts can help you create the trusted data foundation needed to drive measurable business outcomes. Learn more about Perficient’s Snowflake practice.  

The post Snowflake Horizon Catalog: Transforming Metadata Into Business Context for the AI Era appeared first on Perficient Blogs.