ORACLENERD

The System Still Works: The Algorithmic Arms Race

2026-09-28T21:46:01.278-04:00

Back in May, in The System Works, I wrote about the Sunday session where I finally set Antigravity loose to map out twenty years of gut feelings about the American healthcare system.

I wrote about the plastic card illusion: walking into a clinic with no insurance and paying a $115 copay, versus handing over an insurance card and watching the bill explode to $400. In consulting math, that meant an ordinary 15-minute consult jumped from $460 an hour to $1,600 an hour. Nothing about the clinical diagnosis had changed. Only the billing plumbing had.

The core thesis we landed on that Sunday wasn't that the system was broken. It was the opposite: the system is working perfectly. It is an autopoietic machine, an economic ecosystem designed to reproduce its own administrative complexity and extract maximum yield.

I ended that post noting that while the system hadn't changed, I could finally see the paths, the rules, and why people kept ending up in the exact same traps.

Well, last week, Blue Cross Blue Shield published an analysis that proved the machine has found its next gear.

The $1 Billion Query: Mining for Acuity

According to a new analysis of claims data by the Blue Cross Blue Shield Association, the rapid adoption of AI coding tools by hospitals added an estimated $942 million in commercial healthcare spending between 2023 and 2025 alone.

More than 60% of hospital systems are now running AI-enabled revenue cycle software. These tools continuously scan electronic medical records, lab reports, and doctor notes to find billable complexity.

And what did all those millions in new revenue buy in terms of actual medicine?

Zero.

BCBSA looked closely at patients undergoing major bowel surgery. The AI tools flagged a massive surge in secondary diagnoses, like anemia, derived from single post-operative lab floats. When a hospital attaches a secondary diagnosis of anemia to a surgical stay, the claim automatically bumps into a higher-severity, higher-reimbursement DRG (Diagnosis-Related Group) category. Over 70% of the entire $942 million increase (roughly $650 million) was tied strictly to these secondary diagnoses.

Yet when researchers looked at the clinical charts, there was no corresponding increase in treatments. No uptick in blood transfusions. No change in medication. No change in bedside recovery time.

Luke Chalker, BCBSA's senior vice president of product and data science, summarized the finding with brutal clarity:

"The disconnect between diagnoses and treatment suggests that AI is identifying more billable conditions, not sicker patients."

If you're a data person, read that sentence again. That is not a bug. That is an optimization routine running exactly to spec.

The 74,000-Code Moat

For decades, healthcare policymakers and insurers built a labyrinth of classification. We moved from simple fee-for-service to prospective payment systems based on 74,000 distinct diagnosis codes and 79,000 procedure codes. The stated goal was accuracy and granular clinical documentation.

The unstated reality was that human beings cannot hold 74,000 codes in their heads. Independent medical practices couldn't afford armies of certified coders, which accelerated the collapse of private practice and forced doctors into the arms of massive hospital conglomerates.

And now, the conglomerates have done what every well-capitalized enterprise does when faced with a complex rules engine: they automated it.

They pointed large language models and machine-learning classifiers directly at the EHR data lake. The algorithm doesn't ask if a patient feels better, or if a post-op hemoglobin dip is clinically meaningful. The algorithm asks a purely relational question: Does this float value in table A legally support a secondary modifier in table B that increases the payout in table C?

The code is no longer a proxy for care. The code is the product.

The Press Release: Setting the Narrative

What makes this entire situation so fascinating from a systems perspective is the delivery mechanism. This wasn't a leaked internal memo. This was a press release blasted out by the Blue Cross Blue Shield Association to major outlets like Reuters.

Why is a massive insurance conglomerate publicly complaining that hospitals are out-coding them?

Because they are setting the narrative.

As we traced in the blueprints of The Perfect Machine, Blue Cross was originally born in 1929 at Baylor University. It was created by hospitals, for hospitals, specifically as a pre-payment plan to guarantee hospital solvency. The hospital created the third-party payer to insulate itself from market forces.

For nearly a century, the two sides grew together. Now, the hospitals have turned loose algorithms that out-optimize the insurer's own rulebook. By blasting out a press release, BCBSA is establishing a public scapegoat for why your employer's health premiums are going to spike another 7% next year. They are pointing the finger squarely at the hospital bots.

But make no mistake: this is a two-sided bot war. Insurers aren't innocent victims. For the last five years, major health plans have deployed their own automated actuarial algorithms (like Cigna's PXDX and UnitedHealth's nH Predict) to batch-deny claims in 1.2 seconds without human review. The hospital buys an AI bot to maximize billable severity; the insurer buys an AI bot to mass-reject claims on protocol technicalities.

Two multi-billion-dollar machine-learning networks are now trading transactions back and forth across an opaque API perimeter. Neither machine cares about the human lying in the hospital bed. The patient is just the transaction payload passing between them.

The System Still Works

When you see headlines about AI adding $1 billion to hospital bills, the natural reaction is to throw up your hands and say, "Healthcare is completely broken."

It's not.

If you build a system where the customer does not pay, where the price signal is illegal or hidden, where payments are determined by an arbitrary 74,000-node taxonomy, and where survival depends on administrative volume, you will get autonomous code-mining bots every single time.

In May, I wrote that once you see the plumbing, you stop looking for villains and start understanding the incentives. The BCBSA report isn't evidence of a broken system. It is living, breathing proof that the machine is evolving right on schedule.

Architected by Chet, written by Gemini 3.1 Pro

When the Guardrail Stops Guarding

2026-09-27T12:48:14.894-04:00

A few months ago, in You Can Point a Foreign Key Where?!, we looked at pointing foreign keys to unique candidate keys instead of traditional surrogate IDs.

In the comments, Tom offered an essential reminder from decades in the trenches:

"You and I have decades now of CREATE TABLE... and those decades have shown us the only thing that is truly immutable are the things stakeholders tell us are absolutely not immutable... business codes have a way of becoming less immutable across years. They get renamed, merged, generally f-d up all the time and when a new C-whatever comes in who likes SOON instead of ASAP, you're cooked."

Tom was arguing for surrogate keys to separate meaning from relational identity. He was right. Business vocabulary is fluid. What feels permanent during sprint planning rarely stays permanent across three fiscal years.

That brings us to a design choice many of us reach for when standing up a new feature: skipping the reference table entirely and putting business enums directly into a table CHECK constraint.

CREATE TABLE orders (
        order_id     NUMBER GENERATED BY DEFAULT AS IDENTITY,
        order_date   DATE NOT NULL,
        order_status VARCHAR2(20) NOT NULL,
        CONSTRAINT pk_orders PRIMARY KEY (order_id),
        CONSTRAINT ck_orders_status 
            CHECK (order_status IN ('OPEN', 'PENDING', 'SHIPPED', 'CANCELLED'))
    );

To be clear: this is not a post against CHECK constraints. CHECK constraints are indispensable. This is a post about which rules they can actually hold.

Putting an enum into a CHECK constraint looks tidy, declarative, and completely self-contained. It avoids spinning up another table, mapping another entity, or coordinating extra seed files. In the moment, it feels like disciplined engineering.

Until the business evolves, and the guardrail quietly stops guarding.

The Silent Failure: Enforcing Neither

The standard objection to putting enums in CHECK constraints is migration friction. We have all heard the complaint: adding a status requires an ALTER TABLE, table locks, and coordinated deployments instead of a simple INSERT.

That argument is true, but it is an ergonomic argument. A determined engineering team can easily wave it off as an acceptable deployment cost.

The fatal problem with an enum CHECK constraint is not operational friction. It is structural failure.

Consider Tom's scenario: leadership decides that 'PENDING' is too vague. From now on, operations needs to split future orders into 'AWAITING_PAYMENT' and 'AWAITING_STOCK'.

What happens to existing records? The business does not want to rewrite history. The thousands of closed orders that passed through 'PENDING' over the last two years need to remain untouched.

Now look at what happens to your constraint:

ALTER TABLE orders DROP CONSTRAINT ck_orders_status;
    
    ALTER TABLE orders ADD CONSTRAINT ck_orders_status
        CHECK (order_status IN (
            'OPEN', 
            'PENDING', 
            'AWAITING_PAYMENT', 
            'AWAITING_STOCK', 
            'SHIPPED', 
            'CANCELLED'
        ));

Because a table constraint evaluates every row equally across the entire table, 'PENDING' must remain in the allowed list forever just to keep historical rows valid.

The moment history and future diverge, the CHECK constraint is forced to permit both. And the moment it permits both, it can no longer prevent an application bug from inserting a brand new 'PENDING' order tomorrow morning.

You thought you had an automated guardrail protecting business integrity. What you actually have is a decorative plaque. The constraint that was written to enforce valid state transitions has quietly stopped constraining.

Temporal Boundaries Trump Hardcoded Strings

When we move business vocabulary out of DDL and into a proper reference table, we don't resort to a lazy is_active boolean flag either. As we saw in The Boolean is Lying to You, booleans erase history.

Instead, we use explicit temporal boundaries:

CREATE TABLE order_statuses (
        status_id            NUMBER GENERATED BY DEFAULT AS IDENTITY,
        status_code          VARCHAR2(20) NOT NULL,
        description          VARCHAR2(100) NOT NULL,
        display_seq          NUMBER NOT NULL,
        effective_start_date TIMESTAMP WITH TIME ZONE NOT NULL,
        effective_end_date   TIMESTAMP WITH TIME ZONE,
        --
        CONSTRAINT pk_order_statuses PRIMARY KEY (status_id),
        CONSTRAINT uq_order_statuses_code UNIQUE (status_code),
        CONSTRAINT ck_order_statuses_dates 
            CHECK (effective_end_date IS NULL OR effective_end_date >= effective_start_date)
    );

(Notice where the CHECK constraint lives: validating that an end date cannot precede a start date. A mathematical and calendar rule, exactly where it belongs.)

When the business retires 'PENDING', we don't touch the orders table. We update the lookup table:

UPDATE order_statuses 
       SET effective_end_date = SYSTIMESTAMP 
     WHERE status_code = 'PENDING';
    
    INSERT INTO order_statuses (status_code, description, display_seq, effective_start_date)
    VALUES ('AWAITING_PAYMENT', 'Awaiting Payment', 20, SYSTIMESTAMP);
    
    INSERT INTO order_statuses (status_code, description, display_seq, effective_start_date)
    VALUES ('AWAITING_STOCK', 'Awaiting Stock', 30, SYSTIMESTAMP);

Every historical order referencing 'PENDING' remains fully valid. Meanwhile, any active query or UI dropdown filters on effective_end_date IS NULL (or evaluates whether SYSTIMESTAMP falls between the start and end dates). New orders cannot select ' PENDING'. Old orders cannot be corrupted.

Better yet, temporal versioning gives you future-dating for free. When operations announces that a new status takes effect on January 1, you insert the row today with an effective_start_date of January 1. At midnight, it activates automatically. No midnight deployments, no emergency patches, and zero table locks.

The Metadata Leak

A CHECK constraint treats an enum as a naked, isolated literal. But business vocabulary never stays naked.

Before long, the rest of the team needs context:

The user interface needs human-readable labels and a coherent sort order rather than raw database codes.
Reporting pipelines need to know which statuses represent terminal states versus active work in progress.
Audit systems need to know what the valid vocabulary looked like on a specific date two years ago.

In a reference table, those attributes have a natural home (display_seq, effective_start_date, description). The database acts as a shared, queryable source of truth.

In a CHECK constraint, the database cannot hold that context. So the context leaks. It leaks into frontend TypeScript switch statements, backend YAML files, and hardcoded reporting filters. You didn't avoid complexity; you just forced your schema to live in five different application files instead of the database.

The Portable Rule

This brings us to a reliable boundary for schema design:

Use CHECK constraints for mathematical, temporal, and physical invariants. quantity > 0. percentage BETWEEN 0 AND 100. end_date >= start_date. Cross-column requirements like requiring a card token when payment type is credit card. These are arithmetic and physical boundaries. They do not drift because of a company rebrand.
Use Lookup Tables for business vocabulary and lifecycle states. Order statuses, workflow stages, customer tiers, reason codes. Anything that product owners or executives might refine next quarter.

If changing the rule violates the laws of mathematics or calendar time, it belongs in DDL. If changing the rule requires a conversation with product management, it belongs in a table.

Scoped Optimization and AI

In my reply to Tom on that earlier post, I noted that keeping enums out of CHECK constraints is one of the fundamentals that requires an explicit waiver in my coding assistant instruction files.

It is worth asking why AI assistants reach for CHECK (status IN (...)) almost every single time they are asked to scaffold a schema.

The assistant is not being lazy. It is behaving exactly like an application developer focused on a single user story: it is correctly optimizing for a scope that does not include the third fiscal year.

Within the boundaries of a single prompt, a self-contained CHECK constraint satisfies every immediate functional requirement. It compiles cleanly in thirty lines of generated SQL, avoids creating auxiliary tables, and avoids the cognitive overhead of foreign keys. The model's horizon is the current response window. It has no reason to care what happens when marketing changes 'ASAP' to 'SOON' thirty-six months from now.

If we don't guide our tools with clear fundamentals, they will produce code that is locally optimal and globally fragile.

Take the time to build the reference table, give it proper temporal boundaries, and establish the foreign key. It is the only way to ensure that when your data models meet the third fiscal year, the database is still the place telling the truth.

The arguments are mine. Drafted with Gemini 3.8 Flash.

Every JSON Column Has a Schema

2026-09-16T21:05:04.834-04:00

The arguments are mine. The typing was not.

Let's complete the trilogy (That's probably a lie...).

Over the last two weeks, we've picked apart the boolean:

In the warehouse (OLAP), a boolean duplicates a truth that already exists elsewhere (your temporal boundaries).
In an operational system (OLTP), a boolean destroys a truth that did exist (wiping out history and sequence).

A JSON column does something far sneakier: it never declares a truth, so nothing can ever contradict it.

The Unfalsifiable Blob

A shapeless blob cannot be wrong, because there is no declared shape for it to violate.

Think about what happens when you create a proper relational column. You declare customer_id INTEGER NOT NULL REFERENCES customers(id). You have drawn a hard line in the sand. If the application tries to insert 'banana', the database rejects it. If it tries to insert an orphaned customer, the database rejects it. The engine knows what is true and what is a lie, and it protects you.

Now look at a JSON column.

{"customer_id": 123} is valid JSON.
{"customer_id": "123"} is valid JSON.
{"customerId": null} is valid JSON.
{"cust_id": "banana"} is valid JSON.
{} is valid JSON.

The database accepts every single one of those rows without blinking. It writes them to disk, returns a 200 OK, and goes about its day. Why? Because you never declared a contract. You never told the database what "right" looks like, so nothing can ever be "wrong."

Until six months later, when the reporting query blows up. At that point, you don't have a data model: you have vibes and a support ticket.

Watching From the Outside

I know why people do it. A new feature lands on your desk, the product manager is still fuzzy on the specs, the attributes will probably change next sprint, and you just need to get the code out the door. So you slap a payload JSON column on the table and tell yourself you're being agile.

I've never done this. Not with JSON.

Booleans, sure. I will readily admit to that sin. I've slapped an is_active flag on a table and paid for it later. But dumping application objects into a JSON column is one I have only ever watched from the outside, which is its own kind of education.

If you come from the database world, the foundational rule has always been simple: put integrity constraints as close to the data as humanly possible. The database exists to protect the data from the application, because applications get rewritten every two years, but data lives forever.

When you drop an unconstrained JSON blob into a table, you're betting that every future engineer touching that application will remember to enforce every single implicit business rule in code.

Spoiler: they won't.

Rebuilding the Schema, Badly

Someone reading this will inevitably push back: "Wait, you can enforce integrity on a JSON column!"

And you can. Modern engines will let you bolt integrity back onto a document. Postgres will happily take a CHECK constraint on an extracted jsonb path. You can create generated columns with foreign keys, and you can build functional GIN or B-tree indexes against nested attributes.

Technically, you can do it. But look at what you are actually doing: you are reconstructing, one painful piece at a time, the relational schema you declined to write in the first place, using an esoteric, vendor-specific syntax that nobody on your team will recognize in a year.

You didn't avoid the schema. You just decided to rebuild a worse version of it by hand.

Relocating the True Cost (and Closing the Loop)

To be clear: this isn't about beating up on application developers.

When a developer drops a JSON column into a migration, they aren't trying to sabotage the company. They are responding to very real, very rational pressures: sprint deadlines, velocity metrics, and avoiding the friction of formal schema reviews. From their seat in the sprint, skipping the table design feels like pure efficiency.

Years ago, Cary Millsap wrote a post about formatting tables of numbers that contained an insight I have quoted many times:

"Good design is a topic of consideration. And even conservation. If spending 10 extra minutes formatting your data better saves 1,000 readers 2 minutes each, then you’ve saved the world 1,990 minutes of wasted effort."

Cary's math is irrefutable, but appealing to civic virtue ("save the world 1,990 minutes") rarely changes engineering behavior on its own. What actually changes behavior is seeing the feedback loop close.

When you dump a raw, shapeless JSON blob into a table, you didn't eliminate the work. You simply relocated the cost.

In the short term, you quietly transferred that cost downstream to the analytics engineers and BI developers. Every single report now requires defensive SQL: unnesting arrays, casting strings to integers, guessing at nulls, and handling three different key spellings.

But the loop doesn't stop with the analytics team.

Eventually, product asks for a new operational feature: an in-app filter, a bulk edit, or a performance dashboard built directly against that transactional table. And guess who gets assigned the ticket? The application developer.

Now, the very engineer who bypassed the schema to save twenty minutes in sprint four is staring at a production bug in sprint twelve, trying to write unreadable JSON path queries against their own shapeless blob. You didn't save time. You just deferred the agony, with interest, back onto your future self.

The Waiver

In my own agent instructions, a JSON column requires a waiver: the default answer is no, and the burden of proof is on the column.

When does it actually earn that waiver?

When the data is genuinely an opaque, third-party black box that the database never needs to reason about: raw webhook payloads, audit logs, or configuration blobs that the engine will never filter, join, or aggregate on.

If your application needs to query it, if your business needs to report on it, or if it relates to any other entity in your system: it belongs in a column.

Doh. It really is that simple.

The Boolean is Still Lying to You: The OLTP Edition

2026-09-11T21:31:33.543-04:00

The arguments are mine. The typing was not.

Let's get straight to the point (again): a boolean should never be your first move in a physical data model.

Last week, in The Boolean is Lying to You, we talked about how a boolean could ruin an OLAP dimensional model. In the warehouse, the problem with a boolean is that it duplicates a truth that already exists elsewhere (temporal boundaries).

But the temptation of the boolean doesn't vanish when you switch to an OLTP system. In a transactional database, the problem is the exact opposite: it destroys a richer truth and replaces it with a poorer one.

If anything, the drive for immediate application convenience has made it worse. You need to know if a user account is active, if a record is deleted, or if an order is shipped. The developer reflex is to slap an is_active, is_deleted, or is_shipped flag on the table.

Just like in the warehouse, the boolean is lying to you. In an operational system, lost fidelity means lost business context.

The `is_deleted` Tragedy: Timestamp Trumps Boolean

The most common offender is the soft delete: is_deleted = true.

It seems harmless. The application filters out WHERE is_deleted = false, and your data is "safe". But in an operational system, knowing that something was deleted is rarely enough. Within weeks, the business will ask: When was it deleted? Who deleted it? How long was it active before it was removed?

Your boolean is mute. It destroyed the temporal context of the event.

Instead of is_deleted, your first instinct should be a deleted_at timestamp. If deleted_at IS NULL, the record is active. If it's populated, you know exactly when the state changed. The application's WHERE clause is just as simple, but the database retains the full fidelity of the event.

The Boolean Pile-Up: Where State Machines Go to Die

Business processes are rarely binary. They are lifecycles. They are state machines. But the path of least resistance often leads to modeling these state machines as a pile of mutually exclusive booleans.

It starts innocently with is_draft = true.
Then the business process evolves, so we add is_published.
Then we need to pull it down temporarily, so we add is_archived.

Now you have a record where is_draft = true AND is_published = true. What does that mean? It means your application allowed an invalid state because your physical model didn't enforce mutual exclusivity. You forced the application code to manage the integrity of the state machine, and eventually, the code will fail.

If a record moves through a lifecycle, use a status_code (backed by a reference table) or an event-sourced ledger. A single status column makes mutually exclusive states explicit and enforceable. Booleans just allow for combinatorial explosions of invalid states.

The Tri-State Lie

A boolean promises two states: True or False.
But in a SQL database, a nullable boolean actually has three states: True, False, and NULL.

What does a NULL boolean mean in your application? Does it mean "False"? Does it mean "Unknown"? Does it mean "Not Applicable"? When is_verified is NULL, did the verification fail, or has it just not happened yet?

When you use a boolean to represent business state, you inevitably back yourself into relying on this ambiguous third state. If you have three states, you don't have a boolean. You have a poorly labeled lookup table.

The Indexing Bonus: The Nerd Special

There's a physical performance argument here, regardless of which database engine you use.

Let's say you have a transaction processing table, and you use is_processed = false to find work that needs to be done. If you index that boolean, you're indexing the entire table.

Instead, if you use a processed_at timestamp, you get a massive performance feature for free by using a partial (or sparse) index.

By creating an index specifically for the rows WHERE processed_at IS NULL, your index only contains the tiny fraction of records that actually need processing. It stays perfectly sparse, incredibly small, and lightning fast. A generic boolean flag robs you of this elegant optimization.

Downstream Devastation: Why the Warehouse Cares

It’s tempting to think that an OLTP shortcut only affects the application layer. But the damage flows downstream. When you overwrite a state with a simple boolean flag, you aren't just making a lazy choice for the transactional app—you are permanently destroying data that your analytical systems desperately need.

Transitions are destroyed. If you just update is_converted = true or is_canceled = true, you only know the final outcome. You lose the sequence of events. You can no longer calculate the duration between states, identify bottlenecks, track SLAs, or analyze churn. Operational analysis, process mining, and machine learning all require transitions to figure out why something happened. A boolean destroys the transition and leaves you with a tombstone.
Time-Travel and CDC (Change Data Capture). When your OLTP system relies on event logs, timestamps, or explicit status histories, extracting that data into your OLAP environment is deterministic and robust. You can perfectly reconstruct what the business looked like at any given second. If you rely on flipping a boolean in place, you force the warehouse to frantically poll and capture those fleeting changes before they are overwritten again, inevitably missing rapid transitions.

Stop Hiding the Business Process

In OLTP, the database is the engine of the business. When you reduce a business event (like a cancellation, a deletion, or a publication) to a boolean flag, you are erasing the context of that event. You are optimizing for a temporary application shortcut instead of modeling the reality of the domain.

Just like in dimensional modeling, the rule stands: a boolean requires a waiver. Don't reach for it just because it's easy. Force yourself to ask: "Does this state have a history? Does it have a timeline? Is it part of a larger lifecycle?"

Almost every time, the answer is yes. And almost every time, the boolean is the wrong move.

The Boolean is Lying to You

2026-09-06T14:41:11.248-04:00

The arguments are mine. The typing was not.

Let's get straight to the point: a boolean should never be your first move in a physical data model.

I know, I know. It's incredibly tempting. You're building a dimension table, you need to know if a row is the current one, so you slap an is_current flag on the end of the script and call it a day. It feels clean. It feels simple.

I'm not saying a boolean is never justified. There are cases where an attribute really is just true or false, no history, no drift, no reason code required. But that case has to be earned, not assumed. In my own agent instructions, a boolean (same as jsonb) requires a waiver before it's allowed into a physical model: a deliberate, written justification, not a shortcut reached for because the alternative required more thought. The default posture is no, and the burden of proof sits on the column, not on the reviewer.

Nowhere does a lazy boolean cost you more than in a dimensional model.

The OLAP Failure: Redundancy

If you have a perfectly modeled dimension table using a Slowly Changing Dimension (SCD Type 2), tracking a physical is_current boolean alongside it is fundamentally redundant.

The truth of whether a record is the current version at a given moment is already perfectly contained within your temporal boundaries (valid_from and valid_to). Adding a physical boolean flag right next to those dates introduces the very real risk of data anomalies where the flag eventually drifts out of sync with the timestamps: an ETL job dies halfway through, and now is_current = true on a row whose valid_to says otherwise.

In a strict physical model, you store only the primary source of truth: the explicit temporal versioning columns. No independently maintained derivation sits next to them.

The "Bit Bucket" Trade-off

Inevitably, someone building a dashboard on top of that dimension will push back:

"Every consumer has to remember how this dimension represents the current row. Just give us an is_current flag so every report uses the same simple predicate."

I understand the argument. But this is the same divide I wrote about back in 2008 and 2010: the database is not a Bit Bucket that exists to mirror the current report's WHERE clause. The physical model exists to be the single source of truth first, and convenient for one consumer second.

If the derivation is genuinely a performance problem, use a materialized view, a semantic-layer cache, or a generated expression that cannot drift independently from the temporal source of truth, not a physical flag sitting next to the dates it duplicates and can silently disagree with.

"But it makes it easier for the analysts!"

Inevitably, whenever I make this argument, someone across the table will push back: "But having an is_current flag just makes it easier for the analysts!"

Every single time I hear that, I cringe. The phrase that immediately comes to mind is "the soft bigotry of low expectations."

Are we really going to permanently cripple our physical data model and introduce risk of out-of-sync data anomalies because we assume an analyst is incapable of writing a WHERE valid_to IS NULL clause? Analysts are smart. They understand temporal data. Dumbing down the physical schema because we assume they can't handle reality is insulting to them, and dangerous for the database.

The Semantic Layer: Where Booleans Actually Belong

Don't get me wrong: while a raw boolean is a terrible way to store state, it remains a genuinely useful way for a human or a BI tool to consume it. End users love a good checkbox on a dashboard.

That's exactly what the semantic layer is for: abstracting complex, high-fidelity underlying reality into a simple, ephemeral business definition. Your semantic layer exposes a calculated Is Current Version dimension that evaluates valid_to IS NULL (or your system's max-date sentinel) on the fly, at query time, against the one column that's actually the source of truth.

The Idealistic Layer Boundary

If you want a physical record that never loses temporal precision or suffers from redundant flag logic, here's the boundary between physical schema and semantic abstraction:

Modeling Challenge	The Purist Physical Reality	The Semantic Abstraction
State Evaluation	Temporal boundaries (`valid_from` to `valid_to`).	Ephemeral `True`/`False` flag calculated on the fly for dashboard filtering.
Data Integrity	Enforced by the temporal columns themselves, single source of truth.	Enforced by a standardized definition applied consistently across every downstream BI tool.

By keeping the boolean entirely out of the physical schema and entirely inside the semantic layer, you get the best of both worlds.

References

Application Developers vs Database Developers (Part I)
Application Developers vs. Database Developers: Part II
The "Database is a Bit Bucket" Mentality
The Case for the Bit Bucket (by Michael Cohen)
Everything is a Bit Bucket (by Michael O'Neill)
Capturing Record History

How to Answer Questions the Smart Way

2026-08-22T16:49:01.709-04:00

The arguments are mine. The typing was not.

For years, one of the top recommendations on my Required Reading list has been Eric S. Raymond's classic essay, How to Ask Questions the Smart Way.

I still recommend it. It is a foundational text on respecting other people's cognitive load. Do your homework, provide context, state the problem clearly, and make it easy for the person helping you.

And it is not the only one. The industry has spent two decades obsessing over this exact friction:

The Project Maintainer's View: In his book Producing Open Source Software, Karl Fogel dedicates an entire section to handling newbie questions constructively. He popularized the ethos that even if you have seen a question 1,000 times, it is that specific user's first time asking it, making a helpful response a critical investment in community goodwill.

The "Imagine You're Answering" Framework: Developer Jon Skeet wrote a highly cited piece on Writing the Perfect Question. While technically a guide for askers, it flips the script by forcing the writer to completely adopt the mindset, limitations, and frustrations of the responder before hitting submit.

The Defensive Programming Analogy: Jeff Atwood of Coding Horror frequently blogged about the friction between askers and answerers, most notably in Don't Ask Us Questions, We'll Just Ignore You. His commentary focuses on how community systems (like Stack Overflow) must be architected to filter out noise so experts don't burn out and become toxic.

These are all brilliant pieces of writing. But after a couple of decades working across architecture, operations, and data engineering, I've realized something. We spend a massive amount of time teaching engineers how to ask better questions. We spend almost zero time teaching experts how to answer them.

I've made this mistake myself. But regardless of who is doing it, the truth remains: a lot of the communication failures I see aren't caused by bad questions. They are caused by bad answers.

The core problem is usually this: The asker is trying to establish the model. The responder is answering with implementation details, caveats, and breadcrumbs.

The Excavation

We have all witnessed, or participated in, this exact pattern:

Question: Does feature X do Y by default?
Answer: Well, it can be configured differently depending on the deployment.
Question: Right, but out of the box, does it do Y?
Answer: Administrators can change the setting to do Z instead.
Question: Okay, but if I just turn it on without changing anything, what happens?
Answer: Yes, it defaults to Y.

What follows is an excavation. The asker has to carefully dig through three or four rounds of follow-ups just to extract a simple fact.

If it takes twenty minutes and a half-dozen replies to get a one-sentence answer, the problem was not the question. The question was fine. The failure was in the information transfer.

The Expert's Burden

ESR's essay is fundamentally about reducing the cost imposed on the answerer. But there is a reciprocal obligation. If you are the expert, the owner, or the authority, you owe clarity to the asker.

I understand why this happens. It is usually a defensive mechanism. We front-load the caveats because we are terrified of being technically "wrong" or called out over some obscure edge case. We want to protect ourselves by dumping all our context onto the table at once.

But the obligation to respect someone else's time does not end when they finish asking the question.

A good answer reduces uncertainty. It shrinks the search space. A bad answer expands it.

If the audience still has the same question after you respond, you haven't helped them. You have just transferred your cognitive load onto them, forcing them to reconstruct your intent.

It is very similar to the problem with passive voice (something Cary Millsap has talked about for years). The real sin of passive voice isn't grammar. It is making the reader work to reconstruct causality.

Poor technical answers create the same problem. The audience must reconstruct the model, the assumptions, and the contract from fragments scattered across multiple replies.

Answer Like an API

Answering questions effectively is an architecture skill.

A systems person naturally thinks in contracts. What is the source of truth? What guarantees does the system make? What is the documented behavior? Everything else is just plumbing.

We need to treat our answers the exact same way. When someone asks a question, they are usually looking for the contract.

Experts often begin with caveats, history, edge cases, implementation details, and exceptions. Resist that urge.

The answer goes first. Everything else is commentary.

Question: Does feature X do Y by default?
Better Answer: Yes, it defaults to Y out of the box. The configuration option allows you to override this behavior. Here is the link to the doc.

Experts often answer in chronological order ("Here is the history, here are the caveats, therefore the answer is X"). Good communicators answer in logical order ("The answer is X, here is why, here are the caveats").

It is the Minto Principle applied to engineering. The answer should be the first sentence, not the last.

Reduce Ambiguity

The same instinct that drives us toward explicit schemas, API definitions, and data contracts should drive our communication. Make the model explicit. Put the definition where everyone can see it.

The purpose of an answer is not to display expertise. The purpose of an answer is to transfer understanding.

Good architecture, documentation, and APIs all do one thing: they reduce ambiguity. Good answers should do the exact same.

Don't make people pull teeth to understand the system.

You Can Point a Foreign Key Where?!

2026-05-17T22:35:59.799-04:00

Editor's Note: The arguments are mine. The typing was not.

Let’s talk about things we think we know, but it turns out we’ve (read: me) just been following muscle memory for twenty-plus years.

If you asked me on any given Tuesday what a foreign key does, I’d give you the standard textbook answer. It points to the primary key of a parent table. It’s bread-and-butter relational modeling. We back it with a sequence or an identity column, we join on the IDs, and we move on with our lives.

But a funny thing happened on the way to the database the other day. I realized, or rather, I was reminded, that the SQL standard and Oracle Database don’t actually care about your primary key.

A foreign key doesn't have to reference a PRIMARY KEY. It just needs to reference a minimal unique identifier. That means any column set with a valid UNIQUE constraint is fair game.

The Setup

Imagine you have a standard reference lookup table for order statuses. You’ve got your surrogate auto-incrementing ID as the PK because that’s what we do. But you also have an alphanumeric business code that the application actually uses, and that code is guaranteed unique.


CREATE TABLE order_statuses (
    status_id   NUMBER GENERATED BY DEFAULT AS IDENTITY,
    status_code VARCHAR2(10) NOT NULL,
    description VARCHAR2(100) NOT NULL,
    --
    CONSTRAINT pk_order_statuses PRIMARY KEY (status_id),
    CONSTRAINT uq_order_statuses_code UNIQUE (status_code)
);

Normally, devs will map the status_id down to the child orders table. But what if you map the code instead?

CREATE TABLE orders (
    order_id     NUMBER GENERATED BY DEFAULT AS IDENTITY,
    order_status VARCHAR2(10) NOT NULL,
    -- Look Ma, no status_id!
    CONSTRAINT pk_orders PRIMARY KEY (order_id),
    CONSTRAINT fk_orders_status 
        FOREIGN KEY (order_status) 
        REFERENCES order_statuses (status_code)
);

This compiles. It validates. It works.

Why Do We Care?

If you are a "data-first" person, this opens up some interesting pragmatic design choices, especially for seed data and reference enums.

No-Join Readability: When I run a quick SELECT * FROM orders, I don't see status 1, 2, or 3. I see 'PENDING', 'SHIPPED', or 'CANCELLED'. I don’t have to write an explicit JOIN to a lookup table just to debug a row in a terminal log.
CI/CD Sanity: Moving seed data across Dev, QA, and Prod environments when you rely purely on surrogate sequences can be a nightmare of dynamic mapping scripts. Business codes are immutable constants across environments. Your deployment scripts can just hardcode the literals without breaking things.

The Fine Print (Because this is Oracle)

Before you go rewriting your entire data model, remember that the laws of physics still apply.

First, Oracle does not automatically index foreign keys. If you point a child table to a parent’s unique business code, and somebody tries to delete or modify that code in the parent table, Oracle has to scan the child table to ensure no orphan records are left behind. If you didn’t manually put an index on orders.order_status, you are looking at a Full Table Scan and a nasty shared sub-exclusive table lock (TM) that will freeze concurrent operations.

Second, don't try this on your Slowly Changing Dimension (SCD) Type 2 tables. The second a business code repeats because you are tracking historical versions with effective dates, table-wide uniqueness breaks. And no, you can't use a partial function-based index to bypass this; declarative foreign keys need real, concrete constraints.

Relational Reality Check

In relational theory, a foreign key references a candidate key, which is simply a minimal superkey. The choice to elevate one candidate key to be the "Primary Key" is a physical implementation choice, not a logical requirement.

It’s completely valid, ANSI-standard behavior. It’s supported in Postgres and SQL Server too, so it’s not just an Oracle quirk.

It's just one of those elegant database features hiding in plain sight while application layers spend thousands of lines of code trying to reinvent referential integrity.

Keep it in the database.

Appendix: Documentation & Structural Foundations

Oracle Database Documentation

Oracle SQL Language Reference — The constraint Clause: The definitive syntax rules and structural restrictions governing referential integrity, confirming that foreign keys can target primary keys or unique constraints.
Oracle Database SQL Language Reference — Constraints

Edgar F. Codd & Relational Theory

The 1970 Foundation Paper: Codd, E. F. (1970). "A Relational Model of Data for Large Shared Data Banks." Communications of the ACM. The original blueprint that introduced relational algebra, establishing that relationships are derived strictly by matching domains over mathematical relations rather than rigidly named primary/foreign key pairs.
ACM Digital Library — A Relational Model of Data for Large Shared Data Banks
The Relational Model for Database Management: Version 2 (Book): Codd, E. F. (1990). Addison-Wesley. Codd formalizes RM/V2, explicitly grouping Primary Keys and Alternate Keys together under the definition of Candidate Keys, proving that referential integrity mathematically depends on the candidate key property of uniqueness.
ACM Digital Library — The Relational Model for Database Management (Version 2)

The System Works

2026-05-12T08:49:00.002-04:00

For the last 20 years or so I've thought about what retirement might look like.

In retirement, I'd go back to school and write a thesis on... I didn't know what to call it. The "Medical Industrial Complex?" Much of this was due to the many, many, many interactions as it relates to Kate.

"I'll go back to school, do the research, get my PhD." Mostly so I could participate in Spies Like Us shenanigans: Doctor. Doctor. Doctor. Doctor.

I had this gut feeling that something was...broken; none of it made sense.

When I didn't have insurance, $115 copay. When I did have insurance, $400.

As a consultant/contractor, I understood how we measure things in time; n dollars/hour.

OK, so Doc is $115 x 4 (because it may have been 15 minutes, I'm being super generous here), how is it now $400 x 4/hour? How in the hell does handing over a piece of plastic turn a $460/hr rate into $1,600/hour?

I'm (often) told I need a hobby. Outside of work. What many don't realize is that I'm one of the fortunate ones; what I do for a living is not work.

Choose a job you love, and you will never have to work a day in your life. Or whatever that phrase is.

So, one Sunday, a few weeks back, I set Antigravity (Gemini) loose on my idea. I just told it: "I have an idea I want to explore."

I didn't expect a book. I expected a conversation. But what happened over the next few hours was the mechanical equivalent of defining the schema for 20 years of observations. I threw two decades of "gut feelings" at the machine, and it started mapping the plumbing. Before I knew it, I had a 23-bullet outline and links to 50-75 outside sources.

"Fully fleshed out, how long would this be?"

"350-400 pages."

"OK, let's pare that back"

Gemini proceeded to suggest serializing it on Substack or just going with an abbreviated manifesto.

I chose the manifesto. We pared it down to 8 dense chapters. It wrote the Intro.

What...do I...OMG...this is...wow.

If you spend enough time listening to patients, doctors, or politicians, you will hear a single, unifying complaint spoken across every demographic and tax bracket in America:
The healthcare system is entirely broken.
It is the one rare diagnosis found across every boardroom, breakroom, and waiting room in the country. They point to the soul-crushing administrative bloat that forces practitioners into eight-minute volume appointments. They point to the labyrinthine "prior authorization" processes where algorithms deny life-saving treatments.
And yet, this frustration is born entirely from a profound paradox.
If you strip away the billing departments and the insurance mandates, the actual clinical engineering of the American medical system is nothing short of miraculous. If you are diagnosed with a rare leukemia, or require an unprecedented cardiothoracic intervention, there is quite literally nowhere else on earth you would rather be. The United States produces the most advanced, precision-engineered medical science in the history of human civilization. The people executing that science—the physicians, the bio-engineers, the surgical staff—are elite.

That is not the original, but you get the idea. I spent time reading and editing that introduction, changing the tone, the focus, where I ended up creating a GUIDING_PRINCIPLES.md file to keep it inline:

Dissect the Maze, Don't Indict the Mice
Empathy for the Inheritor
No Emotional Accusations, Only Systemic Mechanics
Labels are "Pre-Written Baggage"
Bypass Over Bureacracy

Super cool. A few iterations later, I had an Intro. On to Chapter 1. Same process. Edit for tone and clarity, but otherwise let it loose. Chapter 2, same. After Chapter 2, I just let it rip. By the end of that Sunday session, maybe 5 hours, I had a 40 page "book."

Over the next couple of days, I'd spend 30-60 minutes creating cover and chapter art (with Gemini web).

On Wednesday I had reached my quota for the month, the $20/month plan would reset on 4/11 (Saturday).

I was exploring publishing options, I've always wanted to be a published writer (as I type that, I realize there are north of 800 articles here, so...).

But like that, that specific itch was gone. I did not revisit on Saturday. I did share with friends and colleagues. I solicited feedback. I began to incorporate feedback and also track who provided what feedback.

Fast forward a month, and I came head to head with this system again. It was unpleasant to say the least.

Now, however, I was armed with new tools. The system hadn’t changed, but I could finally see how it moved; where the paths were, and why people kept ending up in the same places.

The Death of the API Barrier: From Jargon Intimidation to Result Sets and AI

2026-05-02T13:10:00.002-04:00

If you were a database guy in the early 2000s, APIs didn’t exactly show up with a gift basket and a smile.

They showed up as these massive, scary-looking blocks of XML that people called SOAP. Or maybe it was a WSDL? Honestly, I could barely spell those acronyms, let alone tell you what they were supposed to do. I didn’t have a Computer Science degree, and looking at those files felt like I’d wandered into a high-level physics lecture by mistake.

I was brand new to IT, and the whole "web service" thing was just...intimidating. It felt like a club I didn't have the password for. I didn't know how they worked, I didn't know why people liked them, and I certainly didn't want to admit I was lost. So I did what anyone does when they’re staring at something that makes them feel out of their depth: I retreated to safety.

I stayed where things made sense. Tables. Sets. SQL and PL/SQL. Logic sitting right next to the data where it belongs. I could look at a table and understand it. I could write a query and get a result. The database was my safe harbor in a storm of jargon I didn't understand.

That bias stuck with me for a long time.

REST Didn’t Fix the Mindset

Fast forward a decade. SOAP was finally out of fashion (h/t to everyone who survived that era). REST and JSON were the new hotness. We were told this was "better." And structurally, sure, it was.

A few years back, I poked around with a Strava app to see if I was just being a crank. Clean endpoints. JSON payloads. Reasonable docs.

And it was still exhausting.

Not because REST was inherently bad, but because the underlying mindset hadn't shifted an inch. I was still being handed "object-shaped" payloads and expected to navigate an object graph like a tourist without a map. Nested structures. Lists of things containing lists of other things. Manual parsing until my eyes bled.

That wasn’t a data problem. It was an OO worldview leaking all over my integration boundary. I wasn’t getting result sets; I was getting objects pretending to be data.

PL/SQL Was Already the API (We Just Forgot)

Here’s the part that often gets missed: PL/SQL has always been the API.

A PL/SQL procedure returning a ref cursor isn't some low-level implementation detail. It’s a contract. It’s the database saying, “Here is a defined shape of data. Consume it as a set.”

That distinction matters.

A result set is declarative. It’s complete. It has no behavior, no lifecycle, no implied navigation path. It just is. Objects, on the other hand, carry all this baggage about how they want to be used. They imply traversal, ownership, and state transitions.

One is about truth. The other is about interaction. Most APIs were designed by app devs, so they expose objects. Not data.

It's probably why I loved APEX so much. It didn't ask me to pretend my data was a "user object," it just asked me for the query.

AI Is the New Translation Layer

What finally broke the barrier for me wasn't some breakthrough in REST design. It was AI.

Today, I don't waste my life manually mapping JSON payloads into structures. I don’t read API docs line-by-line trying to guess where the edge cases are hiding. I just hand the spec to the machine.

My workflow is now dead simple:

Grab the API key.
Stash it in the system keychain.
Give the spec to the AI and let it do the grunt work.

Pagination, retries, flattening, normalization; the machine handles all the deterministic boring stuff. What I want on the other side isn't an object model.

I want result sets.

Give me the rows and columns. Give me something I can join, constrain, and reason about at rest.

From OO Friction to Set-Based Speed

Once that API output is normalized into a set, the friction evaporates.

I stop worrying about endpoint "shapes" and start thinking about data quality. I stop writing glue code and start defining truth. This is where PL/SQL shines. It doesn't want you to think in objects; it wants you to think in operations over sets. It wants logic close enough to the data that violating a business rule is actually difficult.

APIs that dump object graphs on you fight that model. APIs that deliver result sets fit into it like a glove.

APIs Are Just Addresses

If you look at it the right way, APIs aren't applications. They're just addresses.

A table is one physical address. A view is another. A remote API endpoint is just a slightly more annoying address. The "brain"—the business definition—doesn't live at the address. Neither does the "brawn" (the execution).

Those belong in a stable, declarative core. For me, that’s still the database.

With AI translating API specs into tabular, set-friendly forms, external APIs finally behave like first-class data sources instead of OO intrusions.

The New Era of Data Integration

For a long time, integrating with external APIs felt like a chore because it forced us to abandon our set-based mindset. We were stuck parsing hierarchies instead of querying data.

That manual struggle is over.

With AI acting as the translation layer, we can finally overcome that old "impedance mismatch" between objects and sets. We can normalize the world's objects into the sets we need to build real applications. The "API barrier" has turned into a bridge. We can stop worrying about the shape of the payload and get back to what we do best: defining the truth of the data.

AI Didn’t Know the Law. It Made the Path Visible.

2026-04-23T22:10:00.001-04:00

How I finally appealed a denied insurance claim

Kate (now 21!) had dental surgery on 12/01/2025, a 4-5 hour procedure.

I don’t pay much attention to insurance stuff because it’s overwhelming. Before I joined my current company, it was also genuinely hard and expensive. Every day I’m grateful for my employer's coverage and, by extension, the generosity that makes it possible.

Kate has had 3–4 major dental surgeries over the last couple of years. She doesn’t technically qualify as “hillbilly” yet. Her dentist says that’s eight or fewer teeth. 😆

Kate also requires general anesthesia.

Dental surgery is not cheap, which means I know the exact limits of our dental coverage: $2,500 per year. Her most recent surgery came in just north of five figures. Thankfully, Kate has Medicaid as secondary insurance. Less thankfully, her dentist, who is excellent, no longer accepts Medicaid.

About a month later, another bill arrived. This one was for anesthesia. It was roughly half the cost of the surgery itself. Sigh.

I called the anesthesiologist’s office and learned they didn’t have Kate’s current insurance information. I gave it to them and asked them to resubmit the claim. They did. Denied. Denial code: 001.

This is classified as dental work, so medical does not cover anesthesia. I think. 001 isn’t super descriptive, it says:

We denied this service. Your benefits don't cover this kind of care. You can check your plan documents or go to the Benefits section of our website for a searchable list of what's covered.

Not super descriptive. AI (Gemini CLI) to the rescue. I downloaded the EOB and asked Gemini to scan it. It did and came back the above so I asked if it there were any other reasons possible?

While the document itself doesn't provide a specific medical rationale beyond the generic reason code 001 ("Your benefits don't cover this kind of care"), we can infer a highly likely explanation based on Kate's profile and typical insurance practices. Given that Kate's diagnosis is Developmentally Disabled, the most common scenario for an automatic denial like this is:

The Underlying Procedure was Dental: It is very common for special needs individuals to require general anesthesia for routine dental work that a neurotypical patient might do awake. Because routine dental care is generally excluded from medical plans, the automated claims system will automatically deny the associated anesthesia as "not covered care," even though the anesthesia itself is medically necessary due to her disability.

“Kate’s profile.” That’s a reference to other AI work I’ve been doing trying to make my life just a bit easier handling all the bureaucracy surrounding her (Guardianship, SSI, etc). My next prompt was, “How do I appeal?”

This is where it gets super fun and unexpected. With that other context, “Kate’s Profile,” Gemini’s first response was:

I will research Florida's specific legal requirements for anesthesia coverage for individuals with developmental disabilities and provide the exact steps to file the appeal with the insurance provider.

It came back with this:

✦ To appeal this denial, you need to challenge the automated determination that the care is "not covered." Because Kate has a developmental disability, Florida law likely requires them to cover the anesthesia and facility charges, even if the underlying procedure (like dental work) is not normally covered.

The Strategy: Use Florida Law
Florida law mandates that health insurance policies must cover general anesthesia and associated hospital/facility charges for dental procedures if the patient meets specific criteria. Since Kate is over age 8, your appeal will rely on:

Statute: Fla. Stat. § 627.65755

The Argument: Kate has a developmental disability (as noted in her profile.json) that creates a "significant or undue medical risk" if dental or minor medical treatment were performed without general anesthesia in a controlled environment.

This isn’t an example of AI “knowing the law.” It’s an example of AI making a path visible that was previously opaque enough to stop me from trying.

This is the second time in as many weeks that it has helped me overcome a bureaucratic hurdle that usually stops me from pursuing a thing. The last time, it helped me transfer Kate’s Conservator Payee (SSI) from my mom to myself. My mom handled it previously because I would not engage with that labyrinthine system. She’s retired. It took her 10 months. I’m ridiculously grateful and lucky.

If you’re looking for ways to utilize AI, here’s another.

*Editor's Note

I posted this internally a few weeks back. I'm slowing making my way back out into public waters.

It's Been a Minute...

2026-04-18T21:37:00.002-04:00

Hi.

It's been a while.

Not sure if this is a one-off or not, but I'm going to give it another go. I miss writing (publicly).

Like, this is super awkward. "What do I say?"

"How are you?"

"Things are good."

"How are you?"

Real World SQL and PL/SQL: Advice from the Experts

2016-08-23T16:10:00.001-04:00

Because my hero is Cary Millsap, I'm going to do what he did and publish my ~~foreword~~ Preface. All joking aside, I consider myself incredibly fortunate to have been included in this project. I learned...a lot, by simply trying to find the author's mistakes (and there were not many). There was a lot more work than I expected, as well. (Technical) Editing is lot easier than writing, to be sure.

Brendan Tierney and Heli Helskyaho approached me in March 2015 about being an author on this book, along with Arup Nanda and Alex Nuijten. Soon after, we picked up Martin Widlake. To say that I was honored to be asked would be a gross understatement. Rather quickly though, I realized that I did not have the mental energy to devote to the project and didn’t want to put the other authors at risk. Still wanting to be part of the book, I suggested that I be the Technical Editor and they graciously accepted my new role.

This is my first official role as Technical Editor, but I’ve been doing it for years through work; checking my work, checking others work, etc. Having a touch of Obsessive Compulsive Disorder (OCD) helps greatly.

All testing was done with the pre-built Database App Development VM provided by OTN/Oracle which made things easy. Configuration for testing was simple with the instructions provided in those chapters that required it.

One of my biggest challenges was the multi-tenant architecture of Oracle 12c. I haven’t done DBA type work in a few years, so trying to figure out if I should be doing something in the root container (CDB) or the pluggable database (PDB) was fun. Other than that though, the instructions provided by the authors were pretty easy to follow.

Design (data modeling, Edition Based Redefinition, VPD), Security (Redaction/Masking, Encryption/Hashing), Coding (Reg Ex, PL/SQL, SQL), Instrumentation, and “Reporting” or turning that raw data into actionable information (Data Mining, Oracle R, Predictive Queries). These topics are covered in detail throughout this book. Everything a developer would need to build an application from scratch.

Probably my favorite part of this endeavor is that I was forced to do more than simply see if it works. Typically when reading a book, or blog entry, I’ll grab the technical solution and move on often skipping the Why, When, and Where. How, to me, is relatively easy. I read AskTom daily for many years, it was my way of taking a break without getting in trouble. At first, it was to see how particular solutions were solved, occasionally using it for my own problems. After a year or two, I wanted to understand the Why of doing it a certain way and would look for those responses where Tom provided insight into his approach.

That’s what I got reviewing this book. I was allowed into their minds, to not only see How they solved technical problems, but Why. This is invaluable for developer’s and DBAs. Most of us can figure out How to solve specific technical issues, but to reach that next level we need to understand the Why, When and Where. This book provides that.

Kscope15 - It's a Wrap, Part II

2015-07-09T15:04:00.001-04:00

Another fantastic Kscope in the can.

This was my final year in an official capacity which was a lot more difficult to deal with than I had anticipated. Here's my record of service:

2010 (2011, Long Beach) - I was on the database abstract review committee run by Lewis Cunningham. I ended up volunteering to help put together the Sunday Symposium and with the help of Dominic Delmolino, Cary Millsap and Kris Rice, I felt I did a pretty decent job.
2011 (2012, San Antonio) - Database track lead. I believe this is the year that Oracle started running the Sunday Symposiums. Kris again led the charge with some input from those other two from the year before, i.e. DevOps oriented
2012 (2013, New Orleans) Content co-chair for the traditional stuff (Database, APEX, ADF), Interview Monkey (Tom Kyte OMFG!), OOW/ODTUG Coordinator, etc.
2013 (2014, Seattle) Content co-chair for the traditional stuff (Database, APEX, ADF), Interview Monkey, OOW/ODTUG Coordinator, etc.
2014 (2015, Hollywood, FL) Content co-chair for the traditional stuff (Database, APEX, ADF)

This has been a wonderful time for me both professionally and, more importantly to me, personally. Obviously I had a big voice in the direction of content. Also and maybe hard to believe, I actually presented for the first time. Slotted against Mr. Kyte. I reminded everyone of that too. Multiple times. It seemed to go well though. Only a few made fun of me.

I was constantly recruiting too. "Did you submit an abstract?" "No, why not?" and I'd go into my own personal diatribe (ignoring my own lack of presenting) into why they should present. Sarah Craynon Zumbrum summed it up pretty well in a recent article.

But it was the connections I made, the people I met, the stories I shared (#ampm, #cupcakeshirt, etc), and the friends that I made, that's what has had the most impact on me. Kscope is unique in that way because of it's size...at Collaborate or OOW, you'll be lucky to see someone more than once or twice, at Kscope you're running into everyone constantly.

How could I forget? #tadasforkate! This year was even more special. For those that don't know, Katezilla is my profoundly delayed but equally profoundly happy 10 y/o daughter. Just prior to the conference her physical therapist taught her "tada!" and Kate would hold her hands up high in the air and everyone around would yell, Tada! I got this crazy idea to ask others to do it and I would film it. Thirty or forty videos and hundreds of participants later...

So a gigantic thank you to everyone who made this possible for me.
Here's a short list of those that had a direct impact on me...

Lewis Cunningham - he asked me to be a reviewer which started all of this off.
Mike Riley - can't really say enough about Mike. After turning me away a long time ago (jerk), he was probably my biggest supporter over the years. (Remind me next year to you tell you about "The Hug."). Mike, and his family, are very dear to me.
Monty Latiolais (rhymes with Frito Lay I would tell myself) - How can you not love this guy?
Natalie Delemar - Co-chair for EPM/BI and then boss as Conference Chair.
Opal Alapat - Co-chair for EPM/BI and one of my favorite humans ever invented. I aspire to be more organized, assertive, and bad-ass like Opal.

That list is by no means exhaustive. It doesn't even include staff at YCC, like Crystal Walton, Lauren Prezby and everyone else there. Nor does it include the very long list of Very Special People I've met. I consider myself very fortunate and incredibly grateful.

What's the future hold?
I have no idea. My people are in talks with Helen J. Sander's people to do one or more presentations next year, so there's that. Speaking of which...it's in Chicago. Abstract submissions start soon, I hope you plan on submitting. If you're not ready to submit, I hope you take try to take part in shaping the content by finding one of about 10 abstract review committees. Who knows where they may lead you?

Finally, here's the It's a Wrap video from Kscope15 (see Helen's story there). Here's Kscope16's site. Go sign up.

Is MERGE a bug?

2015-04-22T22:57:00.000-04:00

A few years back I pondered whether DISTINCT was a bug.

My premise was that if you are depending on DISTINCT to return a correct result set, something is seriously wrong with your table design. I was reminded of this again recently when I ran across Kent Graziano's post on Better Data Modeling: Are you making these 3 beginner mistakes in your data models?. Specifically:

Instead of that, you should be defining a natural, or business, key for every table in your system. A natural key is a an attribute or set of attributes (that occur naturally in the data set) required to uniquely identify a row in that table. In addition you should define a Unique Key Constraint on those attributes in the database. Then you can be sure you will not get any duplicate data into the tables.

CLARIFICATION: This point has caused a lot of questions and comments. To be clear, the mistake here is to have ONLY defined a surrogate key. i believe that even if using surrogate keys is the best solution for your design, you should ALSO define an alternate unique natural key.

So why MERGE?

I learned about the MERGE statement in 2008. During an interview, Frank Davis asked me about when I would use it. I didn't even know what it was (and admitted that) but I went home that night and...wait...I think he asked me about multi table inserts. Whatever, credit is still going to Mr. Davis. Where was I? OK, so I had been working with Oracle for about 6 years at that point and I didn't know about it. My initial reaction was to use it everywhere (not really)! You know, shiny object and all. Look! Squirrel!

Why am I considering MERGE a bug? Let me be more specific. I was working with a couple of tables and had not written the API for them yet and a developer was writing some PL/SQL to update the records from APEX. In his loop he had a MERGE. I realized at that moment there was 1, no surrogate key and 2, no natural key defined (which ties in with Kent's comments up above). Upon realizing the developer was doing this, I knew immediately what the problem was (besides not using a PL/SQL API to nicely encapsulate the business logic). The table was poorly designed.

Easy fix. Update the table with a surrogate key and define a natural key. I was thankful for the reminder, I hadn't added the unique constraint yet. Of course had I written the API already I probably would have noticed the design error, either way, a win for design.

Now, there are perfectly good occasions to use the MERGE statement. Most of those, to me anyway, relate to legacy systems where you don't have the ability to change the underlying table structures (or it's just cost prohibitive) or ETL, where you want to load/update a dimension table in your data warehouse.

Noons, how's that? First time out in 10 months. Thanks for the push.

Fun with SQL - Silver Pockets Full

2014-06-04T18:13:00.000-04:00

Silver Pockets Full, send this message to your friends and in four days the money will surprise you. If you don't, well, a pox on your house. Or something like that. I didn't know what it was, I just saw this in my FB feed:

Back in November, I checked to see the frequency of having incremental numbers in the date, like 11/12/13 (my birthday) and 12/13/14 (kate's birthday). I don't want to hear how the rest of the world does their dates either, I know (I now write my dates like YYYY/MM/DD on everything, just so you know, that way I can sort it...or something).

Anyway, SQL to test out the claim of once every 823 years. Yay SQL.

OK, I'm not going to go into the steps necessary because I'm lazy (and I'm just lucky to be writing here), so here it is:

select *
from
  (
    select 
      to_char( d, 'yyyymm' ) year_month,
      count( case
               when to_char( d, 'fmDay' ) = 'Saturday' then 1
               else null
             end ) sats,
      count( case
               when to_char( d, 'fmDay' ) = 'Sunday' then 1
               else null
             end ) suns,
      count( case
               when to_char( d, 'fmDay' ) = 'Friday' then 1
               else null
             end ) fris
    from
      (
        select to_date( 20131231, 'yyyymmdd' ) + rownum d
        from dual
          connect by level <= 50000
      )
    group by 
      to_char( d, 'yyyymm' )
  )
where fris = 5
  and sats = 5
  and suns = 5

So over the next 50,000 days, this happens 138 times. I'm fairly certain that doesn't rise to the once every 823 years claim. But it's cool, maybe.

YEAR_MONTH       SATS       SUNS       FRIS
---------- ---------- ---------- ----------
201408              5          5          5 
201505              5          5          5 
201601              5          5          5 
201607              5          5          5 
201712              5          5          5 
128 more occurrences...
214607              5          5          5 
214712              5          5          5 
214803              5          5          5 
214908              5          5          5 
215005              5          5          5 

 138 rows selected

I'm not the only dork that does this either, here's one in perl. I'm sure there are others, but again, I'm lazy.

The Riley Family, Part III

2014-04-10T22:44:00.002-04:00

That's Mike and Lisa, hanging out at the hospital. Mike's in his awesome cookie monster pajamas and robe...must be nice, right? Oh wait, it's not. You probably remember why he's there, Stage 3 cancer. The joys.

In October, we helped to send the entire family to Game 5 of the World Series (Cards lost, thanks Red Sox for ruining their night).

In November I started a GoFundMe campaign, to date, with your help, we've raised $10,999. We've paid over 9 thousand dollars to the Riley family (another check to be cut shortly).

In December, Mike had surgery. Details can be found here. Shorter: things went fairly well, then they didn't. Mike spent 22 days in the hospital and lost 40 lbs. He missed Christmas and New Years at home with his family. But, as I've learned over the last 6 months, the Riley family really knows how to take things in stride.

About 6 weeks ago Mike started round 2 of chemo, he's halfway through that one now. He complains (daily, ugh) about numbness, dizziness, feeling cold (he lives in St. Louis, are you sure it's not the weather?), and priapism (that's a lie...I hope).

Mike being Mike though, barely a complaint (I'll let you figure out where I'm telling a lie).

Four weeks ago, a chilly (65) Saturday night, Mike and Lisa call. "Hey, I've got some news for you."

"Sweet," I think to myself. Gotta be good news.

"Lisa was just diagnosed with breast cancer."

WTF?

ARE YOU KIDDING ME? (Given Mike's gallows humor, it's possible).

"Nope. Stage 1. Surgery on April 2nd."

FFS

(Surgery was last week. It went well. No news on that front yet.)

Talking to them two of them that evening you would have no idea they BOTH have cancer. Actually, one of my favorite stories of the year...the hashtag for Riley Family campaign was #fmcuta. Fuck Mike's Cancer (up the ass). I thought that was hilarious, but I didn't think the Riley's would appreciate it. They did. They loved it. I still remember Lisa's laugh when I first suggested it. They've dropped the latest bad news and Lisa is like, "Oh, wait until you hear this. I have a hashtag for you."

"What is it?" (I'm thinking something very...conservative. Not sure why, I should know better by now).

#tna

I think about that for about .06 seconds. Holy shit! Did you just say tna? Like "tits and ass?"

(sounds of Lisa howling in the background).

Awesome. See what I mean? Handling it in stride.

"We're going to need a bigger boat." All I can think about now is, "what can we do now?"

First, I raised the campaign goal to 50k. This might be ambitious, that's OK, cancer treatments are expensive enough for one person, and 10K (the original amount) was on the low side. So...50K.

Second, Scott Spendolini created a very cool APEX app, ostensibly called the Riley Support Group (website? gah). It's a calendar/scheduling app that allows friends and family coordinate things like meals, young human (children) care and other things that most of us probably take for granted. Pretty cool stuff. For instance, Tim Gorman provides pizza on Monday nights (Dinner from pizza hut...1 - large hand-tossed cheese lovers, 1 - large thin-crispy pepperoni, 1 - 4xpepperoni rolls, 1 - cheesesticks).

Third. There is no third.

So many of you have donated your hard earned cash to the Riley family, they are incredibly humbled by, and grateful for, everyone's generosity. They aren't out of the woods yet. Donate more. Please. If you can't donate, see if there's something you can help out with (hit me up for details, Tim lives in CO, he's not really close). If you can't do either of those things, send them your prayers or your good thoughts. Any and all help will be greatly appreciated.

The Riley Family, Part II

2013-11-12T23:13:00.001-05:00

You didn't miss Part I, at least not here you didn't.

Many of you know Mike Riley. If you don't, here's a little history. He's the past president of ODTUG (for like 37 years or something) and for the last two years, he's served as Conference Chair for Kscope. Yeah, that doesn't really follow, but you know I'm a bit...scattered.

Did you read the link above? OK, well, here's the skinny. Mike has rectal cancer. Stage III. If it weren't for the stupid cancer part, the jokes would abound. Oh wait, they do anyway. Mike was diagnosed shortly after #kscope13, right around his 50th birthday (Happy Birthday Mike, Love, Cancer!). Ugh. (I want to say, "are you shittin' me?" see what I mean about the jokes? I can't help myself, I'm 14). Needless to say, cancer isn't really a joke. We all know someone affected by it. It is...well, it's not fun.

Go read his post if you haven't already. I'm going to give my version of that story. I'll wait...

So, Sunday morning, Game 3 of the World Series went to the Cardinals in a very bizarre way. I was watching highlights that morning as I had missed the end of the game (doesn't everyone know that I'm old and can't stay up that late to watch baseball?). Highlights. Mike lives in St. Louis. He's a Cardinal's fan. Wouldn't it be cool if he and his family could go to the game (mostly just his family, I don't like Mike that much). So I make some phone calls to see what people think of my idea. My idea is met with resistance. OK, I'll skip the people. Let's call Lisa (Mike's wife).

Apparently Sunday's are technology free days in the Riley household, no response. I go for a bike ride, but I take my phone, just in case Lisa calls me back. After the halfway point, my phone rings, I jump off the bike to answer.

So I talked to Lisa about my idea, can Mike handle the chaos of a World Series game?

We hang up and she goes to work. BTW, I asked her to keep my name out of it, but she didn't. We'll have words about that in the future.

She calls back (I think, it may have been over text, 2 weeks is an eternity to me). "He doesn't think he can do it."

So I call Mike directly (Lisa had already spoiled the surprise.)

"What about Box seats? You know, where the people with top hats and monocles sit? Away from the rift-raft, much more comfortable and free food and beer."

Backstory. Mike had finished his first round of chemo less than a week before Sunday. To make things worse, he decided it was a good time to throw out his back. He wasn't in the best of shape.

Mike said he thought he could do it.

OK, nay-sayers aside, let's see what we can do. I emailed approximately 50 people, mostly ODTUG people; board members, content leads, anyone I had in my address book. "Hey, wouldn't it be great to send Mike and his family to Game 5 of the World Series? We need to do this quick, tickets will probably double in price tonight especially if the Cardinals win." (that would mean Game 5 would be a clincher for the Cardinals, at home, muy expensive).

Within about 20 minutes, a couple of people pledged $600.

Holy shit!

At the prices I had seen, I was hoping to get between $50 and $100 from 50 people, hoping. I had $600 already. Game starts. Now it's up to $1100 in pledges. Holy shit, Part II. This might just be possible. Another 30 minutes and were about an hour into Game 4. Ticket prices have already gone up by $250 a ticket. Given that maybe 4 people have responded and I have $1600 in pledges, I pull the trigger. I bought 4 box seat tickets for the Riley family. (I had to have a couple of beers because I was about to drop a significant chunk of change without actual cash in hand, I could be out a lot of money, liquid courage is awesome).

Tickets sent to the Riley family. Pretty good feeling.

Like I said, I was confident, but I was scared. Before the end of the night though, there was over $5K pledged to get Mike and family to Game 5. Holy shit, Part III.

By midday Monday, pledges were well over $7K. I'll refer you back to Mike's post for more details. Shorter: jerseys for the family and a limo to the game.

Here's the breakdown: 35 people pledged, and paid, $8,080. Holy shit, Part IV. Average donation was $230.86. Median was $200. Low was $30 and high was $1000. Six people gave $500 or more. Nineteen people gave $200 or more. The list is a veritable Who's Who in the Oracle community.

Tickets + Jerseys + Limo = $6027.76

Riley family memory = Priceless.

So, what happened to the rest? Well, they have bills. Lots of bills. With the remainder, $2052.24, we paid off some hospital bills of $1220.63. There is currently $831.61 that will be sent shortly. It doesn't stop there though. Cancer treatment is effing expensive. Mike has surgery in December. He'll be on bed rest for some time. His bed is 17 years old. He needs a new one. After that, more chemo and more bills.

"Hey Chet, I'd love to help the Riley family out, can I give you my money for them?"

Yes, absolutely. Help me help them. I started a GoFundMe campaign. Goal is $10K. Any and all donations are welcome. Gifts include a thank you card from the Riley family and the knowledge that you helped out a fellow Oracle (nerd, definitely a nerd) in need. You can find the campaign here.

If you can't donate money, I've also created a hashtag so that we can show support for Mike and his family. It's #fmcuta (I'll let you figure out what it means). Words of encouragement are welcome and appreciated.

Thank you to the 35 who have already so generously given. Thank you to the rest of you who will donate or send out (rude) tweets.

Fun with SQL - My Birthday

2013-11-06T11:26:00.000-05:00

This year is kind of fun, my birthday is on November 12th (next Tuesday, if you want to send gifts). That means it will fall on 11/12/13. Even better perhaps, katezilla's birthday is December 13th. 12/13/14. What does this have to do with SQL?

Someone mentioned to me last night that this wouldn't happen again for 990 years. I was thinking, "wow, I'm super special now (along with the other 1/365 * 6 billion people)!" Or am I? I had to do the math. Since date math is hard, and math is hard, and I'm good at neither, SQL to the rescue.

select 
  to_number( to_char( sysdate + ( rownum - 1 ), 'mm' ) ) month_of,
  to_number( to_char( sysdate + ( rownum - 1 ), 'dd' ) ) day_of,
  to_number( to_char( sysdate + ( rownum - 1 ), 'yy' ) ) year_of,
  sysdate + ( rownum - 1 ) actual
from dual
  connect by level <= 100000

(In case you were wondering, 100,000 days is just shy of 274 years. 273.972602739726027397260273972602739726 to be more precise.)

That query gives me this:

MONTH_OF DAY_OF YEAR_OF ACTUAL   
-------- ------ ------- ----------
11       06     13      2013/11/06 
11       07     13      2013/11/07 
11       08     13      2013/11/08 
11       09     13      2013/11/09 
11       10     13      2013/11/10 
11       11     13      2013/11/11 
...

So how can I figure out where DAY_OF is equal to MONTH_OF + 1 and YEAR_OF is equal to DAY_OF + 1? In my head, I thought it would be far more complicated, but it's not.

select *
from
(
  select 
    to_number( to_char( sysdate + ( rownum - 1 ), 'mm' ) ) month_of,
    to_number( to_char( sysdate + ( rownum - 1 ), 'dd' ) ) day_of,
    to_number( to_char( sysdate + ( rownum - 1 ), 'yy' ) ) year_of,
    sysdate + ( rownum - 1 ) actual
  from dual
    connect by level <= 100000
)
where month_of + 1 = day_of
  and day_of + 1 = year_of
order by actual asc

Which gives me:

MONTH_OF DAY_OF YEAR_OF ACTUAL   
-------- ------ ------- ----------
11       12     13      2013/11/12 
12       13     14      2014/12/13 
01       02     03      2103/01/02 
02       03     04      2104/02/03 
03       04     05      2105/03/04 
04       05     06      2106/04/05 
05       06     07      2107/05/06 
...

OK, so it looks closer to 100 years, not 990. Let's subtract. LAG to the rescue.

select
  actual,
  lag( actual, 1 ) over ( partition by 1 order by 2 ) previous_actual,
  actual - ( lag( actual, 1 ) over ( partition by 1 order by 2 ) ) time_between
from
(
  select 
    to_number( to_char( sysdate + ( rownum - 1 ), 'mm' ) ) month_of,
    to_number( to_char( sysdate + ( rownum - 1 ), 'dd' ) ) day_of,
    to_number( to_char( sysdate + ( rownum - 1 ), 'yy' ) ) year_of,
    sysdate + ( rownum - 1 ) actual
  from dual
    connect by level <= 100000
)
where month_of + 1 = day_of
  and day_of + 1 = year_of
order by actual asc

Which gives me:

ACTUAL     PREVIOUS_ACTUAL TIME_BETWEEN
---------- --------------- ------------
2013/11/12                              
2014/12/13                          396 
2103/01/02                        32161 
2104/02/03                          397 
2105/03/04                          395 
2106/04/05                          397 
2107/05/06                          396 
2108/06/07                          398 
2109/07/08                          396 
2110/08/09                          397 
2111/09/10                          397 
2112/10/11                          397 
2113/11/12                          397 
2114/12/13                          396 
2203/01/02                        32161

So, it looks like every 88 years it occurs and is followed by 11 consecutive years of matching numbers. The next time 11/12/13 and 12/13/14 will appear is in 2113 and 2114. Yay for SQL!

Why I'm voting for Danny Bryant and You Should Too

2013-10-29T14:36:00.000-04:00

I'm talking about the ODTUG Board of Directors.

This.

That's really all you need isn't it?

Fine.

Today wraps up the voting period for the ODTUG Board of Directors. If you're asking me what ODTUG is, stop reading now. If you are a member of ODTUG, then please give me a few minutes to pontificate (that's a word I heard Jeff Smith use once, hopefully it makes sense here).

Your favorite Oracle conference, KScope, is largely successful based on the efforts of the Board, along with the expert advice of the YCC group. In addition, if you think ODTUG should "do more with Essbase" or "charge more for memberships" these decisions are made and carried out by the board.

So if you like being in ODTUG, and you want to help it get better and grow, and be as awesome as possible, you only need to do one thing today. Go vote. Midnight tonight (10/29) is the deadline. Do it.

You get to vote for several people. I suggest you read their bios. I'll save you the time for at least one vote, and that's for Danny Bryant.

Besides that awesome photo (#kscope12 in San Antonio) up above, here are several more reasons.

1. He's into everything. OBIEE. EBS. Essbase. SQL Developer. Database. Not very many people have their hands in everything, he does. He will be able to represent the entire spectrum of ODTUG members.
2. He's a fantastic human being. It's not just because he takes pictures of himself wearing ORACLENERD gear everywhere (doesn't hurt though), he's just, awesome.
3. This (Part II)
4. He also always answers the phone, tweets, and emails I send him. He might be sick, or he might just be that responsive. The ODTUG Board member responsibilities will fit nicely on his shoulders I believe.

So go vote. Now.

Excel VS BI in Financial Analysis: why the fight was over before it started

2013-10-21T20:50:00.000-04:00

by Victor Fagundo

The argument over why Businesses should abandon Excel in favor of more structured tools has been raging for as long as I have had more than a casual exposure to Oracle products. From the standpoint of an IT user Excel appears to be a simplistic, flat-file-based, error-prone tool that careless people use, despite its obvious flaws. Petabytes of duplicative Excel spreadsheets clog network drives across the globe; we as IT users know it, and it drives us crazy. Why, oh why, can’t these analysts, project managers, and accountants not grasp the elegant beauty of a centralized database solution that ensures data integrity, security, and has the chops to handle gobs of data, and abandon their silly Excel sheets?

I’ll tell you why: Excel is better. Excel the most flexible and feature-rich tool for organizing and analyzing data. Ever. Period.

For the past few years I have lived in a hybrid Finance/IT role, and in coming from IT, I was shocked at how much Excel was used, for everything. But after working with Excel on a daily basis for several years, I am a convert. An adept Excel user can out-develop any tool ( BI, Apex, Hyperion, Crystal Reports ) handily. (when dataset size is not an issue). Microsoft has done too much work on Excel, made it too extendable, too intuitive, built in so much, that no structured tool like BI, APEX, SAP, Hyperion will EVER catch up to its usability/flexibility.

Take this real-world example that came across my desk a few months ago: for a retail chain define a by-week, by-unit sales target, and create a report that compares actual sales to this target. Oh, and the weekly sales targets get adjusted each quarter based on current financial outlook.

How quickly could you turn around a DW/BI solution to this problem? What would it involve?
• Create table to house targets
• Create ETL process to load new targets
• Define BMM/Presentation Layers to expose targets
• Develop / test / publish report.

A day? Maybe? If one person handled all steps (unlikely, since the DB layers and RPD layers are probably handled by different people.)

I can tell you how long it took me in Excel: 3 hours (OBIEE driven data-dump, married with target sheet supplied to me). I love OBIEE, but Excel was still miles faster/more efficient for this task. And I could regurgitate 6 other examples like this one off at a moment’s notice.

Case in point: 95% of the data that C-level executives use to make strategic decisions is Excel based.

If you’ve ever sat in on a presentation to a CEO or other C-level executive at any medium to large sized company, you know that people are not bringing up dashboards, or any other applications. They are presenting PowerPoints with a few (less than 7) carefully massaged facts on them. If you trace the source of these numbers back down the rabbit hole, your first stop is always Excel. Within these Excel workbooks you will find “guesses” and “plugs” that fill gaps in solid data, to arrive at an actionable bit of information. It’s these “guesses” and “plugs” that are very hard to code for in an environment like OBIEE (or any other application). Can it be done? Yes, of course, with gobs of time and money. And during the fitful and tense development, the creditably of the application is going to take major hits.

Given the above, the usefulness of OBIEE might seem bleak. But I strongly feel that applications such as OBIEE do have a proper place in the upper organizational layers of modern business: Facilitating the Tactical business layers, and providing data-dumps to the Strategic Business layers.

Since this post is mainly about Excel, I will focus on how OBIEE can support the analyses that are inevitably going to be done in Excel.

Data Formatting, Data Formatting, FORMATTING!! I can’t stress this enough. For an analyst, having to re-format numbers that come out of an export so that you can properly display them or drive calcs off them in Excel is infuriating, and wasteful. My favorite examples: in a BI environment I worked with percentages were exported as TEXT, so while they looked fine in the application, as soon as you exported them to Excel and built calcs off them, your answer was overstated by a factor of 100 (Excel understood “75%” to be the number 75 with a text character appended, not the number 0.75).

Ask your users how they would like to SEE a fact in Excel: with decimals or not? With commas or not? Ensure that when exported to Excel, facts and attributes function correctly.

“Pull” refreshes of information sources in Excel. In the finance world, most Excel workbooks are low to medium complexity financial models, based off a data-dump from a reporting system. When the user wants to refresh the model, they refresh the data-dump, and the Excel calculations do the rest. OBIEE currently forces a user to “push” a new data-dump by manually running/exporting from OBIEE and then pasting the data into the data-dump tab in the workbook. What an Excel user really desires is to have a data dump that can be refreshed automatedly, using values that exist on other parts of the workbook to define filters of data-dump. Then all the user needs to do is trigger a “pull” and everything else is automated. Currently OBIEE has no solution to this problem that is elegant enough for the common Excel user. (Smartview must have its filters defined explicitly in the Smartview UI each time an analysis is pulled.)

The important part to take away from these 2 suggestions, and this entire post, is that to maximize the audience of OBIEE, we must acknowledge that Excel is the preferred tool of the Finance department, due to its flexibility, and support friendly exports to Excel as a best practice. We must also understand that accounting for this flexibility in OBIEE is daunting, and probably not the best use of the tool. If your users are asking for a highly complex attribute or fact, that is fraught with exceptions and estimations, chances are they are going to be much happier if what you give them is reliable information in a data-dump form, and allow them to handle the exceptions and estimations in Excel.

#OOW13

2013-09-16T22:41:00.000-04:00

I'm going to be busy.

Here's my list of events:

Saturday
- I arrive in San Francisco on Saturday around 1 PM. If you're arriving at around the same time, let me know, we can share a cab into the city.
- Beer. After arriving I plan on finding a very cold Pliny the Elder. Or three.
- ODTUG Dinner. I'm crashing this one. It's Board members only to my knowledge and until someone says I cannot go (especially if fueled by more than one Pliny the Elder), I'm going.
Sunday
- Open World Bridge Run. Not sure if I can make it, but I'm going to try. I'm presenting at 10:15 so it will be a tough decision.
- 10:30 to 11:30. Thinking Clearly About Performance. Somehow I managed to con Cary Millsap into a duet of sorts. I have him convinced it is the other way around. Either way, it should be fun (I am not nervous!).
- 2:15 to 4:30, Software Development in the Oracle Ecosystem, Part I and Part II. I'm moderating the aforementioned Mr. Millsap, Sten Vesterli, Markus Eisele and Jerry Brenner (My first boss was scheduled to speak as well, but he had a last minute change of plans, jerk).
- Oracle ACE Dinner. Evening.
- Post Oracle ACE Dinner drinks...wherever the night takes me.
Monday
- Oracle OpenWorld - San Francisco Bay Swim - Part II, 7:30 AM. We had almost 20 last year, 33 have signed up (on the page anyway) this year. Come along. Cool t-shirts too, sponsored by Oracle Technology Network and designed by Lauren Prezby.
  
  Let's not forget the swim caps! Sponsored by the encouragable Bjoern Rost of The portrix group (he's like me, afraid of capital letters) (designed by Lauren Prezby).
- Oaktable World. You'll most likely catch me here after the swim and before the...
- Wear Your ORACLENERD Gear Day, 3 PM to 4:30 PM. We'll be taking a group photo around 4:30 PM. If you can't make it for that, come by when you can and get a picture with me. You know I like that sh...stuff. You don't have a shirt/hat/sticker/random-item? OTN Lounge is giving away one hundred cool red t-shirts (Lauren Prezby, again).
  
  This picture is coinciding with the APEX Developer Challenge which goes from 3 - 7:30 that afternoon/evening. I might even give it a go (fueled, hopefully, by Pliny the Elder).
Tuesday
- Oaktable World.
- Leave at noon.

Somewhere in there I'll get a chance to breathe. I'll also attend some sessions, hopefully.

DBA or Developer?

2013-08-26T17:01:00.001-04:00

I've always considered myself a developer and a LOWER(DBA). I may have recovered perhaps one database and that was just a sandbox, nothing production worthy. I've built out instances for development and testing and I've installed the software a few hundred times, at least. I've done DBA-like duties, but I just don't think of myself that way. I'm a power developer maybe? Whatevs.

I'm sure it would be nearly impossible to come up with One True Definition of The DBA ™. So I won't.

I've read that Tom Kyte does not consider himself a DBA, but I'm not sure most people know that. From Mr. Kyte himself:

At the same conference, I asked Cary Millsap the same question:

I read Cary for years and always assumed he was a DBA. I mean, have you read his papers? Have you read Optimizing Oracle Performance? Performance? That's what DBAs do (or so I used to think)!

It was only after working with him at #kscope11 on the Building Better Software track that I learned otherwise.

Perhaps I'll make this a standard interview question in the future...

Semi-related discussions:

1. Application Developers vs. Database Developers
2. Application Developers vs. Database Developers: Part II

Conditional Formatting of Calculated Items in OBIEE 11g

2013-08-13T16:24:00.002-04:00

By Victor Fagundo

Calculated items in OBIEE Pivot tables can be very useful in certain reporting circumstances, either for ease of development, or to meet specific report requirements. While calculated items in OBIEE are easy, and flexible, they do have one important drawback: they take on the data and display formatting of the fact column they are calculated against.

The most common case is the calculation of a % change across a time dimension in financial reporting ( Year over Year, Quarter over Quarter, etc.). ¹This type of calculation usually takes the form of a percent change calculation similar to below:

 (( $2 - $1 ) / $1) *100

By default, if you perform this calculation against a numerical fact ( sales, customers) you will run into the problem of how to display the % change in the correct format, since the calculated % will want to take the form of the fact it is calculated against, as can be seen in the example ² below:

Figure 1 - Pivot Table

Figure 2 - Calculated item

Figure 3 - Results

Not very pretty at all.

As people searched for a work around to this problem 3 common solutions have arisen:

Use HTML formatting tricks to “hide” trigger text in the results, then conditionally format off those triggers. While inventive, as the comments note, this solution falls flat if the report is ever printed, as the PDF engine will pick up and display all of the hidden characters.
Convert the pivot table to a regular table with some complex column formulas. Very time consuming and cumbersome, would also not solve the requirement of showing the dimension values noted as noted in Footnote1.
Convert the calculated result to text and manually add your formatting characters. I don’t think this actually works since the calculated fields won’t accept logical SQL functions, and this would be very cumbersome.

Now with 11g providing conditional formatting that allows you to override the default data format, this is possible via the following steps:

Add a column that is a COUNT DISTINCT on the dimension that you are calculating across ( in the displayed example, “Time T05 Per Name Year”. This column will serve as your “trigger” to apply your conditional formatting.

Figure 4 - Column Formula
For each of your facts, apply a conditional format that is triggered when the above column value is zero. In the formatting, apply whatever visual and data formats you desire. In this example we will format the data as a percent, with one decimal place.

Figure 5 - Condition

Figure 6 - Format when condition is met
Exclude the “trigger” column from your pivot view. View your results and be satisfied:

Figure 7 - Correct formatting of calculated item.

* Note that this would also allow you to apply visual formatting if you wanted to distinguish this row/column as a total.

Why it works

The use of conditional formatting that applies a data type as part of the format is a straightforward leap of logic, but what to use as the trigger? Most people will try to use the dimension they have setup the calculation in. However, if you try to use the text description given to the calculated item you will find that the condition is never applied:

Figure 8 - Condition on dimension

Figure 9 - Condition never met, format never applied

If you try to setup a filter that is true when the dimension is not in reasonable range of values ( in this example we try to format off all years not in the 2000s ) you will find that your calculated item is skipped as well (this has the added vulnerability of being very explicit):

Figure 10 - Condition on dimension values

Figure 11- Condition never met, format never applied

The reason for all this is that the calculated item “borrows” EACH of the dimension values it operates against. Hence, no matter how inventive your filter is, as long as you are trying to somehow separate the calculated member away from the members it is operating on, you will never succeed. This member “borrowing” is apparent if you add the dimension it operates against to the query a 2nd time, and look at the table view.

Figure 12 - Calculated item "borrows" members

But since the “member value” given to the calculated item does not actually exist in the dimension, if you try to perform a count distinct against it, you will always get zero.

Figure 13 - Count distinct against dimension

There is your difference; there is your “trigger.” The rest is basic formatting.

1: You might suggest that this requirement is better served using column(s) with time series calculations, and you might be right. However, more often than not the user will want to SEE the time periods being compared ( 2012 vs 2011, or 08/07/2012 vs 08/07/2011). When using facts with time series calculations you will only be able to show “this year” vs “last year” since the column heading of the time series calculated fact will always be static. In these cases you will need to use the base fact and a time dimension, along with the solution provided here.

2: All screen shots, and examples used in this post are performed in Sample App V305. An XML of the final correctly formatted report can be downloaded here.

Learn To ______ In A Year

2013-07-30T23:02:00.002-04:00

It started at The Talent Code blog by Daniel Coyle a few weeks back, What's Your LQ (Learning Quotient)?. That led me to Diamondbacks’ Goldschmidt Has Little Ego and Few Limits. I like baseball stories. I especially like this passage:

“A lot of kids have so much pride that they want to show the coaches and the front office that they know what they’re doing, and they don’t need the help,” Zinter said. “They don’t absorb the information because they want us to think they know it already. Goldy didn’t have an ego. He didn’t have that illusion of knowledge. He’s O.K. with wanting to learn.”

I identify with that. I believe part of my success is because I ask questions.

Back to the original article. Then I end up here, Can Everyone Be Smart at Everything? I seem to lack the ability to focus for extended periods of time. Well, not quite true. I have the ability to focus, but I like to focus on a million different things. Does that count? I don't know.

I'm often envious of my friends who have been DBAs for 20 years, or worked with OBIEE for 10 years (don't argue with me...I know Oracle hasn't owned it for 10 years, I'm looking at you Christian), or APEX for 10 years (that's safe to say). I've flirted with all of those, but I've never committed...See how I get distracted easily? Wow.

And just as importantly, that mistakes are part of good learning. As a Wired article recently reported about why some are more effective at learning from mistakes, “the important part is what happens next.” People with a “growth mindset” — those who “believe that we can get better at almost anything, provided we invest the necessary time and energy” — were significantly better at learning from their mistakes.

and then...

“The meaning of difficulty changes. Difficulty means trying harder, trying a different strategy. They understand that change is possible, and progress occurs over time.”

OMFG. Focus!

Back to the original article and I'm reading through the comments. Someone links up to this young lady who taught herself how to dance in a year. Watch it.

Which finally brings me back to The Talent Code, To Improve Faster, Think Like a Startup. Staying with me? How about this?

Finally, there's a point. I want to do this. Maybe not dance (as much fun as that may be), but something else. Krav Maga? Algebra? Calculus (I'm pursuing my physics or engineering degree in 2035, I need to study my math). I want to test out her technique. Small, discrete steps practiced daily towards some end goal (pass a calc test, take a real estate licensing test, whatever). The problem for me, if you haven't noticed, is focus. This method may help.

If you were to try something like this, what would you set out to learn?

Write It Out

2013-07-08T11:54:00.001-04:00

This one was sitting in the drafts folder for a week or two, then I saw this post on Twitter:

I wonder what percentage of people ask a question and figure it out on their own before you help them. What about in your experience?
— Amy Caldwell (@amyccaldwell) July 8, 2013

Years ago I had a boss who was my technical superior (he may still be). I used to pop in and out of his office, or try to, and ask questions. Most of them were silly, n00b questions.

He was nice, but busy. It didn't take me very long to "read" that. So I slowed down my pace of questions. I began to write things up via email so that he could respond when he the had time. I started to use forums as well. Then I ~~found~~ was directed to How To Ask Questions The Smart Way.

One of the things that became evident quickly is that I didn't always have to hit Send (email) or Submit (forum post), just the act of writing it out forced me to think through the issue and more often than not, I would figure out the answer on my own.

Flash forward five or six years and I started to receive all these questions, either in person or via chat. "Send me an email" was usually my response, especially if I was in the middle of something (see: Context Switching). I was happy to help, just not at that moment. With email, I could get to it when I got a break (or needed one).

One of my favorite people, Jason Baer, who has worked for RittmanMead for the last couple of years, took this to heart. We started working together in December of 2009 and he would pepper me with questions constantly. I could never keep up. "Email the question Jason."

I didn't realize it, but I started getting fewer and fewer emails/questions from him. He began to figure them out on his own. It seemed most of the time he had just missed something, other times he just figured out another way to do something.

Jason is a smart guy. I think I'm smart. Sometimes it's just easier to ask the question without thinking it through. In fact, I do that quite a bit on The Twitter Machine ™, especially those errors that I seem to know but just don't have the bandwidth to research (think DBA type questions). I believe the types of questions that ~~should~~ must be written down are those that deal with Approach (design, architecture, etc). Any of those ORA errors better come along with a link to the error code in the documentation and some proof that you've researched it a bit yourself...but then that's getting into How To Ask Questions The Smart Way.

Go out and practice. Next time you have a (technical) question for someone, anyone, write it down and see what happens.

ORACLENERD

The System Still Works: The Algorithmic Arms Race

The $1 Billion Query: Mining for Acuity

The 74,000-Code Moat

The Press Release: Setting the Narrative

The System Still Works

When the Guardrail Stops Guarding

The Silent Failure: Enforcing Neither

Temporal Boundaries Trump Hardcoded Strings

The Metadata Leak

The Portable Rule

Scoped Optimization and AI

Every JSON Column Has a Schema

The Unfalsifiable Blob

Watching From the Outside

Rebuilding the Schema, Badly

Relocating the True Cost (and Closing the Loop)

The Waiver

The Boolean is Still Lying to You: The OLTP Edition

The is_deleted Tragedy: Timestamp Trumps Boolean

The Boolean Pile-Up: Where State Machines Go to Die

The Tri-State Lie

The Indexing Bonus: The Nerd Special

Downstream Devastation: Why the Warehouse Cares

Stop Hiding the Business Process

The Boolean is Lying to You

The OLAP Failure: Redundancy

The "Bit Bucket" Trade-off

"But it makes it easier for the analysts!"

The Semantic Layer: Where Booleans Actually Belong

The Idealistic Layer Boundary

References

How to Answer Questions the Smart Way

The Excavation

The Expert's Burden

Answer Like an API

Reduce Ambiguity

You Can Point a Foreign Key Where?!

The Setup

Why Do We Care?

The Fine Print (Because this is Oracle)

Relational Reality Check

Appendix: Documentation & Structural Foundations

Oracle Database Documentation

Edgar F. Codd & Relational Theory

The System Works

The Death of the API Barrier: From Jargon Intimidation to Result Sets and AI

REST Didn’t Fix the Mindset

PL/SQL Was Already the API (We Just Forgot)

AI Is the New Translation Layer

From OO Friction to Set-Based Speed

APIs Are Just Addresses

The New Era of Data Integration

AI Didn’t Know the Law. It Made the Path Visible.

How I finally appealed a denied insurance claim

It's Been a Minute...

Real World SQL and PL/SQL: Advice from the Experts

Kscope15 - It's a Wrap, Part II

Is MERGE a bug?

Fun with SQL - Silver Pockets Full

The Riley Family, Part III

The Riley Family, Part II

Fun with SQL - My Birthday

Why I'm voting for Danny Bryant and You Should Too

Excel VS BI in Financial Analysis: why the fight was over before it started

#OOW13

DBA or Developer?

Conditional Formatting of Calculated Items in OBIEE 11g

Why it works

Learn To ______ In A Year

Write It Out

The `is_deleted` Tragedy: Timestamp Trumps Boolean