The post Modeling Virus Transmission on an Airplane appeared first on Method and Style.

]]>As a proxy for COVID-19 data, we do have one example of transmission of SARS to air passengers on a flight from Hong Kong to Beijing in 2003 (Olsen SJ, Chang HL, Cheung TY, et al. Transmission of severe acute respiratory syndrome on aircraft. *N Engl J Med* 2003; 349:2416–22). It looks like this:

The infected individual (“Index patient”) is seated in row 14. From inspection it appears the risk of infection depends on the distance in rows from the Index patient, skewed toward rows in front of the patient. There does not appear to be a lateral component to the risk. Ignoring empty seats, and assuming “No illness” whether the passenger was interviewed or not, this diagram implies the risk of infection vs distance in rows from the Index patient. Smoothing the data slightly, we get the table below:

Here *x* is the row number relative to the Index patient, and *y* is the observed risk of transmission for a passenger in row *x*. Graphing this data reveals roughly a bell-shaped curve with a peak around x = -1 or -2:

To model this we will assume the familiar bell curve for a normal distribution, called a gaussian. The formula for a gaussian is

y = A * exp-(½*(x - m)²/s²)

where *A*, the amplitude, is the height of the curve; *m*, the mean, is the value of *x* at the center of the curve; and *s*, the standard deviation, measures the width of the curve.

We want to find the values of *A, m, *and* s* that best fit the observed data. To do that we use a machine learning algorithm called *linear regression, *which finds the best straight-line fit to a dataset. Here the data is clearly not linear, but we will transform the dataset so that it is.

First we take the log of *y*:

log y = log A - ½*(x-m)²/s² = (log A – ½*m²/s²) + m/s² * x – ½/s² *x²

You could say this equation is still not a linear function of *x*, but suppose we define a new variable *x2* equal to the square of *x*. Now we have a linear equation for *log y* as a function of *x* and *x2*:

log y = p0 + p1*x + p2*x2

Here *p0, p1, *and *p2* are the coefficients determined by linear regression. We can convert those back to *A, m, *and *s* using the relationships:

p0 = (log A – ½*m²/s²) p1 = m/s² p2 = – ½/s²

Effectively we have transformed a curve in 2-dimensional space into a straight line in 3-dimensional space, a common technique used in machine learning models. If that is a little hard to bend your brain around, don’t worry about it.

I used Knime to find the linear regression values for *p0, p1, *and *p2*, and create a PMML model for it. Knime is a free business-friendly tool for creating machine learning models as a graphical workflow. It can also save models in PMML format that can be executed by Trisotech Decision Modeler or Workflow Modeler. Knime itself can also execute the model, post-process and plot the results, but the post-processing and plotting are not captured in PMML.

We feed the workflow with a CSV table of *x, x2, *and *log y* and it returns *p0, p1, *and *p2*.

The linear regression returns *p0* = -1.272, *p1*= -.126, *p2*= -0.032, which is equivalent (in the original formula for *y*) to

*A*=0.3174, *m*=-1.97, *s*=3.95.

Now we can use DMN to execute the PMML, returning log(*y*) given the relative row number *x*, and then calculating the probability *y* = exp(log(*y*)). We take the generated PMML model and execute it as a BKM.

The BKM *fit* invokes the PMML regression model. Its parameters *x*, the relative seat row, and *x2*, the square of *x, *are passed by the invocation *SARS transmission probability*. The BKM returns the *log* of the transmission probability *y, *so the final result box converts that to *y*.

In the graph below, the observed data is shown in blue, and the fitted gaussian is pink. The gaussian underestimates the risk for passengers 1 to 3 rows in front of the Index patient and overestimates it for passengers 0 to 2 rows behind, but the fit is reasonable given the limited data available. It says that the peak transmission risk is in the range 30-40% in the 1-2 rows in front of the Index patient, and the standard deviation of the gaussian is around 4 rows, i.e., from 6 rows in front of the Index patient to 2 rows in back.

Of course, for a simple regression like this, we don’t need to invoke a PMML model. We could just use a literal expression with the fitted coefficient values:

y = exp(-1.272 - 0.126*x - 0.032*x²)

This will give the same result. I show the PMML method because it illustrates how much more complex machine learning models can be applied to the problem and invoked as a decision service.

Officials are scrambling now to try to collect similar data for COVID-19, made more difficult by political and commercial risks for the government and the airlines. No doubt eventually we will have the data. For now, I’m sitting in the back of the plane.

The post Modeling Virus Transmission on an Airplane appeared first on Method and Style.

]]>The post Using Decision Services appeared first on Method and Style.

]]>The introduction of *decision service* as a formal element in DMN came about in a surprising way. The spec’s only real example of a decision model, a lending decision, featured a DRD meant to be executed in two or three steps. With reference to the diagram below, depending on the outcome of the decision *Strategy*, the decision *Routing* would be executed, and based on its outcome, a human decision, *Adjudication*, might be required.

While DRDs meant to be executed in multiple steps are technically allowed, it’s rare to see them. One reason is that a DMN model meant to be executed all at once is instantly deployable as a REST microservice. In other words, a DRD containing no human decisions and in which all the input data is available up front implicitly defines a stateless decision service. In fact, the Trisotech tool allows you to publish such a DMN model as an executable cloud service in one mouse click.

However, the Lending example in the DMN spec was not meant to be executed all at once. The input data *Bureau data*, presumably because obtaining it incurs a cost, is not available up front, since depending on the outcome of *Strategy* it might not be needed. And the human decision *Adjudication* is only performed if *Routing* cannot make an automated determination.

None of that is obvious from the DRD, but this DRD is not meant to stand alone. It is meant to be paired with a *process model* – a decision flow – that executes the decisions *Strategy*, *Routing*, and *Adjudication* in the proper sequence and gathers the needed input data only when necessary. Putting aside the human decision, what the decision tasks in that process invoke are two *decision services*, one with *Strategy* and *Bureau call type* as the output and the other with *Routing* as the output. The original idea was to define the services by drawing a rounded rectangle enclosing the decisions in each one. Information requirements going into that rectangle would identify the service’s input parameters. In practice, defining the service boundary this way requires a polygon with many twists and turns. Note that some decisions, like *Application risk score* and *Required monthly installment,* are included in both services. That means these polygons could possibly overlap, so a single decision could be enclosed by more than one.

Defining decision service boundaries in this way isn’t really practical. If you need to extract a portion of a DRD as a decision service, Trisotech just asks, what are the output decisions – there could be more than one – and the encapsulated decisions, and the tool deduces the parameters. Once defined in this way, we can represent each decision service as its own separate DRD, using the *expanded decision service* shape, a rounded rectangle with a horizontal dividing line. Decisions above the line are *output decisions*, those below the line are *encapsulated decisions*, and information requirements from outside the service shape are the *service input parameters*. The three dots below the decision name means that additional details – such as BKM invocations – are not shown in the diagram.

Although this was how decision services found their way into DMN, it is not their typical use. Apart from the “whole model” service implied by any stateless decision model, the most prevalent use of decision services involves execution by another DMN element: a decision, BKM, or decision service. Like BKMs, decision services are decision logic modeled as a *function*, meaning they are *invoked* by passing in values for their parameters and receiving back values of the output decisions.

The expanded decision service shapes shown above are used to *define* the logic of a decision service, typically in its own DRD. To *invoke* a decision service within a DMN model, the invoking DRD contains the *collapsed decision service* shape, a small rounded rectangle with a thick border and a plus sign at bottom center. Like a BKM, the collapsed decision service is connected to the invoking element with a *knowledge requirement,* the dashed arrow. Also like a BKM, a collapsed decision service is opaque: You cannot see the logic inside it. Instead, it is hyperlinked in the tool to the expanded service DRD.

The benefits of using decision services in a decision model closely parallel the benefits of using BKMs, including business-friendliness, logic reuse, and delegated implementation.

The business-friendliness benefit arises because decision services do not have BKMs’ constraint that its logic must be a single value expression. When BKM logic is complex, this requires a context, but the logic of a decision service is modeled as a DRD. Given a choice, non-technical modelers seem to prefer DRDs over contexts.

Any time some bit of logic is repeated – such as iteration of rows to construct a table – DMN requires use of a function, either a BKM, context entry function definition, or a decision service. Often creating that function as a decision service is the most business-friendly way. To illustrate this, let’s revisit the decision model from last month’s post. Given a table of mortgage loan products differing in interest rate, “points”, and other fees, in combination with a specified purchase price and down payment, create a table showing the monthly payment for each loan product. The DRD from last time looked like this:

In last month’s post, we modeled *Mortgage payments by Lender *as a context:

But now let’s say the modeler has created this DRD as decision requirements but is unable to create the logic of *Mortgage payments by Lender *himself. He knows it requires iteration over *Loan Products* but is unsure how to calculate the monthly payment for a single loan product, what we called *Table Row* in the previous post. So he delegates the implementation to a more technical modeler in the organization. Previously, *Table Row* was a context entry function definition that creates a row of the output table given the purchase price, down payment, and a single row of the loan products table. Since it nests another context *contextRow*, business stakeholders might find it hard to decipher. It would be more understandable to them if modeled as a decision service.

It is quite common for a decision service requested in this way to be created in a separate DMN model and then *imported* into the original model for invocation. So let’s see what that looks like.

In *Table Row Service*, we’ll keep the same three parameters as in the original context entry function definition: *Purchase Price, Down Payment, *and *Loan, *where *Loan* is a single row of *Loan Products*.

Decisions below the line, called *encapsulated decisions*, are not returned in the service output. Here to keep things really business-friendly, we’ve split all the original context entries into separate encapsulated decisions with simple literal expressions:

We want our service to return two components, *Lender* and *Payment*. We could do this in two ways. We could give the service a single output decision *Table Row* with two components, modeled as a context with two context entries and no final result box. Alternatively, we could give our decision service two output decisions, *Lender* and *Payment*, as shown in the DRD above. A decision service with multiple output decisions creates a structure with one component per output decision, which is exactly what we want. Either way, the invoking DRD looks like the DRD below, and the call from *Mortgage payments by Lender* returns a structure with the same two components, *Lender* and *Payment*.

Now, in the original DRD, we need to import *Table Row Service*. The mechanics of import are tool-dependent. In Trisotech we click *Include* and navigate to the model and then the service we want to import. Imported models by default are prefixed with the name of their definition model, but we can edit it to something shorter, like *service*. This prefix is only used in the importing model.

This example combines two uses of decision services. First, it supports delegated implementation of decision logic, implemented as a function invoked by the original decision. Second, it supports reusable logic, such as an iterated expression, which also requires a logic function. In both cases, the larger and more complex the logic of the function, the greater the business-friendliness advantage of a decision service over a BKM. Also, unlike a BKM, a decision service can also be invoked by an external client, such as a BPMN process.

If you want to do real DMN modeling, you need to be able to use BKMs, contexts, and decision services, and know which one to choose in any situation. Our DMN Basics and DMN Advanced training teaches you that. Each class gives you 60-day use of Trisotech’s DMN Modeler for hands-on exercises and post-class certification. Check it out!

The post Using Decision Services appeared first on Method and Style.

]]>The post Use Contexts to Simplify Your Decision Models appeared first on Method and Style.

]]>Unlike BKMs, contexts are not DRG elements, meaning distinct shapes in the DRD. Instead, a context is a type of *boxed expression*, a standard tabular format for the decision logic of a decision or BKM. Contexts provide three essential benefits:

- They simplify complex decision models, providing a compromise between models with simple-looking DRDs containing complicated literal expressions and DRDs containing many decisions, each with simple decision logic.
- They allow you to model complicated decision logic – even a whole DRD – as a single value expression. This is critical if you want to encapsulate that logic in a BKM.
- They allow you to construct structured data out of simple values.

We’ll illustrate all of these in this post.

Even though contexts are primarily about managing complexity, we will illustrate their use with something simple, the same example we used last time: calculating the monthly payment for a home mortgage. As before, typically this would be just a small part of some larger end-to-end decision.

When you shop for a mortgage, it is common to see loan products specified as a combination of the annual interest rate, an origination fee called “points”, and possibly additional fixed fees. Points are a percentage of the requested loan amount that is paid up front. Typically the points and fees are simply added to the requested amount and paid off over time along with the rest of the loan.

Even something as basic as this can be modeled in a variety of styles, according to the tastes and skills of the modeler. For example, James is a business analyst who likes the DRD to break down the logic into lots of simple decisions. He modeled the mortgage payment logic like this:

Each element in James’s model has a very simple value expression but it requires four decisions, plus a BKM *payment* for the amortization formula. Since this calculation represents only 5-10% of the expected end-to-end decision logic, that projects out to a complete DRD with dozens of elements. That could be a problem, because beyond 20 or so elements in the DRD, the decision model becomes difficult to maintain.

Edson is another member of the team. He’s more technical and prefers his logic to get right to the point. He modeled the same scenario in a single literal expression:

Edson’s DRD has just a single decision node and no BKMs, but that literal expression has a lot going on. If we were to model the end-to-end logic in this style, the DRD won’t have too many elements, but their decision logic likely will be too hard for most other stakeholders to understand… and too hard to debug, even for Edson. Both versions of the logic give the same answer for *Monthly Payment – *$1427.16 *– * but they represent an extreme difference in modeling style.

Contexts provide a compromise, a way to reduce the number of decision nodes in the DRD while making their decision logic easy to understand. A context effectively collapses a DRD fragment into a single two-column table representing the *entire value expression of a decision or BKM*. Each row of the table is called a *context entry.* The first column is the *name* of the context entry, a variable used in the context’s logic. The second column is its *value expression*, usually a literal expression but possibly any other boxed expression, including decision table, invocation, or even another context. The last row of the context has no name, just a value expression. That is called the *final result box*, holding the value of the context as a whole.

Each context entry defines a local variable, meaning it can be referenced by subsequent context entries in the context and by the final result box, but not by expressions outside the context. While they are not outputs of the overall decision logic, context entries allow the final result box logic to be simpler and easier to understand.

With *Monthly Payment* modeled as a context, the DRD still has just a single decision node, but its logic is now easier to understand.

The context breaks out the logic into five context entries, each with the simple decision logic favored by James. It could have fewer context entries, of course, with slightly more complex literal expressions in some of them. Note how, for example, the first context entry, *Requested Amount*, is referenced by the value expression of the second context entry, *Points, *and so on, continuing down to the final result box.

Also note that not all the value expressions are literal expressions. Number 4, *payment*, is a rarely seen type called a *function definition*. Normally a function definition is modeled as a BKM, but it can also be modeled as a context entry. Here we use it in place of the BKM in James’s DRD. Context entry 5, *Monthly Payment Exact,* is a boxed invocation calling the function *payment* described above. It has many digits after the decimal point, so the final result box simply applies the function *decimal()* to it, rounding to two places. Again the calculation gives the same result, $1427.16.

That illustrates the first benefit of contexts, simultaneously reducing the complexity of both the DRD and the decision logic of each node.

Now consider the scenario we discussed in last month’s post: We have an input data table of bank loan products, each specified by *Rate, PointsPct, *and *Fees*, and would like to compare their respective *Monthly Payments by Lender *by iterating over the input table *Loan Products*. That involves iterating a call to the BKM *Table Row* to generate each row of the of the output table, which simply lists the lender name and the monthly payment. The latter is a call from *Table Row* to the BKM *Monthly Payment*.

In this example both *Table Row* and *Monthly Payment* are modeled as contexts, illustrating the second and third benefits of this expression type. *Table Row* is a context with no final result box. In that case the *context output is a structure,* with one component per context entry. Its datatype *tLenderPayment* has two components, *Lender* (Text) and *Payment* (Number), and the context entries of *Table Row* reflect that. In fact, this is the normal way to create structured data one component at a time. Contexts with no final result box are especially useful when working with table data. Here the context entry *Payment* is modeled as a boxed invocation of the BKM *Monthly Payment*.

The BKM *Monthly Payment* is a context with a final result box. It looks the same as in the previous example, except now as a BKM it has parameters: *Purchase Price, Down Payment, *and *Loan*. The context entry *Payment* in the BKM *Table Row* invokes this BKM.

Executing the model gives the results below:

This example illustrates the second and third benefits of a context.

- Since
*Monthly Payment*is a BKM, it must be modeled as a single value expression, not a DRD fragment. Contexts provide a way to reduce any decision logic, no matter how complex, to a single value expression. - Creating the table
*Mortgage payments by Lender*one row at a time uses a BKM modeled as a context with no final result box to construct each row. The BKM*Table Row*is an example of that.

You might notice that this model is still not a single value expression, since it requires two BKMs in addition to the decision. Could we do it all with just one decision and no BKMs? With a context, we can!

The trick is to turn the BKMs into function definition context entries, like this:

Here the context has two context entries, both modeled as function definitions, plus a final result box that iterates over *Loan Products*. The context entry *payment* is just the amortization formula, formerly a BKM. The context entry *Table Row i*s also a function definition. It includes a context entry *contextRow* modeled as a context with no final result box – so it creates the two-column structure. * Table Row* also contains context entries not in the output that create local variables to simplify the decision logic. Since we want *Table Row* to output only the two-column structure, its final result box just points to *contextRow*. This does the trick; now the whole logic is a single value expression!

Contexts should be a part of every DMN modeler’s arsenal of tools. You can learn how to use them in my DMN Method and Style Advanced training. It’s unfortunate that some vendors that claim DMN conformance do not support them. Every year in the Revision Task Force they tell us, “Business users don’t like the tabular format; they like DRDs.” But as in the case of BKMs, these vendors typically support only the decision requirements portion of DMN – the DRDs – and offload the decision logic portion to their proprietary rule language. That violates the spirit of a decision language standard, and it’s unnecessary.

If your DMN tool does not support contexts or BKMs, please urge them to do so!

The post Use Contexts to Simplify Your Decision Models appeared first on Method and Style.

]]>The post Handling Complex XML Input in DMN appeared first on Method and Style.

]]>Trisotech DMN Modeler can automatically create a FEEL item definition (datatype) from XML schema and map XML input data to FEEL on execution. But if we do that directly with the MISMO XSD, we wind up with FEEL elements that – like the original XML data – are effectively impossible to use in decision logic. In the referenced post I solved that problem by mapping MISMO XML to a more DMN-friendly XML schema.

I used XSLT, an XML mapping language, executed as a service in a BPMN model.

Although XSLT is programming, you can generate even complicated mappings graphically in tools like Altova Mapforce. However, to be honest, you need to be pretty familiar with XPath and MISMO to do it. So I can understand why that solution might be unattractive to someone in the mortgage industry who wants to use DMN. They would prefer to create decision models that operate directly on MISMO XML.

It’s not pretty, but you can do it. Even better, you can encapsulate the ugly part in a decision service, created once and reused in any MISMO decision model. My updated solution makes use of Trisotech’s MISMO Accelerator, which lets you import individual container elements of the MISMO schema rather than the whole thing. That helps a bit, but not as much as I’d hoped. In reality, MISMO data is sparse, meaning in a typical instance only a minuscule fraction of the XML elements are present. In the XML instance they are simply omitted, but in the FEEL item definition generated from the XSD they are all present and assigned the value null at runtime. So, even with the Accelerator, the FEEL input data element is huge, over 10 MB, almost all components null.

The scenario is the same as last time. Ken Customer wants to borrow $300,000 on a $340,000 purchase of a single-family home. Our decision logic uses Fannie Mae’s published manual underwriting rules applied to the data on Ken’s URLA form in MISMO XML format. As explained previously, the logic determines the minimum credit score required to be eligible for purchase by Fannie Mae. The key inputs to *Min credit score* are the loan-to-value ratio *LTV*, Ken’s debt-to-income ratio *DTI*, and *Reserves* – his liquid assets after closing, measured in months of housing expense (*Qualifying PITIA*). But unlike last time, now the decision model operates directly on MISMO data.

The decision model can be deployed to the Trisotech cloud as a decision service that accepts XML input in MISMO format. The model must have a single input data element, and its name must be the name of the MISMO XML root, *MESSAGE*. But its datatype is not the one generated by the tool by converting MISMO *MESSAGE*, which would be the entire MISMO schema. Instead tMESSAGE is manually constructed from the seven container elements selected using the Accelerator: *ASSETS, COLLATERALS, EXPENSES, LOAN, LIABILITIES, PARTY, *and *RELATIONSHIPS*, and omitting all the others. In this way, the XML input is valid per the full MISMO schema but contains elements of those seven containers only.

The only decision that operates on *MESSAGE* is called *Mapping*, and it simply passes *MESSAGE* to the decision service *MapMISMO*. *MapMISMO* is the DMN equivalent of the XSLT service mentioned earlier. It maps MISMO elements – here in FEEL, not XML – to a more DMN-friendly data structure. *MapMISMO* is shown below:

The decisions labeled in all caps connected to the input data *MESSAGE* are the MISMO container elements selected from the Accelerator. They simply drill down from the MISMO root to the selected container. For example, *ASSETS* is shown here:

The other decisions are the FEEL variables used in the main model decision logic. I will show just a bit of the mapping logic to give an idea of the expressions involved. *Borrower1* is a fairly large context. The first several context entries create local variables used in the final result.

*theParty* selects the MISMO *PARTY* element representing *Borrower1*. *BorrowerRole* selects that *PARTY*‘s *BORROWER* container. *CurrEmployer1* selects the first *EMPLOYER* of that *BorrowerRole* with a status of “Current”. You get the idea… it’s a lot of filter expressions.

To get the Borrower’s current income, itemized by type (Base, Bonus, etc.), we need to create a join using MISMO’s *RELATIONSHIPS* element. The join returns a list of income item ids for the Borrower’s current employer:

The expression above has one tricky part: Because a filter always returns a list, the income items associated with a particular Employer is a list of lists, so we need the FEEL *flatten()* function to make it a simple list.

We also need to define a *function* – a context entry that acts like a BKM – that returns a row of the income item table, including the income type and amount, given its *id*. This is a context with no final result box, one context entry per column of the table:

To generate the table, we need to iterate the function call over the list of income item *id*s :

This gives you the flavor of the mapping logic. It’s just FEEL, nothing fancy, filters and occasionally a join, table row context, and iteration. There is a lot of detail to it, but by creating the mapping as a decision service, you just need to define it once. Then you can import and reuse it in a wide variety of decision models operating on MISMO data.

Everything you need to know to create decision models like this is explained in my DMN Basics and DMN Advanced training, which includes 60-day use of the Trisotech tool plus post-class certification. Understanding the MISMO data model… that’s up to you.

The post Handling Complex XML Input in DMN appeared first on Method and Style.

]]>The post Why Does DMN Have BKMs? appeared first on Method and Style.

]]>It turns out there are a number of excellent reasons to use BKMs. After reading this post, maybe you’ll use them more in your own decision models.

Let’s start with a simple decision model. This typically would be just a fragment of a larger decision, but let’s just focus on this small part, calculating the monthly loan payment for a home mortgage. Here is the DRD:

The input data *BankX Loan* has three components:

The decision *Mortgage payment* simply calculates the monthly payment based on those component values using the well-known amortization formula:

Wow, that’s a mouthful! Yes, the formula is “well-known” in the sense that it’s used all the time in mortgage lending, but do we expect a decision modeler to be able to enter it from memory, or even type it in correctly? Maybe not. Let’s say this decision is used by a mortgage broker who is dealing with a number of lenders. BankY’s data looks like this:

It’s the same information BankX uses, but BankY uses different names: *Principal* instead of *Amount*, *Rate* instead of *Interest rate*, etc. The amortization formula for BankY looks different because it has to use the new names:

This simple example points out several problems where BKMs can offer some help. First, the decision modeler may know that a formula exists to calculate the loan payment given the loan amount, interest rate, and term, but has no idea what it is. Second, even knowing the formula, the FEEL expression involves the specific names of the loan components, which differ from lender to lender. And third, you want this calculation to work exactly the same way with every lender. Deep down, the formula is the same, even though the names of the variables change from one lender to the next. You’d prefer to be able to reuse the formula in any decision logic, no matter what the input data elements in the DRD are called.

Fundamentally this is what a BKM does. It expresses decision logic as a *parameterized function*. The names of the parameters – the inputs to the logic – are defined by the BKM itself, not by the particular decision model that uses it. The decision model that uses it simply “invokes” – that is, calls – the BKM, passing values to each of its parameters and receiving the BKM result back in return. That solves all three problems above.

- The modeler who knows what the decision requirement is but does not know how to implement it may incorporate that implementation, created by someone else, a subject matter expert, in the decision. The modeler does not need to know how to develop the decision logic, or even how it works. He or she simply needs to know how to invoke it, and this – as you’ll see – is very easy. That is part of what DMN’s originators had in mind when they included BKMs in the standard. More on that later.
- Second, the FEEL expression in the BKM is independent of the variable names used in each calling model. That allows the formula to be shorter and easier to understand.
- And third, the BKM by definition works the same in every decision model that uses it.

Here are the DRDs for BankX and BankY using BKMs:

They look the same, and the BKM *payment* is exactly the same in both:

What you see here is the boxed expression for a BKM. The code F means FEEL, the default. A BKM could also invoke external logic, specifically Java (code J) or PMML (code P). To the right of that are the three parameters. Here they are *p*, the principal or loan amount; *r*, the loan rate expressed as a decimal not a percent; and *n*, the number of months in the loan term. We could name them whatever we want. They are used only by the BKM’s internal logic. The amortization formula, expressed in terms of *p*, *r*, and *n*, is shorter than we saw previously. The *decimal()* function wrapping the arithmetic expression simply rounds the result to two decimal places. This formula is used for both BankX and BankY.

What is different in the two DRDs above is the invocation of the BKM performed by the decision *Mortgage payment*. The invoking decision simply has to map its inputs – here the components of either *BankX Loan* or *BankY Loan*, to the BKM parameters *p*, *r*, and *n*. There are two different ways to do this, and both are easy.

The first way, called *literal invocation*, doesn’t refer to the parameter names at all, just their *order* in the BKM definition. As you see above, *p* is the first parameter, *r* the second, and *n* the third. So for BankX’s *Mortgage payment* we have this:

Literal invocation means a FEEL literal expression with the name of the BKM followed by its *arguments* – values of its parameters, in the correct order – in parentheses. So here the first argument, *BankX Loan.Amount*, is mapped to parameter *p*; the second argument, *BankX Loan.Interest rate*/100, is mapped to parameter *r*; and the third argument, *BankX Loan.Term*, is mapped to parameter *n*. We had to divide by 100 in the second argument in order to convert the rate expressed as percent to a decimal value. If we used literal invocation for BankY, the arguments would reference the variable names used by that lender.

But for BankY, we won’t do it that way. Instead of literal invocation, here we illustrate *boxed invocation, *a standard tabular format for invoking BKMs. It’s a two-column table with the parameter names in the first column and the mapping expression in the second column. It looks like this:

The name of the invoked BKM, *payment*, is at the top. Below that is the two-column table, the BKM parameter names on the left, and the mapping expression for each on the right. The boxed invocation is maybe easier for beginners to understand, but both forms of the invocation work the same way and give the same result on execution. For a 30 year loan of $300,000 at an annual rate of 4%, the payment is $1432.25.

Now if you’ve ever shopped for a home loan, you know it’s a little more complicated than this. In addition to the annual loan rate, lenders often charge *points* – a percentage of the requested loan amount – and possible a fixed *fee*. In most cases, these extra amounts are amortized along with the original requested amount, increasing the loan amount. So in reality, comparing the monthly payment between lenders requires taking this into account as well.

So here is a more complex DRD that involves multiple BKMs. The decision *Mortgage payments by Lender* returns a table of monthly payments by lender by invoking BKM *Lender payment*, which in turn invokes two BKMs, *principal* and *payment*. The connection of the called element to the calling element is indicated in the diagram by the dashed arrow, called a *knowledge requirement*. Here input data *Loan Products* is a table of lender offerings, specifying the interest rate, points, and fees:

Its datatype tLoanProducts is a collection (list) of the row type tLoanProduct, with the five components shown here. Here we see another use of BKMs, iteration over a list. The decision *Mortgage payments by lender* is also a table, one row per row of *Loan Products*, simply showing the lender name and the calculated monthly payment given values for the *Purchase price*, *Down payment*, and *Loan Product* details. We need to create this table one row at a time using iteration:

Iteration uses the FEEL syntax

for [range variable] in [list variable] return [expression of range variable]

where the range variable is simply a name meaning one item in the list – here one row of *Loan Products, *and the expression is a literal invocation of the BKM *Lender payment*. Note that the first argument is the range variable *loan*, again meaning a row of *Loan Products, *a structure with the five components shown above.

Here you see the definition of the BKM *Lender payment*. The parameters are *Loan*, *Price* and *Down*. Note the datatype of each parameter must match the type of the arguments passed in the invocation. This BKM returns a row of *Mortgage payments by lender*, which has two components, *Lender* (the lender name) and *Payment*. The way to create such a structure is with a *context*, another boxed expression type, having a row (context entry) for each component of the output, its name in the first column and value expression in the second column, and an empty final result box (the bottom row, labeled Result). The component *Lender* is taken from *Loan*, which is a row of *Loan Products. *The component *Payment* has a more complicated value expression, illustrating what you can do with BKM invocation. It invokes the BKM *payment*, and for the first argument invokes the BKM *principal*, which calculates the loan amount. In other words, *Mortgage payments by lender* invokes the BKM *Lender payment*, which in turn invokes the BKM *payment* and – as an argument of payment – the BKM *principal*.

If you don’t like the complicated nested literal expression above, you might find nesting boxed invocations more understandable if less compact:

The BKM *principal* calculates the total loan amount from the price, down payment, lender points and fees, and this is used in turn to calculate the monthly loan payment for that lender.

It can make your head spin, but when working with tables such use of BKMs is not uncommon. Let’s try it out. Here is a short table for BankX, which offers a 4% loan with no points or fees, and BankZ, which offers a 3.875% loan with 1% points and $499 fees. Which has the lower monthly payment? It’s not obvious from inspection.

When we run the model with *Purchase Price* $375,000, *Down Payment* $75,000, we get this:

BankZ has a lower monthly payment, but only slightly.

When dealing with lists and tables, BKMs are indispensable. They let you work on each item in the list or row in the table at a time, and iterate the result.

Many of these BKM benefits arise from its nature as *reusable* decision logic. In fact, it is common to save BKMs in a library and simply *import* them into any decision model that needs that bit of logic. Nevertheless, the creator of the BKM idea has always insisted that his motivation for BKMs was never reuse, but *methodology*. This goes to a deeper issue, a somewhat divisive one within the decision management community. I’ll address it briefly here and possibly return to it in a future post.

BKMs originated in a FICO methodology called Decision Requirements Analysis. The idea is that a complex decision model, as expressed in the DRD, requires”knowledge” not available to the DRD modeler but instead distributed across a variety of subject matter experts in the organization. Thus the DRD modeler, capturing the end-to-end view, defines only the *decision requirements*, the names of each decision node, its datatype, and the names and types of its inputs. The actual *decision logic* generating the output from the inputs is implicitly created by someone else, a subject matter expert in that specific part of the logic. BKMs allow the DRD modeler and BKM authors to work independently and glue their contributions together using invocation.

I think this is why all the decisions in the spec Chapter 11 example have attached BKMs. The example is a little misleading in that the BKM logic is not especially reusable or too difficult for the modeler to figure out for himself. It simply illustrates the structure implied by that methodology.

Hmmm… OK. But why not simply incorporate the decision logic directly in the decision node? That’s what the FEEL language, designed for business users, was expressly intended to do! The unspoken part of the above methodology is that these subject matter experts are not using FEEL. They are using a programming language like Java or a proprietary rule engine language. Encapsulating that programming in a BKM allows FEEL – the language of the DRD – to be mapped to the programming language. This separation between decision requirements created by business and logic implementation created by programmers is what DMN supposedly was trying to avoid. The key idea – the reason why FEEL has names with spaces and other “business-friendly” attributes – was always that business wants to “own” its decision logic. That means being able to create and test it themselves. The methodological rationale for BKMs at least gets you halfway there. While implementation may be delegated to programmers, encapsulating it in BKMs at least allows business to test the end-to-end DRD logic themselves.

That is in contrast to some tools that create DMN-like DRDs but simply link decision nodes directly to a proprietary implementation. Those are the tools that claim to be DMN but don’t support BKMs at all. To me, that’s using DMN simply as a RFP checkoff item, not as a true standard. In those tools, business is confined to its traditional role of creating requirements handed off to programmers. I’m not sure how that is better than doing the whole thing in the proprietary business rule environment.

Whether decision logic implementation is done in FEEL by non-programmers or delegated to programmers in some other language, BKMs in the end allow business to own the end-to-end decision logic and test for itself that it is valid and complete.

The post Why Does DMN Have BKMs? appeared first on Method and Style.

]]>The post Helping the Mortgage Industry Go Digital appeared first on Method and Style.

]]>Their 2018 survey of lenders revealed the following leading challenges to adoption of new technology:

- Lack of industry-wide data standards (46%)
- Inflexible processes (38%)
- Lack of industry-wide system/platform standards (26%)

Fannie’s white paper focuses primarily on the data standards problem and points to recent gains in that area spurred by MISMO, the data standards arm of the US Mortgage Bankers Association. These include a logical data model and XML schema for mortgage-related data elements. MISMO data serves as the basis for things like the Uniform Mortgage Data Program and Home Mortgage Disclosure Act, as well as the new Uniform Appraisal Dataset, which enables automated acceptance of appraisals in Fannie’s Uniform Collateral Data Portal. Not discussed in the white paper, but nearly as important as standardizing the data, is MISMO’s recent adoption of BPMN and DMN standards for process and decision modeling. BPMN allows the definition of new, more flexible processes. DMN allows decision logic to be deployed as microservices behind REST APIs, emerging as the next-generation system/platform architecture. Thus in combination, MISMO data, BPMN, and DMN address all three of the leading barriers to going digital. In this post we’ll see how they can be used together.

It’s one thing for standards to exist or to be “adopted” by an industry association. It’s quite another for the companies participating in the industry to actually incorporate those standards in their work. As Fannie notes at the outset, lenders have been slow to break free of their legacy systems and replace them with new digital technology. In my own interactions with these folks, for example, in MISMO’s Decision Modeling Community of Practice, I would say that the biggest obstacle is simply lack of awareness of how these new technology standards can be applied and work together. That’s unfortunate, because MISMO data, BPMN, and DMN can be applied in digital mortgage apps *today*.

Let’s start with MISMO data. MISMO’s logical data model standardizes the names of data elements used in all aspects of the mortgage business – origination, servicing, securitization, etc. This not only eliminates terminology differences between various lenders and systems, but prevents the same element having different names and attributes when used for different purposes across the enterprise. There is great benefit in that, but the downside is that the resulting schema is huge, so XML data conforming to that schema can be difficult to access and manipulate. MISMO is working on reducing that difficulty, but for the moment what we have is a mortgage data standard based on a large unwieldy schema. Still, it’s a system-independent standard and we can work with it.

The key advantage of having such a standard is interoperability of mortgage data across participant systems. For example, it has allowed Fannie and Freddie Mac, the other major GSE, to unify their loan application forms in a common Uniform Residential Loan Application, or URLA. Each field in the form is linked to a MISMO data element, and Fannie and Freddie require that lenders deliver loans to them as MISMO-conformant XML. That means lenders’ Loan Origination Systems must be capable of outputting loan application information as standard MISMO XML. New applications and microservices can then make use of this data standard to streamline customer interactions.

For example, let’s suppose a lender wants to determine the maximum loan amount available to a prospective borrower consistent with the lender’s ability to sell the loan to Fannie. With DMN you can model and execute that logic. You can use BPMN to handle the mapping between MISMO data, URLA form data, and the DMN decision logic, and deploy the whole thing as a microservice. And to top it off, with tools like Trisotech’s Digital Enterprise Suite, it can all be done by subject matter experts who are not programmers. Let’s see how this works.

The details of Fannie’s Automated Underwriting System are undisclosed, but they publish them for Manual Underwriting, so we’ll use that for this example. For simplicity, we will be concerned with 30-year fixed rate mortgages for purchase of a single-unit primary residence. Fannie’s eligibility matrix is shown below:

We will assume the lender will approve the loan if it meets these eligibility requirements, which have four inputs: the credit score, the loan-to-value ratio (LTV), the debt-to-income ratio (DTI), and the borrower’s reserves, meaning liquid assets after closing, measured in months of housing expense payments. We can turn this table into a DMN decision table by determining the minimum credit score required for a given combination of LTV, DTI, and Reserves.

Values of LTV over 95%, DTI over 45%, or Reserves less than zero are always ineligible, so the minimum credit score for those is null.

Of course, LTV, DTI, and Reserves are derived values, not raw data reported in URLA. As is usually the case, the bulk of the DMN Decision Requirements Diagram (DRD) is deriving those values from the input data, based on the calculations detailed in the Guide on Fannie’s website. That DRD is shown below:

The top-level decision *Result* simply consolidates reporting of various details of the loan application, including whether the loan is eligible nor not. To assist lenders in implementing the URLA form and MISMO XML data standard, Fannie publishes some test cases, including the one for Ken Customer shown below:

This is just page 1 of Ken’s URLA form, which contains many pages detailing the various sources and types of his income, assets and liabilities, plus details of the loan, property value, and cash required at closing entered by the lender. On delivery to Fannie, this data is formatted as MISMO XML. Ken wants to borrow $300,000 for a $340,000 purchase, and we’ve assigned him a credit score of 660.

In this case, the minimum credit score is 680, so Ken is ineligible to borrow $300,000. Of course, that’s not news. Undoubtedly the lender’s Loan Origination System already knows this.

But let’s say we want to be able to tell Ken the maximum loan amount he would be eligible for given his financial details. That’s more difficult to model, since as the loan amount changes (here we assume the same purchase price), the LTV, DTI, and Reserves all change. In fact, even the loan rate may change. So for that we have the modified DRD shown below:

Now we’re going to vary the loan amount (LTV), and calculate an adjusted interest rate based on the lender’s daily best rate, LTV, and credit score. With the resulting change in down payment and mortgage payment, the model will also recalculate DTI and Reserves. In DMN, even though the logic is detailed, it does not require programming. Using FEEL and boxed expressions, a subject matter expert can enter it directly. For example, below is the decision Qualifying PITIA, Ken’s monthly housing expense payments:

We can test the logic by entering input data values in the form here on the left and running the model. Recall that Ken did not qualify for 300,000, but for a loan amount of 272,000, corresponding to LTV of 80%, Ken’s credit score of 660 is equal to the required minimum, so he is eligible at this LTV. Once testing gives confidence that the model is correct, it can be published in one click as an executable decision service we could call for any borrower given URLA and the current best interest rate.

As a practical matter, we need to take account of the fact that MISMO XML, because of its nature as an enterprise data dictionary, makes expressions referencing specific data elements more complicated than they need to be. We’d like the format of our input data element URLA to be more DMN-friendly, simplifying those expressions. Here you see a side-by-side comparison of a fragment of Ken’s data in the two formats:

As output by the Loan Origination System, Ken’s data is MISMO XML, so data mapping is required. Trisotech automatically creates the corresponding FEEL datatypes when you import an XML Schema and provides a special system task, called Context Parser, to map XML data to its FEEL equivalent. So our plan is to create a DMN-friendly XML Schema for the input data element URLA, map MISMO XML to that, pass that to the Context Parser, which then invokes the DMN.

We can model those steps as a *process* in BPMN and deploy the whole thing as a REST service that takes MISMO data as input and outputs the result of our decision. It looks like this:

The first activity is a service that maps MISMO XML to our DMN-friendly schema. You can write a program to do this, or use a tool like Altova MapForce to create the mapping graphically. The output of that task, here called URLA DMN XML, is passed to the system task *Context Parser*, which converts the XML to FEEL. The decision task *Max Loan amount* simply executes the DMN model we showed earlier, with its three input data elements URLA, LTV, and Best rate pct. The DMN decision *Result* is passed to the process data output, which is also tested by a gateway to provide process end states Eligible and Ineligible.

We can deploy this BPMN process in one click as an executable microservice in the Trisotech cloud. When we run it with an LTV of 80%, Ken is Eligible for the loan. That’s the service output shown below on the left. But if we bump it up just slightly to 81%, as shown on the right, the minimum credit score required jumps to 680 and Ken is Ineligible. This slight increase in loan amount pushes DTI above 36% and also requires private mortgage insurance (PMI). Thus we can report to Ken that the maximum he can borrow for this property is $272,000, or LTV 80%.

Executable modeling by non-programmers is a basic feature of DMN, but BPMN historically has required programming to make it executable. Trisotech, however, has borrowed FEEL and boxed expressions from DMN to let non-programmers create executable processes like this one. Just like invocation mappings in DMN, the tool provides boxed expressions to map between BPMN data objects (variables) and task data inputs/outputs. For example, below you see the mapping from the process data objects LTV, URLA FEEL, and Best rate pct to the corresponding DMN input data elements. In this case the mapping is trivial, but as in DMN, any FEEL expression can be used.

It’s a lot to process, but let’s review what we just showed:

- A DMN decision service, created by a subject matter expert – not a programmer – to determine loan eligibility per Fannie Mae manual underwriting rules (perhaps with modifications) based on URLA data.
- A mapping of URLA data from MISMO XML format to a more DMN-friendly format. This could be created by a non-programmer using a third party graphical tool, or by a programmer as a reusable service.
- A BPMN process, also created by a non-programmer, to convert the MISMO data to DMN-friendly XML and then to FEEL, execute the DMN, and branch based on the decision result.

Assuming the lender’s LOS can output MISMO data, none of this depends on the details of the underlying systems. It is all based on *standards –* for mortgage data (MISMO), decision logic (DMN), and executable processes (BPMN). The whole thing can be packaged and deployed easily as a microservice, and it can be done by subject matter experts without programming.

The tools for the mortgage industry to go digital are here today. Lenders should get to know them!

The post Helping the Mortgage Industry Go Digital appeared first on Method and Style.

]]>The post EU-Rent Customer Support with Method and Style appeared first on Method and Style.

]]>I developed the Method and Style approach to BPMN modeling to address that problem, in particular, the “style rules”, which complement the rules of the spec. They are not official, just what I consider best practice if the goal is to communicate the process logic clearly, completely, and consistently from the diagrams. (Students in my BPMN training must use them in their certification exercise if they want to pass.)

Trisotech BPMN Modeler can check models automatically for style rule violations in one click. On the BPMN ribbon, just click M&S Style Validation. That is really useful, both for improving your process model quality and internalizing the rules.

According to the rules of the spec, the Trisotech EU-Rent examples are valid, but they were not created following Method and Style. The original *Customer Support* model, for example, has many style rule violations. A good way to illustrate Method and Style is to compare the original model with an updated one that follows the style rules.

Version 1.1 of the model is shown below. If we click M&S Style Validation, we see it has 17 style errors. That’s quite a few! In this post, I will explain the issues and fix the model to conform with Method and Style. [Note: To access a particular version, from the File/Open dialog, right-click the model name and select Versions. If you don’t do that, the tool opens the current version.]

Figure 1. Customer Support model (original)

Figure 2. Style errors in the original model

Before dealing with the specific style errors, there are structural issues with this model. A BPMN model may depict interactions between a process and other entities, such as additional processes or abstract external entities. Such a depiction is called a *collaboration model*. In a collaboration, the process and each of the other entities are called *participants*. The term *participant* is often misunderstood by modelers. It does NOT mean the *performer* of a process activity. A participant simply means a party involved in the collaboration. Each participant is represented in the diagram as a *pool*. Participants interact with each other through the mechanism of *message flow*, the dashed arrow in the diagram. This is all from the spec, not Method and Style. It’s just that the spec doesn’t explain it very well.

A process model normally describes a collaboration from the perspective of a single process. That means that participants other than the process are depicted abstractly as *black-box pools*. Black-box pools are empty, identified only by the participant name in the pool header. They contain no lanes, no activities or flow nodes of any kind. Interaction between the process and a black-box pool is visualized entirely through message flows connecting the pool with a process node.

Separate roles or departments within the organization providing the process should be modeled as *lanes* within a single process pool, NOT as separate pools. Lanes identify the *performer* of the activities they contain. Normally a collaboration model contains a single process pool, which may contain multiple lanes, interacting with one or more black-box pools via message flow. The only time a single logical process is modeled as multiple process pools is when the instances of the pools do not have 1:1 correspondence. Let’s leave that topic for a future post.

In this EU-Rent model, the pool labeled *Customer Support* and the pool labeled *2 ^{nd} Level Support Agent* should be

In the Method and Style version of this model there is thus one process pool, *Customer Support Process*, containing two lanes, *1 ^{st} Level Support Agent* and

Many of the style rules concern labeling. Labels are extremely valuable for communicating meaning, but the BPMN spec does not require *anything* to be labeled. What were they thinking? Here are the basic style rules concerning labeling:

All activities – including subprocesses – should be labeled, preferably as an action, verb-object.

Message flows should be labeled as the name of the message, a noun, not as an action (verb) or state (adjective phrase). For example, *Info Request* is a good name for a message flow, not *Send Info Request* (an action) or *Request Sent* (a state). Also, a message flow to or from a child level must be replicated in the parent level, connecting to a black-box pool of the same name. Style rule validation checks for this. When you add a pool to a child level that has the same name as a previously defined pool in the parent level, the tool will ask if you intend these to represent the same participant. In Method and Style, the answer is YES.

Message start events should be labeled “Receive [name of message flow]”. Timer start events should be labeled with the frequency of occurrence. All intermediate events should be labeled.

In Method and Style, end events represent distinct *end states* of a process or subprocess. If a process level has more than one, each should be labeled with the end state name, an adjective phrase, such as *Issue resolved*. If there is only one, it should not be labeled.

In Method and Style, the gates of an XOR gateway represent the *end states of the activity immediately preceding the gateway*. The activity’s end state – an adjective phrase – simply means how did the activity end. Each activity end state corresponds to a different next step following the gateway.

- The end states of a
*task*are indicated by the labels of the gateway following the task. For example,*Get issue description*in the original model has three possible next steps, meaning it has three end states. Thus it should be followed by an XOR gateway with three gates, each labeled with an adjective phrase naming the end state. When there are two next steps, meaning two end states, an alternative labeling is allowed in which the XOR gateway is labeled as a question – typically the success end state followed by a question mark – with gate labels*Yes*and*No*. - When a
*subprocess*has multiple end states, the child level diagram will have a separate end event for each, labeled with the end state name. In the parent level, the collapsed subprocess is followed by an XOR gateway with gates matching those end event labels. Style rule validation checks for this.

Note the XOR gateway labeling rules do not apply to OR gateways. The gates of an OR gateway do not represent distinct end states of the preceding activity.

The gates of an AND gateway or event gateway should never be labeled, nor should the gateway itself. Merge gateways also should not be labeled.

In the original model, the subprocess *Escalate issue* is missing start and end events. This notation, involving so-called *implicit start and end nodes*, is allowed by the spec under certain conditions, but not in Method and Style. In Method and Style, all flow nodes (except event subprocess) must lie on a continuous chain of sequence flows, within a single process level, leading from a start event to an end event.

In the original model, style error [0005] says the model should not have more than 10 activities in a diagram. This one has 11, but the reason here is that there are activities in *Supplier*, which should be black-box. The intent of the rule is no more than 10 activities in a single process level, so when printed it fits on a single page.

Version 2, the fixed model, is shown below. M&S Style Validation shows 0 errors. Both versions of the model are available in the EU-Rent repository. Readers are invited to compare them and explore the working of the M&S Style Validation feature.

Figure 3. EU-Rent *Customer Support* (fixed)

Figure 4. Child level, *Review Issue*

Figure 5. Child level, *Escalate Issue*

The post EU-Rent Customer Support with Method and Style appeared first on Method and Style.

]]>The post DMN, Meet Machine Learning appeared first on Method and Style.

]]>You often hear that the logic of DMN models is explainable or transparent but the logic of machine learning models is opaque, a black box. Decision tables are really the most explainable part of DMN. Their logic is simple because decision table inputs are typically not raw input data but refined high-level features provided as supporting decisions. The top levels of a DRD are thus often decision tables while the bottom levels consist of all manner of value expressions that refine the input data into those high-level features. In contrast, machine learning algorithms, lacking intuition about what the high-level features should be, operate on the raw data directly. This further contributes to the opacity of the logic. We are primarily interested in so-called supervised learning models, which are “trained” on a large number of examples, each containing values of various attributes or “features” and having a known outcome. In training, the model parameters are adjusted to maximize the prediction accuracy. The trained model is then used to predict the outcome of new examples.

As machine learning gains influence on day-to-day decision-making, its inability to explain the reasons for its decisions is increasingly seen as a problem. One answer to the demand for “explainable AI” is DMN and machine learning working together. Although little explored to date, this can occur in a couple different ways:

- Machine learning can be used in supporting decisions to refine raw input data into the high-level features used in decision tables. Even though the machine learning decision logic may be opaque, the top-level decision logic is transparent.
- Some machine learning models can be transformed into DMN models, revealing hidden relationships in the data. This adds intuition to the decision logic which can be fed back to the machine learning model in the form of additional refined attributes. The result is often simpler models and more accurate prediction.

In this post we’ll see both of these at work in a customer retention scenario at a wireless carrier.

You may not realize it, but the ability to execute machine learning models is a standard feature of DMN. From the start, the DMN spec said that in addition to the normal value expressions modeled in FEEL, decisions can execute machine learning models packaged using the *Predictive Model Markup Language*, or *PMML*. PMML, administered by the Data Mining Group, is not a marchine learning algorithm but a common XML format encompassing a wide variety of machine learning algorithms, including decision trees, logistic regression, support vector machine, neural networks, and others. By providing a standard format for a wide range of machine learning model types, PMML enables interchange and interoperability across tools. That means the tool or platform used to execute a machine learning model – for example, a DMN tool – does not need to be the same as the one used to create it. Although Trisotech does not create PMML models, its DMN Modeler can execute them. In this post we’ll see how that works and how DMN can be used to make machine learning human-understandable.

DMN’s mechanism for incorporating PMML is as an* external function BKM*. In addition to a FEEL expression, the spec says a BKM’s logic may be an external function, either a Java method or a PMML model. Trisotech DMN Modeler supports PMML execution in this way.

To illustrate this, consider the scenario of a wireless carrier trying to retain a customer at risk of “churn”, meaning switching to a new carrier. The decision model is shown below:

The input data are *Customer*, with attributes taken from billing records, and *Offers* that could be extended in order to retain the Customer. The DMN decision *Churn risk* executes the machine learning model *Churn prediction PMML, *returning a Boolean value. If *Churn risk* is true, the decision *Best action* invokes a second machine learning model *Best offer PMML* that suggests the Offers likeliest to be attractive to this Customer. Supporting decision *Customer value* computes a value metric for retaining this Customer and for each Offer, the BKM *Offer cost* computes its cost. Based on these inputs, *Best action* selects an Offer, its decision logic explainable even if the logic of *Churn prediction PMML* and *Best offer PMML* is not.

To illustrate how DMN and PMML work together, we will focus on *Churn risk* and *Churn prediction PMML*. The example is an adaptation of a sample app supplied with the open source machine learning tool Knime. The machine learning data is taken from the billing records of Customers who contacted the carrier with an issue. The table below is a fragment of a file containing over 3000 examples. Column 8 “Churn” is the output, with the value 1 if the Customer did not renew the carrier agreement (Churn = true) or 0 if he did renew (Churn = false). The other columns serve as inputs to the machine learning model. Already you see one basic difference between machine learning and DMN: Machine learning models typically operate directly on the raw data found in systems of record, whereas DMN typically refines that data in supporting decisions before making the final decision.

The goal of the machine learning model is to predict, for a new Customer, whether he is likely to churn. And here is a second difference from DMN: The modeler starts with little or no intuition about what combination of these Customer attributes will predict churn. But the machine learning model does not depend on intuition. You just feed it enough examples from past experience and it figures it out.

*Customer* attributes and summed, best predict churn. (Note: This is why machine learning often models Boolean inputs like *VMail Plan* as numbers, either 0 or 1.) Others, like decision tree, define a sequence of yes/no tests, each on a single Customer attribute, that best separate churn from no-churn examples. This branching is continued until there are very few examples on a branch, at which point the majority output value becomes the prediction for the branch. Decision trees are especially interesting for DMN, because the resulting tree model can be converted to a decision table, which can be inspected and its logic understood.

The goal of any machine learning model is maximum prediction accuracy, but what does that really mean? Like many decision models, the outcome of this one is binary, either churn or no-churn. If we call churn “positive” and no-churn “negative” and we call the prediction true if the predicted value matches the actual value, we have the following possible outcomes for each prediction: true positive (TP), true negative (TN), false positive (FP), and false negative (FN).

The metric *accuracy* is defined as (TP + TN)/(TP + FP + TN + FN), but this is not necessarily the most important measure when the examples are highly skewed, meaning many more negatives than positives. *Precision*, defined as TP/(TP + FP), measures what fraction of predicted positives are actual positives. *Recall*, defined as TP/(TP + FN), measures what fraction of actual positives are predicted. The metric you are trying to optimize – or perhaps some combination of these – depends on the circumstances of the decision. The four numbers TP, TN, FP, and FN, comprising the model’s *confusion matrix*, are provided by the tool used to create the model.

Actual / Predicted | 0 | 1 |

0 | TN | FP |

1 | FN | TP |

For this scenario, the machine learning model was created using a free open source tool called Knime. Most machine learning models are created by programmers in Python or R, but Knime is nice for non-programmers because models can be created simply by assembling graphical widgets in a workflow.

Here the File Reader node ingests the Excel file of Customer attributes. The Partitioning node randomly splits the examples into a training set (80%) and test set (20%). This is important for any type of machine learning model but especially important for decision trees, which are prone to “overfitting”. Overfitting gives high accuracy on the training set but degraded accuracy on untrained examples. For this reason, you train the model on the training set examples but measure accuracy on a separate test set, called *cross-validation*. The Equal Size Sampling node ensures that the test set has an equal number of churn and no-churn examples. The Decision Tree Learner node trains the decision tree. This is passed to the Decision Tree Predictor node, which applies it to the test set examples. The Scorer node reports the confusion matrix for the test set. The PMML Writer node creates a PMML file for the model.

One way to control overfitting in decision trees is to increase the minimum number of examples on a branch. When the branch contains fewer examples than that, no more splitting of the branch is allowed. Using the Knime app’s original value of 6 (out of a total of 194 examples in the test set), the Scorer node reports the following confusion matrix:

Actual / Predicted | 0 | 1 |

0 | 91 | 6 |

1 | 26 | 71 |

Accuracy 83.5%, Precision 92.2%, Recall 73.2%

Gradually increasing the minimum number of examples on a branch leads to an optimum value of 10, with the improved confusion matrix below:

Actual / Predicted | 0 | 1 |

0 | 93 | 4 |

1 | 24 | 73 |

Accuracy 85.6%, Precision 94.8%, Recall 75.3%

This model was saved as *churn.pmml*. Trisotech DMN Modeler is able to invoke PMML models and use the returned result within the decision logic. (Note: This requires the PMML subscription feature. Contact Trisotech for details.) As mentioned earlier, DMN *functions *– typically BKMs – by default use FEEL as their expression language, but the spec also allows them to be *externally defined *as either Java methods or PMML models. A PMML-based BKM is distinguished by the code P in the Kind cell of its boxed expression, displayed to the left of its list of parameters:

The parameters of the BKM are the inputs (DataFields) specified in the PMML model. The PMML DataField denoted as “predicted” is the BKM output.

The Body cell of a PMML BKM is always a context with the following two context entries:

*document*, specifying the imported PMML filename*model-name*, the name of a model in the imported file

In DMN, a decision invokes this BKM in the normal way, passing parameter values and receiving back the BKM result. A PMML model must be imported into DMN before it can be invoked.

In the DRD shown here we have a decision *Churn risk* invoking BKM *Churn prediction PMML*, both type tChurnRisk, with allowed values 0 and 1 to match the PMML DataField *Churn*. Now, on the DMN ribbon (assuming you have the PMML feature enabled in your subscription), click on the PMML icon to upload *churn.pmml* to a repository in your workspace.

Then click the icon again and Add the PMML model to your DMN model. Adding a PMML file works like the *Include* button for normal DMN import. The PMML is imported in its own namespace, which is referenced as a prefix to the names of any imported variables and types. Trisotech DMN defaults the prefix to the name of the imported model – in the figure below it is *churn* – but using the pencil you can edit it to something else if you like.

PMML has its own type system, which is slightly different from FEEL, but on PMML import Trisotech *automatically* creates a corresponding FEEL type for each input (PMML DataField) that is not a simple base type. The created types are namespace-prefixed, as mentioned. You need to convert them to non-imported types, but this is simply done by clicking the copy/paste icon for each one. In this model most inputs are numbers, but because Knime reports their minimum and maximum values in the dataset, Trisotech creates item definitions for them with these constraints. Actually, we’d rather not have these constraints, as it’s possible a new test case could have an input value outside of the range or the original dataset. We can simplify the model by eliminating those item definitions in favor of the FEEL base type Number. The only ones we need to keep are the enumerations tAreaCode and tState.

Finally we need to add the structure tCustomer describing our input data element, with one component per input DataField in the PMML, and assign it to input data *Customer*:

Returning to the DRD, we now define the decision logic for the BKM *Churn prediction PMML*, a context. Changing the default Kind code F (for FEEL) to P (for PMML) automatically creates the context entries *document* and *model*. Clicking in the right column automatically extracts values for those entries from the included PMML file.

The decision *Churn risk* invokes the BKM. Selecting Invocation as the expression type makes the mapping trivial:

And that’s all you need to do. Now you are ready to execute the model.

As with any decision model, running it in Execution/Test will reveal any problems. In the Execution/Test dialog, I manually entered values from the first row of the Customer attribute spreadsheet shown earlier. On execution it returns the predicted value 0, which matches the actual value from the spreadsheet (*Churn*, column 8). At this point you could deploy the model as a DMN decision service.

What we’ve seen so far is that you can take a PMML model created in a third party tool and execute it as a BKM in a DMN model. That’s impressive, but we haven’t seen yet how DMN makes the PMML logic more explainable. To get some idea of the logic, you could scrutinize the PMML file and draw the decision tree as a BPMN flow with gateways. That’s not only a lot of work but the result is not so easy to understand. A much better way is to *convert the decision tree into a DMN decision table*. Again, you could do this by hand, but it’s easier to write a program to do it.

The results are quite enlightening. Here is a fragment of the table created from *churn.pmml*:

The full decision table has 21 rules, and my program has sorted them in order of counts in the training set to see which rules are matched most often. Recall there were 20 inputs in the original PMML, but only 9 of them are involved in the actual decision logic. Columns for the unused inputs have been deleted from the table. (The original PMML model, with a minimum of 6 examples per branch, resulted in a table with 10 inputs and 37 rules, a table with twice as many cells but less accurate on the test set!)

We can examine this table to try to understand the *reasons* why a Customer will churn. From rule 4 above, a high Day Charge and Night Charge looks like one factor. From rule 5 and others, we see that a high number of Customer Service Calls is another factor. From rule 7, a high International Charge looks like a factor, and from rule 9 an International Call Plan but low International Charge looks like another factor. On the other side, it looks like Customers with a Voice Mail plan are not likely to churn. All of these make some sense intuitively.

My takeaway from looking at this table was that perhaps we should consider the sum of Day, Evening, and Night Charges as an additional “refined” input, also the average cost per International call. State doesn’t make a lot of sense to me, so let’s get rid of that one. I went back to the Customer attribute spreadsheet, eliminated the unused inputs (plus State) and added the refined inputs Total Charge and International Charge per call. Then I created a new decision tree with Knime using the revised spreadsheet and converted that to a decision table, just to see if it made any difference.

It did. Not only is the new model more accurate…

Actual / Predicted | 0 | 1 |

0 | 95 | 2 |

1 | 19 | 78 |

Accuracy 89.2%, Precision 97.5%, Recall 80.4%

… but it is simpler and easier to interpret:

Once the Total Charge input was added, the Day Charge, Eve Charge, and Night Charge columns dropped out. Now the table has just 6 inputs and 10 rules.

My intuition from examining the earlier decision table was confirmed. Factors leading to churn include high Total Charge, high number of Customer Service Calls, and high International Charge per call. Having a VoiceMail plan discourages churn. Armed with this data, the carrier can make more informed decisions about what type of retainment offers to make and possibly changes in their pricing policies.

The accuracy of this model is surprisingly good, given that we started with zero intuition about the factors suggesting churn risk. A full 97.5% of Customers predicted to churn actually did in the test set. Recall is more difficult, but over 80% of Customers that churned were predicted by this model. Now that we have a better idea of the significant inputs, we could go back and use them in a different machine learning model – say, logistic regression or SVM, which often have higher accuracy than decision trees – and see if we can do even better.

The bottom line is that while machine learning is doing just fine on its own, integrating it with DMN can not only add explainability but actually improve accuracy.

The post DMN, Meet Machine Learning appeared first on Method and Style.

]]>The post Matrix Operations in DMN appeared first on Method and Style.

]]>But what if you want your DMN model to work with numeric tables having *an unspecified number* of columns and rows? In DMN that is not a relation. It is a *matrix*. In a matrix*, *the cells have numeric values, the columns are unnamed, and there can be any number of them. In DMN, a matrix is modeled as a list of number lists.

Some decision models involving tables of data can be implemented either using relations or matrices. While some FEEL expressions are simpler using relations, matrices have the advantage that the decision logic does not depend on the number or names of the columns. That means you can use the same decision model with tables with different numbers of columns. Also, in some cases the decision logic is simpler using matrices than the equivalent relation. For example, using relations, mapping table *T1*, with columns *c1, c2*, and *c3*, to table *T2* of the same type requires a context with a context entry for each component and a cell expression for each one. With matrices, it requires just a literal expression involving iteration.

Manipulating matrices in FEEL is a little tricky, but in this post we’ll see how to do it.

To illustrate the difference between a DMN relation and a matrix construct, consider *T*, a table with columns named *c1, c2,* and *c3*. *T* is a DMN relation. Each row of *T* is defined as a data structure with named components representing the columns of *T*. The datatype of the row – let’s call it tRow – specifies those components. Since *T* is a list of rows, the datatype of the table *T* – let’s call it tTable – is a collection of tRow.

The rows of *T* do not have names; they are referenced by number, modeled as an *index* enclosed in square brackets. *T*[1] means the first row of the table. The columns of *T* have specific *names,* for example, *c1, c2*, and *c3. *Models involving elements of *T* need to specifically reference *c1, c2*, and *c3 *by name. You cannot reference them by number, e.g. the second column. For example, in FEEL the column *c2* is simply *T.c2*. Cell *c2 *of the first row is either *T*[1].c2 or *T*.c2[1]. Both have the same value. The first expression means take the first row, then column c2 of that row. The second means take column c2, then the first row in that column.

Figure 1. Relation *T*, a list of rows with named components

Now consider a matrix *M*, a list of lists. The datatype of a row of *M* – let’s call it tMatrixRow – is simply a collection of the FEEL base type Number, and the datatype of the matrix *M* as a whole – let’s call it tMatrix – is a collection of tMatrixRow. In a matrix, neither the rows nor columns have names. Both rows and columns are referenced by number, i.e., as indices. *M*[1] means the first row, a list of numbers. *M*[1][2] means the second column of the first row, a number.

Figure 2. Matrix *M*, a list of number lists

Working with matrices in FEEL requires *iteration*, which may be an unfamiliar construct. Iteration in FEEL has two forms both involving the *for..in..return* construct:

for <<range variable>> in <<list>> return <<expression of range variable>>

is called *iteration by item value*. The *range variable* is simply a name assigned to an item in the list. The expression following return uses the range variable name wherever the expression involves the list item. Evaluation of the expression generates one item in the output list. So iteration by item value says for each item in an input list generate a corresponding item in the output list.

The other alternative, called *iteration by item index*, looks like this:

for <<range variable>> in 1..count<<list>> return <<expression of range variable>>.

The difference is the term following in is a range of integers from 1 to the number of items in the input list. In this case, the expression uses the range variable as an index.

We use iteration by item value to extract the jth column of matrix* M*:

for row in M return row[j]

Here *row* is the range variable, an item in the list *M*. Recall that *M* is a list of rows, so the range variable *row* (we could have called it *x,* or anything) simply means a row of *M*. And since *M* is a list of lists, *row* is also a list. Thus the expression *row[j]* means return the jth item in the row, i.e. the jth column.

Alternatively, we could have used iteration by item index, as follows:

for i in 1..count(M) return M[i][j]

Let’s take that apart. Here the range variable i is an integer from 1 to the number of rows in M. The expression M[i][j] means take the ith row of M and then the jth column of that row, iterating over i, i.e., over rows of M. So this also returns the jth column of M.

When using matrices, for the time being there is a minor tool issue we need to deal with. Trisotech DMN Modeler can handle lists of lists internally, but the Execution/Test dialog for entering test data and viewing the results cannot display them. That’s ok. We will enter input data as a relation and convert it to a matrix using the little-known FEEL function *get entries()*. *Get entries* maps a data structure, such as a row in our relation, to a two-column table with one row per structure component, the first column *key *holding the component name and the second column *value *holding its value. A companion function *get value()* extracts the value of a selected *key *in the structure.

When dealing with matrices, it is convenient to call upon a library of basic operations on any mxn matrix, meaning m rows and n columns, or any n-element vector. By clicking Include on the DMN ribbon and selecting the Matrix operations model, you can import BKMs from this model into your own DMN models. They include:

- xij(M, i, j), returning the matrix cell value at row i, column j
- dimension(M), returning m, the number of rows, and n, the number of columns
- transpose(M), returning the transpose of the matrix
- column(M, j), returning the jth column of the matrix
- smult(M, a), returning the product of a scalar and a matrix
- madd(M1, M2), returning the sum of 2 matrices with equal dimensions
- mmult(M1, M2), returning the matrix product of two matrices with compatible dimensions
- vmult(V1, V2), returning the dot (inner) product of 2 vectors
- magnitude(V), returning the root sum squared of the vector elements

Figure 3. Matrix operations library

The BKMs that map a matrix into another matrix generally rely on nested iterations. These look complicated at first but if you work with matrices you will get used to them. They take the form:

for i in 1..count(M) return for j in 1..count(M[1]) return <<cell value>>

The outer for loop iterates over an index to rows of the matrix *M*. The inner for loop iterates over an index to the columns, modeled as the items in a row of the matrix.

As mentioned earlier, input data can be modeled as a relation and then converted to a matrix using *get entries*. The decision *Alternatives, *shown below, converts *AlternativesTable*, a relation, into tMatrix, a list of number lists:

Figure 4. Conversion of a relation to a matrix

The first context entry *AltCriteria* generates a list of column names in *AlternativesTable*, applying *get entries* to a row of the relation and selecting the component *key*, returning a list of strings. We don’t want the column *Name* in our matrix, so we exclude it with a filter. We then iterate over rows of the relation, within each row iterating *get value* over a list of keys to create the matrix.

I recently created a model for a client implementing an algorithm for multi-criteria decision making called TOPSIS. You can read more about it here. It can be implemented in DMN either using tables or matrices. I will briefly describe the table implementation and go into more detail with the matrix implementation.

The DRD below illustrates the table implementation.

Figure 5. DRD of TOPSIS table implementation

The algorithm basically works as follows:

- You want to compare m
*Alternatives*using n*Criteria*. Each Criterion contains a rating from 1 to 9, a*type*, either a Benefit or a Cost, and a*weight*. For Benefit Criteria, a rating of 9 is best and 1 is worst. For Cost Criteria, 1 is best and 9 is worst. The sum of the weights must be 1. Using tables (relations), each row of input data*Alternatives*is a structure with a component for each Criterion. The component name is the Criterion name and its value is the rating for that Criterion. - The
*Normalized*matrix divides each cell value of*Alternatives*by the root sum squared of the column values. - The
*Weighted*matrix multiplies each cell value of*Normalized*by its Criterion*weight*. - The vector
*Ideal*is a list of n values representing, for each column of*Weighted*, the max column value for Benefit types or the min column value for Cost types. The vector*NonIdeal*is a similar list, but using the min column value for Benefit types and the max value for Cost types. This DRD assumes the decision knows which Criteria are Benefits and which are Costs. Alternatively, you could encode that in the input data*Criterion Weights*and reference that in the decision logic. - Each
*Alternative*can be considered a point in n-dimensional feature space, the rating value for each of n Criteria.*Ideal*and*NonIdeal*are similarly points in that space. Thus for each*Alternative*you can calculate the Euclidean distance between it and*Ideal,*and between it and*NonIdeal*, where Euclidean distance means the rool sum squared of the difference vector. The decisions*Distance to Ideal*and*Distance to NonIdeal*are lists, one item per*Alternative*, of those distances. *Topsis Score*is a table of m rows, one for each*Alternative*and its score value,Distance to NonIdeal/(Distance to Ideal + Distance to NonIdeal).

Values closer to 1 are closer to the

*Ideal*and further from the*NonIdeal*

The model using relations is straightforward. Each Criterion name is a column name in the table *Alternatives*. That’s fine, but if you add a new Criterion, the datatype tAlternatives changes and the expressions for each decision in the model must change as well. Alternatively, using matrices you can create a decision model in which the Criteria are just input data. You can have any number of them and the model doesn’t change. You do have to deal with the parsing of the input data and the nested iterations.

To illustrate the matrix implementation, I created an EU-Rent example, a recommender of available rental cars based on the customer’s stated preferences. In addition to allowing any number of Criteria (and adding new ones), it also addresses something that has bothered me about TOPSIS, which is the fact that the Criterion ratings mix objective metrics, like the car’s rental price or miles per gallon, and customer-specific preferences, such as style or number of passengers. My example deals with that, although it could be improved in a couple respects:

- More cleanly separate the objective measures – maybe in a zero-input decision – from the customer preferences, which are input data.
- Better map certain objective measures to user preferences. For example, a higher passenger capacity is not necessarily preferred.

I encourage readers to make those improvements and email the DMN export to bruce@brsilver.com.

The DRD is shown below. It is similar to the previous one, but inserts two new decisions, *Rated* and *Scaled*, between *Alternatives* and *Normalized*, and uses BKMs imported from the matrix operations library to deal with common matrix functions.

Figure 6. Matrix implementation of TOPSIS

Input data *Criteria* is a table, collection of tCriterion, shown below. The last three components are used, for certain criteria only, to convert between numeric measures and a rating. For Benefit criteria it makes the assumption that a higher number is better, which is not always the case, so this aspect could be improved.

Figure 7. Type tCriteria includes components used in *Rated*

Input data *AlternativesTable* is also a table for data entry purposes, but it will be mapped to a matrix. Each row of *AlternativesTable* is a structure containing its *Name* and scores on various criteria.

Figure 8. Type tAlternatives models input data as a relation

Decision *Alternatives* maps *AlternativesTable* to a matrix, type tMatrix, a list of number lists. As shown earlier, it uses the FEEL functions *get entries *and* get value. *Because *get entries* does not return the list of components in the same order as the structure definition, we need to use the values of the column *Criteria.name* as the *key* name in *get value*. This ensures the order of the columns in the matrix matches their order in input data *Criteria*.

Figure 9. *Alternatives* maps the list of delimited strings to a matrix

Decision *Rated* maps certain cells of *Alternatives* to a rating based on customer preference by calling the BKM *rate*. The mapped cells are those for Criteria with a non-null value in component *bestIfAtLeast*. The BKM rate compares the cell value to the values of components *bestIfAtLeast, okIfAtLeast*, and *mustHaveAtLeast* with the assumption that more is better. The logic of this BKM is not the best; it should be improved.

Figure 10. *Rated* maps some raw Criteria values to rating based on user preference

Figure 11. BKM *rate* implements the preference mapping

Decision *Scaled* is a departure from “official” TOPSIS in that it rescales the value of any Criterion to a range from 1 to 9 using the mean and standard deviation across all Alternatives. A value equal to the mean is assigned a rating of 5. A value 2 standard deviations higher is assigned a rating of 9, and a value 2 standard deviations lower is assigned a rating of 1. Values in between are interpolated and rounded to the nearest integer. Values beyond 2 standard deviations from the mean are clipped at 9 or 1.

*Scaled* is a context that makes use of the matrix operation *transpose*. The first context entry *columns* transposes *Rated*, making it a list of columns. Context entry *scaled columns *iterates over that list, calling the BKM *scale, *with arguments the cell value, the column mean, and the column standard deviation. This is a simple decision table. The reason it has columns for the mean and standard deviation is that for a BKM, the parameters are specified by the input column names. Yes, stupid, I know. It’s in the spec. The final result box uses *transpose* again to put the matrix back in proper order.

In my implementation, I applied scaling even to *Criteria* values that are manually rated. That is probably incorrect. An improved implementation would apply scaling only to objective *Criteria* values.

Figure 12. *Scaled* uses transpose twice to operate on columns of *Rated*

Figure 13. BKM *Matrix operations.transpose*

Figure 14. BKM *rate* implements the scaling algorithm

Decision *Normalized*, from the original TOPSIS algorithm, divides each cell in *Scaled* by the root sum squared of its column values. Since a column of the matrix is a vector, we use the imported *Matrix operations.magnitude* function to calculate that. As in *Scaled*, we use *Matrix operations.transpose* to work on columns and then transpose back again.

Figure 15. *Normalized* uses transpose twice to operate on columns of *Scaled*

Figure 16. *Matrix operations.magnitude* calculates the root sum squared of a vector

Decision Weighted applies customer-selected weights to each Criterion. The weights must total 1.0.

Figure 17. *Weighted* multiplies each cell of *Normalized* by the Criterion weight

Decision *Ideal* calculates the list, one element per Criterion, representing the maximum value of *Weighted* for Benefit Criteria or the minimum value for Cost Criteria. *NonIdeal* is the same but swapping Benefit and Cost. For n *Criteria*, *Ideal* and *NonIdeal* can be thought of as *vectors* in n-dimensional feature space.

Figure 18. *Ideal* selects the max *Weighted* value for Benefit Criteria, min value for Cost Criteria

Figure 19. BKM *Matrix operations.columnj* extracts the jth column of a matrix

Each row of *Weighted* – corresponding to a different *Alternative* – can similarly be envisioned as a vector in the same n-dimensional feature space. Decision *Distance to Ideal* calculates, for each row of *Weighted*, the Euclidean distance between that row vector and the *Ideal* vector, applying *Matrix operations.magnitude* to the difference vector. Decision *Distance to NonIdeal* does the same, substituting the *NonIdeal* vector for *Ideal*.

Here *Distance to Ideal* is modeled as a context. Each row of context entry *deltaMatrix* is the difference vector between the corresponding row of *Weighted* and *Ideal*. The final result box calculates the magnitude of each difference vector. *Distance to NonIdeal* is the same, substituting *NonIdeal* for *Ideal*.

Figure 20. *Distance to Ideal* computes the difference vector between each row of *Weighted* and *Ideal*, then calculates the magnitude of that difference vector.

For the top-level decision *Topsis Scores* we want a table that lists each *Alternative* by name along with its *Score*, calculated as *Distance to NonIdeal*/(*Distance to Ideal* + *Distance to NonIdeal*), and sorts the list in decreasing score order. Context entry *unsorted* iterates over each Alternative to call BKM *get Row*, which simply formats a row of the table output. The final result box uses the FEEL *sort* function to list *Alternatives* in decreasing score order. The second argument of sort is always a boolean function of two range variables – here *x* and *y.* A true value means *x* precedes *y* in the sorted list, so *x.Score>y.Score* means the list is sorted in descending order of the component *Score*.

Figure 21. *Topsis Scores* calculates the score for each Alternative, then sorts the list

Figure 22. BKM *get Row* simply formats rows of *Topsis Scores*

As an example, consider a rental car recommendation decision for EU-Rent. In this example, we consider *Criteria* to include 5 attributes, as shown below in Execution/Test:

Figure 23. Rental car selection Criteria

Figure 24. Raw rental car data, BOS weekly rental, hertz.com

“Objective” values, taken from hertz.com weekly rental in October at Boston Logan Airport, include Fuel Economy (MPG) and Price. Capacity (Seats) and Luggage (Suitcases) are “semi-objective” in the sense that the customer might not always rate a higher value as preferable. For the semi-objective attributes, *Criteria* asks the customer to provide values for *bestIfAtLeast*, which gives a rating of 9; *okIfAtLeast*, a rating of 6; and *mustHaveAtLeast*, a rating of 3. As noted previously, this logic is not the best, as a customer might rate a car seating 5 higher than one seating 7. The purely subjective attribute *Style* is selected by the customer as the first *scoresText* entry in input data *Alternatives Raw*:

Figure 25. Input data *AlternativesTable *in Execution/Test

These values are saved as Test Case 1. To run the model with these values, in Execution/Test click Load and select Test Case 1. Running the model gives the following sorted *Topsis Scores*, with Jaguar XF being the top rated. A customer giving a higher weight to Price would likely have a different recommendation.

Figure 26. *Topsis Scores* output in Execution/Test

The model could certainly be improved by better separating purely subjective criteria like Style, from semi-objective criteria like Capacity and Luggage and purely objective criteria like Fuel Economy and Price. I think best if the purely objective ones are not input data at all but a zero-input decision and the semi-objective ones use a better mapping of the raw numeric values to a customer rating from 1 to 9. Also, it’s probably better to apply scaling only to the objective measures, since the others are already in a 1-9 range.

Readers are invited to make those improvements and send them to me. We will post the good ones on the site.

The original idea of exploring TOPSIS multi-criteria decisioning using DMN was suggested by Professor Koen Vanhoof of Hasselt University.

The post Matrix Operations in DMN appeared first on Method and Style.

]]>The post Practical DMN: The Basics appeared first on Method and Style.

]]>Want to learn DMN decision modeling without training? DMN Method and Style, 2^{nd} edition is a good place to start. But to really learn it, you need to go beyond books. You need to get your hands dirty in a tool. I suggest starting with a free trial of Trisotech DMN Modeler.

When you log in, go to the EU-Rent repository (File/Open/EU-Rent) and open the model called EU-Rent Pricing. I’ll walk you through it here. Then click File/New and try to re-create the model yourself. That might sound like busywork but, trust me, you’ll learn a lot just doing that.

This model selects the best price for a car rental request based on the type of car, length of rental, applicable promotions, and customer loyalty discount.

The *Decision Requirements Diagram* is shown here. In any DMN model, you start by creating the DRD.

- The rectangles are
*decisions*. Each decision has a name, a datatype or “type”, and a value expression, a formula for calculating the decision value from the values of its inputs. T - The ovals are
*input data*. This is the data supplied to the model in order to calculate the decision values. Here I’ve colored them yellow, but that’s just to highlight them. You can change the color from the Fill button on the Home ribbon. Input data has a name and a type but no value expression. You enter its values on execution of the model. - The solid arrows leading into a decision are called its
*information requirements*, identifying the dependency of the decision’s value expression on supporting decisions and input data. - The rectangles with clipped corners are
*business knowledge models*, or BKMs. Here I’ve colored them blue for convenience. A BKM’s value expression is defined as a function of one or more parameters. The function is called by a decision, or possibly another BKM, identified by a dashed arrow called a*knowledge requirement*. For example, here*Weekly Price*calls the BKM*Weekly Type Price*by supplying values to its parameters and receiving the function result in return. - Each decision, BKM, and input data element in the DRD is represented by a
*variable*with the same name, used in the value expressions.

You should get in the habit of specifying the datatype of each DRD element. It’s not absolutely necessary but always best practice, so just do it. The type language for DMN is called FEEL, which is also the language of its value expressions. FEEL base types include string (text), number, boolean, date, time, date and time, and a couple duration types. A variable assigned to a base type can have any value in the domain of that type. Sometimes that’s fine, but often you want to define constraints on the domain of allowed values. A type with defined constraints is called an *item definition*, meaning a user-defined type. These are not special things. You will use them all the time. An item definition can also be a data structure, a list of *components*, each with a name and type. And finally, an item definition may be specified as a *collection* of another type. To see the item definitions in this model, go to the DMN ribbon and select Data Type.

There are 7 item definitions in the model. Click on the pencil for *tCustomer* to view its definition.

You see it is a structure with 6 components. The last component, *Past Rentals*, is type *tRentals*, a collection of type *tRental*, where *tRental* is a structure with 3 components, the last of which, *CarType*, is Text with an enumeration constraint. Its value can only be “Economy”, “Compact”, or “Full Size”.

To set the type of a DRD element, right-click Attributes/Input Data Type (for input data) or Output Data Type (for decision or BKM). You can select a base type, an existing type, or create a new type. Here input data *Customer* is type *tCustomer* and *Rental Request* is type *tRental*.

Let’s look at the overall decision logic. The top-level decision *Best Price,* a number, depends on 4 supporting decisions: *Weekly Price* and *Daily Price*, both numbers; *Applicable Promotions*, a 2-column table; and *Discount Percent*, a number representing a percent. Those dependencies are specified by the 4 information requirements into *Best Price* in the DRD.

If you click on the icon in the upper left corner of *Best Price*, you see its value expression. DMN supports a variety of value expression formats. This one, a text formula, is called a literal expression. It uses the FEEL expression language.

The syntax may look daunting at first, but it’s actually simpler than, say, Excel Formulas. FEEL was designed for business users, not programmers. It contains a number of built-in functions, including three used here:

- append(list, item1, item2, …) means append items to an existing list
- min(list) returns the minimum value of a list of numbers or dates
- decimal(number, integer) rounds a number to a specified number of decimal places

A list variable followed by an expression in square brackets represents a *filter* of the list. Here *Applicable Promotions* is a 2-column table, a list of rows with two components, *Description* and *Price*.

So *Applicable* *Promotions.Price* is the *Price* column of the table, a list of numbers, and the filter [item>0] here just ensures that if there are no *Applicable Promotions* the expression is an empty list rather than null. Reading the literal expression from the inside out, we append to the list of prices from *Applicable Promotion*s the discounted *Weekly Price* and the discounted *Daily Price. *Then we take the minimum value of the resulting list and round it to two decimal digits following the decimal point. This style of nesting one function inside another is economical but maybe hard for beginners to understand. Alternatively, you could have separate decisions in your DRD for each part of it – the list append, finding the minimum, and rounding – or you could use another value expression type called a *context* to do each of those steps in a separate table entry.

The decision *Weekly Price* illustrates a context. We want to multiply the weekly price for the car type by the number of weeks in the rental. We could have written it as a long literal expression as in Best Price, but here we broke out the calculation of *Weeks count* as a separate context entry. Each context entry has a name, type, and value expression. The last row of the context is the final expression.

*Weeks count* subtracts the rental start date/time from the end date/time, giving a days and time duration. We divide that duration by the duration 168 hours (7 days x 24 hours), to get the number of weeks, and round that up to the next higher integer using the *ceiling* built-in function.

The final result multiplies *Weeks count* by the result of the BKM *Weekly Type Price* with the parameter value *Rental Request.CarType*, meaning the type of car rented. Weekly Type Price’s value expression is a decision table. It has one input, *Car Type*, type *tCarType*, which must match the type of the argument in the call, *Rental Request.CarType.*

Putting it all together, *Weekly Price* passes the car type to the BKM, gets back the weekly price for that car type, and multiplies that by the number of weeks in the rental.

Finally, let’s look at *Applicable Promotions*. EU-Rent has a list of *Date Based Promotions*, here modeled as a BKM decision table. When a BKM is a decision table, the table inputs specify its parameters, so there are 6 of them. The table has 2 output columns, meaning a structure with 2 components, and the table has hit policy C (Collect), meaning return a list of outputs for each matching rule. So what is returned is a list of 2-column structures, the table type *tPromotions*. Also note that not all outputs are specified as literal values. In the last item in *Date Based Promotions*, the *Price* component of the output is an expression calling the BKM *Daily Type Price*.

*Applicable Promotions* calls this BKM by supplying values to the 6 parameters. It is modeled as a context. The first 4 context entries use FEEL calendar type components to extract the month and day value from the rental start and end date-times. The fifth calculates the number of days in the rental. Those context entries are used in the call to the BKM in the final result expression.

Once you’ve completed a decision model you need to validate it. On the DMN ribbon select Validate. That should return no errors. A second form of validation on that ribbon is Method & Style Decision Table Analysis, which looks for gaps, overlaps, and other errors in decision tables. In this model, that validation returns 3 errors on the *Discount* decision table in *Discount Percent*.

From the table you can see the reason. *Qualifying Rentals Count* is really an integer, but FEEL has no integer type, only number. So the reported rule gaps are for intervals like greater than 0 but smaller than 1, impossible for integers. In a case like this you have two choices: (1) Understand the validation error and just accept it; or (2) Fix the table, even though that looks odd. The fixed table is shown below:

Now it’s time to execute the model. I don’t think I can emphasize this enough: If you cannot execute your model (and make sense of the results), you can’t have confidence that the logic is correct. Many DMN tools cannot execute the logic. Fortunately, Trisotech can.

In the Execution ribbon, select Test. On the left, you enter values for the input data, then click Run. The tool reports outputs of all decision nodes in the DRD. After a run, Save preserves the input and output values in a Test Case. When entering values in the Test panel, the Load button lets you populate the form with values from a saved Test Case. The screenshots below show the input data for Test Case 2, followed by the execution output.

That’s pretty much it. We’ve talked about the DRD, datatypes, FEEL expressions, BKMs, contexts, validation, Decision Table Analysis, and execution. It’s a lot of information in a short space. Remember, I said at the top you need to get your hands dirty in the tool to really get it. So do this: If you haven’t done so already, go to https://www.trisotech.com/methodandstyletrial to get a free trial account, then log in and open EU-Rent Pricing. In the Import-Export ribbon, click HTML Documentation and print out the web page. That captures all the information in this model. Then click File/New and re-create the model from the documentation. Make sure it validates, then run it.

If that seems overwhelming, I suggest the training. DMN Method and Style Basics covers DRDs and datatypes, decision tables, BKMs, simple FEEL, and decision services. DMN Method and Style Advanced dives deeper into FEEL functions and operators, contexts, calendar arithmetic, and manipulating lists and tables. They go at a slower pace, with exercises using the tool at each step of the way and post-class certification based on an online exam and mail-in exercise that you iterate until it is perfect. That is the best way to become proficient in DMN.

The post Practical DMN: The Basics appeared first on Method and Style.

]]>