Andy Orrock | Payment Systems

jPOS-CMF!

2009-11-08T12:51:45-06:00

Yesterday, I made my first contribution to the jPOS Common Message Format (‘CMF’) effort. Alejandro writes about this good endeavor here. I’m really dipping my toe in the water here, as I have a lot to learn about DocBook, <oXygen/> (the XML authoring tool I’m using per APR’s suggestion) and GitHub, where the jPOS-CMF sources are located as a public repository. I made some small changes based on my observation that fields like Function Code (ISO 8583:2003 DE 24) and Reason Code (DE 25) and a few others needed some further definition. The result of my updates is v1.0.4 of the doc, which you can get here.

This was an appropriate juncture for me to take a stab at jumping into the jPOS-CMF pool with Alejandro, Dave and Mark. OLS has also based our IMF on the ISO 8583:2003 standard and with our recent focus on a jCard-related implementation, Dave, Allen and I have spent many hours over the past couple of weeks iterating over our ‘filter’ mechanics (i.e., how to ‘map’ into our CMF from a gateway or other interface) and, specifically, how to use Function Code and Reason Code values to differentiate transaction types for proper TransactionManager identification, routing and participant flow.

Thats a lot more candy

2009-11-01T21:26:38-06:00

With Halloween falling on a Saturday this year, I thought Friday might be the big day for shoppers to stock up on candy.

As Maxwell Smart would say, “Missed it by that much.”

Saturday, this nation’s proud procrastinator majority once again rose up and demanded to be counted! Saturday ended up slightly outpacing Friday, 1,314,260 transactions to 1,294,736. That’s slightly over 2.5 million payment transaction requests serviced over a two-day period at OLS’ flagship acquirer implementation.

OLS new offices

2009-10-31T14:43:46-05:00

OLS is movin’ on up.

To the 11th floor that is. Our hard-earned successes and recent growth spurt have led us to build out nice new offices at our 7920 Belt Line Road location in Dallas, Texas. For those of you that are familiar with the North Dallas area, we’re at the southwest corner of Belt Line and Coit, equidistant from North Central Expressway, I-635 and the North Dallas Tollway. We are, respectively, about two miles west, north and east of those major thoroughfares. Check us out on Google Maps.

And we’ve got a Starbuck’s within walking distance. What’s not to love (even though they do keep it a sub-arctic 53 degrees or so inside the store)?

Want to discuss your payment systems strategy and hear what we’ve done for others? We look forward to hosting you at our offices.

HP Greenbook: BASE24 upgrade options

2009-10-31T13:51:48-05:00

Check out some great analysis by HP about the BASE24 displacement market.

Money line:

The Standish Group estimates that BASE24-eps migration coupled with a platform migration (IBM System Z9), both conducted within one year, would only be completely successful in 18 percent of the cases.

As my good friend Mitch Green of Distra says: ACI and IBM have given everyone in the payment systems industry a huge gift.

Thats a lot of candy

2009-10-31T13:33:37-05:00

We processed 1,294,736 transactions at OLS’ flagship Acquirer-side payment systems implementation yesterday. Friday’s the peak day of the week there anyway, but the numbers got goosed by Halloween falling on a Saturday. Over a million of the transactions were to service ‘core’ payment requests (Debit, Credit, EBT and Check).

We’ll probably see another like-sized day today as retail customers across our client’s four US time zone presence stock up for Trick or Treat duties.

All you guys ever do is write documents

2009-10-31T13:17:57-05:00

A few months back, a customer told me in a spurt of frustration “All you guys ever do is write documents.” This was early in our relationship. I could see how he could form that conclusion. We spend a good portion of the early part of the project gathering business requirements and getting those on paper. We don’t use any special approach or ‘school’ – we only seek to get the fine-grain details on paper in as ‘plain English’ as possible.

It is true that we can drive people a bit crazy with frequent revisions. But we control that by using Basecamp, where it’s a cinch to corral and categorize documents and easily point to the latest revision of each one.

Of course, at a certain point in the project, we stand aside and let Dave and his development team design and develop a solution based on what we’ve compiled. That’s one thing we don’t do in the doc: direct the developers to a certain design or approach. If I’m telling Dave or Alejandro how to do something technical, we’re in trouble. I accept that. Like Harry Callahan always said: Man’s got to know his limitations.

Anyway – the proof is in the results: the success of our payment systems projects lies in the strength of these documents, which serve as a firm starting point. This isn’t just me talking: Alejandro has told me the same thing – that when he does projects not related to OLS, he misses our documentation and attention to detail.

Randy tells me that when looking to demonstrate OLS’ capabilities to a sales prospect, he selects the most applicable documents from our growing repository, does the requisite redaction and presents those in lieu of any PowerPoint deck. He says its an incredibly powerful tool.

Now, youre the switch

2009-09-26T13:48:01-05:00

Occasionally, along the OLS.Switch lifecycle – either at sales time, or during the initial installation, or somewhere down the road – we hear things like this from our customers and prospects:

“Hand us the transaction because we can make a better decision on it.”

“We have a rules engine that is our ‘special sauce.’”

“We know more about the cardholder than you ever can.”

“Your proposed price is too high, so give us the transaction and we’ll implement the meat of it.”

“We just want you to act as a ‘pass-through’; we’ll do the real work.”

“The decision-making is where our IP is, so we want to do that.”

These are all very legitimate comments (well, expect the ‘pass-through’ comment – that grates on me…trust me, it’s never as easy as ‘just a pass-through’…grrrr). We certainly don’t want or need to control all aspects of a transaction. Nor do we want to stand in the way of the ability to make a better decision about a transaction request.

But here’s the thing: once we send the transaction to you, now you’re the switch. What I mean by that is: your application is now beholden to the same throughput, speed, efficiency, extensibility and 24x7x365 availability concerns that define our lives. And while there have been many that have been up to the task, there have been countless other instances where that’s not been the case. We’ve seen applications…

…toppling once past 1 TPS
…needing to be recycled every Friday afternoon
…not letting go of transactions threads
…getting hopelessly confused and giving up responding to us
…responding with crap system error messages
…running out of resources
…timing out 50% (or more) of all our requests

If you go this route, just be careful what you ask for. It’s about more than your application rules or that special thing you do: now, you’re the switch.

More TransactionManager Gymnastics

2009-09-26T09:52:30-05:00

Related Readings:

Giving the TransactionManager a Workout

The jPOS TransactionManager and Its Participants

I was doing some more work today with jPOS TransactionManager (‘TM’) instances. We’re bringing a second FSA-enabled endpoint on-board at one of our Acquirer payment switch implementations. My favorite part of the on-boarding is always the juncture when I sit down to figure out how to best implement the required transaction participants in the TM. When our clients and prospects ask to see “our authorization rules” or wonder “how do we implement new features and concepts,” we go directly to showing them some of our production TM implementations. That’s always the “a-ha!” moment.

The FSA implementation really brings a lot of best practices into play…especially in the part where I was diving in today: Reversals. Here’s a list of the things I had to consider while doing this:

[NOTE: The list below refers to the flow exhibited in this example.]

Identify the transaction type in play (in the example that I’ve attached here, I’ve isolated the FSA Reversal logic for reader review).
Create the tranlog entry
Figure out what endpoint to route to (see our ‘SelectEndPoint’ participant – and its variations). We do this by examining a table we construct from card types and bin ranges (lookup the BIN range to determine the card type which – in turn – is related to an endpoint).
For some of the endpoints (see ENDP2, ENDP3 and ENDP4 in my example – I did some masking here), simply handle the Reversal locally. [What’s going on there is that we’re not supposed to get FSA transactions related to these card types…but my experience dictates: you never know. So, we play prevent defense. Moreover, you can never flat-out reject a reversal of any kind because originators SAF them…see why here.]
Try to find the original. [We’ve got two routines in our toolkit: what I call “find original soft” and “find original hard.” Early in the TM processing for reversals we need the ‘soft’ option because we need to get the PAN from the original if it exists (reversals typically don’t have PAN or track info); but even if we can’t find an original, we don’t want to abort the flow just yet, because we have other stuff to do). Later in the flow, we want to abort if no original was found (where not found = never got an original – OR – the original was declined – OR – the original was already reversed – OR – the original was already settled or extracted), so we do the ‘hard’ option.
For Endpoints 1 (ENDP1) and 5 (ENDP5), prepare to go remote…but first we need to check if the original was handled locally or remotely. [In this implementation, there are circumstances in which an original request could have been handled at the point-of-sale as a ‘manager override’ which we handle locally (it’s not an auth – the sale happened, and the customer is long gone)…in those cases the endpoint doesn’t know about the original, so it makes zero sense to send them the reversal.]
If the original was handled locally for cards associated with either ENDP1 or ENDP5, handle the reversal locally.
If the original was handled remotely for cards associated with either ENDP1 or ENDP5, handle the reversal remotely.
Create either an ENDP1 or ENDP5 outbound reversal request. [As a point of interest, the ENDP1 here is pure ISO8583, while ENDP5 uses APR’s very useful FSDISO approach (this is our 6th usage!)]
Drop the reversal into the SAF queue (we always SAF outbound reversals!)
Do our FlagReversal thing (tags the original’s revInd = ‘T’ and cross-links the reversal and original with cross-referencing transaction IDs in their respective refId columns).
Do a force approval to make sure we get a controlled result code if the process aborts. [We use our internal result code = 4000, which is what we call a Force Approval. You never send a non-approval back on a reversal. This is a critically important concept. You learn this the hard way.]
Create a response back to the originator.

Analysis by Transaction Class at an Acquirer

2009-09-19T10:51:59-05:00

Here’s a summary of some transaction analysis I was performing this morning at one of our OLS.Switch acquirer implementations…

I took all the transactions from yesterday at this customer location (1,077,556 at approx 5,000 locations across four US continental time zones), grouped them into transaction class (e.g., Debit/EBT, Credit, Stored Value, etc.) and card type (different credit card authorizers, various subclasses of internal transactions, etc.), gave the resulting categories meaningful names, arranged in ascending order by average duration of the transaction (in milliseconds) and provided the associated number of transactions for the class.

Here’s what I came up with:

Transaction Class   No. Txns   Avg Dur
=================   ========   =======
Discount Coupon       14,484        33
Loyalty (1)               25        36
Loyalty (2)              107        38
Loyalty (3)               63        39
Rewards                  440        42
Employee Validation   60,674        44
Loyalty (4)              180        51
Loyalty (5)               35        53
Check (Internal)      12,010        55

--------------------------------------
Authorized Returns       652       140
Stored Value (1)      17,941       255
Stored Value (2)         897       269
Stored Value (3)       8,367       606
MethCheck             14,544       669
FSA/HSA               10,584       742
Credit/Offline Debit 389,038       747
EBT Cash                3049       857
Electronic Check      74,733       863
Stored Value (4)      16,521       868
Stored Value (5)      25,470       935
Online Debit         399,286      1215
Stored Value (6)         756      1313

The presence of the red line is of interest: for any of the transaction categories above the red line, our authorization path is completely internal, i.e., no QueryHost action inside our TransactionManager is required to get an authorization. So, when staying internal, we get the work done in 1/20th of a second or less (on < $20k in combined, core application/DB server hardware).

Below the red line, we have to query either an internal server somewhere else in the enterprise (where another application has better knowledge and can make a more informed decision than we can) or an external server, like for example routing a Credit/Offline Debit request to the FDR North payment gateway for forwarding to the appropriate card issuer or authorizing agent.

I put these into graph format for reader review and further use. Here’s the Excel 2007 (.xlsx) version; and here’s the Excel 2003 (.xls) version.

Two follow-up notes (added subsequent to my original post):

I’ve tallied and compared only the approved transactions for the day, so as to eliminate the skewing effect of timed-out transactions (i.e., because no response was received from the remote authorizer) on the smaller transaction classes.
All durations are inclusive of time spent ‘off box,’ i.e., waiting for a response from a remote authorizer.

On-Line Strategies

2009-09-12T09:28:31-05:00

Our firm, On-Line Strategies, is going through a growth spurt, the result of hard work, attention to detail, successful projects and happy customers. We’re moving into new, bigger offices next month. And, we’re on the lookout for new people to join our team.

If you’re a regular reader of this blog, you know what we do. If not, glance through some of my previous posts to get a feel for our business.

We have three immediate needs:

First: Developers, Developers, Developers, Developers! Take a look at our Monster job posting. These positions report to Dave Bergert, OLS’ Technology and Development Director. [See Dave’s blog here.] Submit your CV to the Monster posting or reach out to Dave directly to present your qualifications.

Second: We’re scouting for Project Managers. Our PMs are payment systems specialists who: interact with customers; gather and document business requirements; identify and create add-on business opportunities; present and explain development needs to Dave and his team; review unit tests; create test plans for customers; craft go-live plans; create internal documentation and release notes; manage customer expectations; interact with other PMs to find synergies and identify like solutions; expand and refine our on-boarding processes; participate in technical sales calls; assist in production support efforts; write blogs to expand our online presence; and…you get the idea.

Put another way, our PMs do everything in the customer lifecycle except sales (we leave that to the expert) and design, development, technology choices and hard-core production support (Dave’s team).

Send me an e-mail expressing your interest and qualifications.

Third: The backbone of OLS was founded on our production support capabilities. As one important piece of our business, we provide specific support and development capabilities to the Stratus VOS community (more specifically, to users of the ON/2 payment processing platform). If you have expertise in this arena, we have opportunities for you in either an FTE or consulting capacity. If you’re interested, contact me.

We look forward to hearing from you!

800,000,000 and counting

2009-09-06T16:33:52-05:00

The tranlog odometer eased past the 800,000,000 Transactions Processed threshold earlier this week at our flagship acquirer payment switch implementation.

ETA for one billion transactions: Early March 2010.

Service Status Totals

2009-08-19T08:14:13-05:00

As our production OLS.Switch implementations grow in scope and complexity, one challenge is how to reflect the current system status in an immediate and visceral way. For example, one of our acquirer-side clients has a production implementation that has grown inexorably and steadily over the past five years to encompass 93 separate system components.

With enterprise-class operations, you’ve got to be realistic: the operations team there does not have the time nor the inclination to scroll through the three to four screens-full of information that comprise the current system status. We asked this group what we could do to make there lives more easy. They responded: “We want one big-ass Green/Red button at the top of the screen depicting overall application status.”

Using this descriptive requirement as a spur, my OLS colleague Dave Bergert implemented something a bit more nuanced and informative, but no less helpful: the Service Status Totals concept (see image at left). He tallies up green (OK), yellow (Warning), red (Error) and bouncing red (Critical) lights and plops that neat and tidy little summary on the top of our UI page.

Now, no Ops scrolling unless any of those three right-hand figures go greater than zero.

Timeouts: Active vs. Passive

2009-08-19T07:46:36-05:00

Yesterday, an Acquirer switch client of ours called telling me that one of the production channel connections to their ECA provider had been down for the last 24 hours. He wondered why there was no history of the outage reflected in the application syslog.

Answer: they'd not finished configuring the channel. See to the left our jPOS-EE-fueled UI screen depicting the channel configuration. You've got to specify Timeout and On Timeout values in order for the system component to be moved to the proper error status; and you need to specify Max Events in order for the corresponding component status change history to be reflected in the syslog.

I've shown the FDR channel above as an example. I recommended the ECA channels get set up in a similar manner. My client asked a really good question: "60 seconds....isn't that a little tight for a channel timeout?" Ah - this is a confusion between the active and passive timeout concepts. Both are in play here.

He was thinking about what I call the 'active timeout.' As described here, this is a concept implemented at the channel's socket level and is specified inside of the deploy file (see image at left - production host:port information is redacted). As described in that earlier post:

Alejandro advised us to implement a property value that will set a socket-level timeout. The ‘receive’ function in the multiplexer’s ‘Channel Adapter’ will fail (with log event ‘<io_timeout>’) if nothing is received within the specified timeout period. The channel will disconnect and then attempt a reconnection.

We've got FDR set up, as you see, to 800000 ms (13 mins, 20 secs). FDR is a high-volume gateway. If we've not received a response of any kind on it for that long a duration, we want to proactively rupture and cycle the connection. This simple, easy-to-implement concept has greatly improved our up-time and reduced the volume of our support phone calls.

The 'passive timeout' - implemented in these examples by the 60-second value depicted above - is something quite different but no less important. As described here, in the jPOS-EE framework...

...the status subsystem works in a passive way. Each status entry is 'touched' from the system component that controls it. Each component has an associated timeout. If a status entry has not been touched within the given timeout period, then somebody needs to move that entry from its current ‘OK’ status to the destination error condition (e.g., WARN, CRITICAL, OFF, etc.). When a jPOS-EE Web UI user hits the status page, we run that check on all entries; but if for some reason the users are not looking at that page, we need a way to check all statuses and move timed-out entries to the specified error condition. 02_status_heartbeat is in charge of that!

-- 2009-08-19 Update to Original Post --

Federico asks: Seems kind of redundant to me to have an 'active timeout' and 800s at the same time. Am I missing something?

AAO: Very good comment. The 800-series network messages aren’t foolproof. They’re great at keeping the line active during your volume troughs so that the endpoint doesn’t disconnect and cycle your connection. They’re not great at determining and resolving hung lines. The ‘active’ timeout approach works to resolve the ‘half-baked’ channel connection, i.e., the situation where you think you’re connected, but you’re actually not.

Non-approval approvals

2009-08-18T11:53:43-05:00

In OLS' role as an Acquirer-side payment switch, we engage in a lot of practices that fall into the category of what I call 'Defending the Transaction Originator' (not to be confused with 'Defending Your SAF'...though the principles of protective measures are the same). These practices have led to a group of Internal Result Codes that I refer to as the 'Non-Approval Approval' range (4xxx in the OLS.Switch framework). We have:

4000 (FORCE_APPROVAL) - Whenever we get a message that comes from the originator's SAF queue, we're obligated to approve it. But there are situations where we may encounter a legitimate error, most typically in timeout reversal processing where, more often than not, we can't find a corresponding original. The originator's SAF queue doesn't care about your decision-making details. It only wants to know: did you receive the transaction or not? FORCE APPROVAL tells us internally of the error condition and then we map that as an APPROVAL going back to the originator so that we keep their SAF queue clean...lest you run into the dreaded endless transaction loop between originator and host. Not pretty.

The Force Approval transaction participant looks like this:

[See a general discussion of transaction participants here, including a short discussion of the force approval.]

4001 (DUPLICATE_APPROVAL) - We always check for duplicates. If we determine internally that the current transaction is a duplicate, we flag it internally with an IRC marker that denotes it as a known duplicate. The originator doesn't have a clue about that (which is why it sent the same transaction in the first place). We map this 4xxx series code to an APPROVAL going back to the originator. Both sides of the transaction equation are happy.

4002 (STANDIN_APPROVAL) - Some of our customers want stand-in approvals to be denoted with a special result code. So, we've implemented that here.

jPOS config files with markup resolved at runtime

2009-08-03T08:39:05-05:00

jPOS User Group super-member Victor Salaman this weekend posted news of a very useful advancement: "string interpolation and macro support for jPOS config files." Translation: Config files that can contain markup which would be resolved at runtime.

In the piece, Victor made things very relevant by using an example of a real-life TransactionManager transaction participant sequence that I've posted on my blog. You can see the original piece here.

As noted by Victor. we currently use substitution values that are resolved during build time. We have a properties file that looks like this (I've masked some of the details and thinned this out considerably for the blog post):

nodename = APP01
status-id = heartbeat1
status-name = Heartbeat 01
thales-status_0 = HSM_A_0
thales-status_1 = HSM_A_1

server-id = server1
logon-interval = 43200000
fdr0-delay = 60000
fdr1-delay = 5000

fdr0-monitor = FDR1 FDR Hagerstown 01
fdr1-monitor = FDR3 FDR Melville 01
amex0-monitor = AMEX1 AMEX IPC 01
amex1-monitor = AMEX3 AMEX NROC 01

saf-id = saf1 FDR SAF 01 Status

thales_0 = 10.99.2.88:1500
thales_1 = 10.99.2.88:1500

fdr0.host = 206.99.50.88
fdr0.port = 21999
fdr1.host = 206.99.50.88
fdr1.port = 21999
amex0.host = 148.99.39.77
amex0.port = 12777
amex1.host = 148.99.39.77
amex1.port = 12777

fdr.logon.space = jdbm:fdrlogon:log/fdrlogon
amex.logon.space = jdbm:amexlogon:log/amexlogon

lmk = cfg/test.lmk
bdk = cfg/test-bdk.cfg
debug = true
auditTrace = true

sessions.main=112

Now, in Victor's concept these things can get moved to runtime substitution for more flexibility. We have a 'main' TransactionManager that we currently start off like this in our SVN repository:

<main-txnmgr class="org.jpos.transaction.TransactionManager" logger="Q2">
<property name="sessions" value="@sessions.main@" />
<property name="space" value="tspace:default" />
<property name="queue" value="MAIN.TXN" />
<property name="debug" value="@debug@" />

In Victor's concept, you'd instead have a 'deploy-properties.ftl' like this:

[#-- ********* Transaction Manager properties ********* --]
[#assign tmSessions = '112']
[#assign tmDebug = 'true']

...and then:

<main-txnmgr class="org.jpos.transaction.TransactionManager" logger="Q2">
<property name="sessions" value="${tmSessions}" />
<property name="space" value="tspace:default" />
<property name="queue" value="MAIN.TXN" />
<property name="debug" value="${tmDebug}" />

I really the additional level of flexibility this approach provides us. It gives structure to our customers' change requirements in the occasional circumstances where they outstrip the pace of our normal release cycles.

TPS Inflation

2009-07-25T10:42:27-05:00

Transaction Per Second ('TPS') estimates and requirements from potential OLS.Switch clients seem to be doing a reasonable impression of the Zimbabwean Dollar recently. Over the past year, we've gone from sober discussions of credible peak projections of 100, 250, 400 TPS to increasingly chin-scratching numbers of 800, 1,000 and 5,000 TPS. As Alejandro joked to me: Hell, if you're picking numbers, why not go for it? How about 10,000?

I will grant you, Oh Mighty Payment Industry Cog (FDR, Fifth Third, AMEX, etc.), Oh Bentonville, Oh Tar-ZHAY your 1,000+ TPS. But Mr. Big Plans Acquirer? Hats off to your confidence, but I come possessing my doubts and glancing askance at your projections.

Perspective available here [OLS Acquirer Switch at Fortune 100 Retailer - Peak Days by Half Hour]

Fortune 100 Retailer. 5,000 stores. 25,000 points-of-sale. Four US continental time zones. 21 separate applications supported. ~USD30 billion in annual revenue. Sustained (30-minute) peak on all-time highest day...not quite 60 TPS.

Your mileage may vary. But I'm just saying.

[Flash creation courtesy of the awesome Swiff Chart from GlobFX.]

The Return/Refund: It Ain't a Reversal

2009-07-25T09:53:44-05:00

Here's a source of confusion that arises again and again in our payment systems discussions: the difference between a Return and a Reversal. [A 'Return' can also go by the names 'Merchandise Return' or 'Refund.']

What a Return/Refund is emphatically not: a Reversal.

There are two types of reversals (I'm looking at this from the Acquirer perspective): a Terminal-based reversal; and a Host-based reversal. Terminal-based reversals have a "close cousin" in the Void (also referred to a 'cancellation').

We hear things from our OLS.Switch customers, prospects, partners and trial-drivers like:

"Don't we need to keep 60 days of our tranlog online in case returns come in?"

No, you don't. Returns are not tied to originals. They are standalone transactions.

"How often/reliably does an original invoice # get passed back in a message for a return?"

Exactly 0% of the time. Returns are not tied to originals. They are standalone transactions.

"What are our key match-up fields for returns?"

There are none. Returns are not tied to originals. They are standalone transactions.

"Why didn't the issuer reject that return?"

Because most times they don't even see the auth request (I'll show you this further on in the post). Returns are not tied to originals. They are standalone transactions.

In each case, the questioner has conflated the concepts of the Return and the Reversal.

But enough words, the best thing I can do is to show you how these concepts get implemented so you can see the differences in Real Payment Systems.

Here's a group from a TransactionManager that does Return transaction processing for a large acquirer:

<group name="CreditReturn">
<participant class="org.jpos.ev.PopulateCreditTranLog"
     logger="Q2" realm="populate-credit-tranlog">
   <property name="itc" value="03000" />
   <property name="cardType" value="CR" />
   <property name="checkpoint" value="populate-credit-tranlog" />
</participant>
&validate_terminal;
<participant class="org.jpos.ev.FindDuplicate">
   <property name="itc" value="03000" />
   <property name="dupe-check-window" value="2700" />
   <property name="checkpoint" value="find-duplicate" />
</participant>
<participant class="org.jpos.ev.SelectEndPoint">
   <property name="EP1" value="CreditReturnResponse LogAndReply" />
   <property name="EP2" value="CreditReturnResponse LogAndReply" />
   <property name="EP3" value="CreditReturnResponse LogAndReply" />
   <property name="EP4" value="QueryEP4Return LogAndReply" />
   <property name="DUPLICATE" value="DupCreditReturnResponse LogAndReply" />
   <property name="MANAGER_OVERRIDE" value="DummyCreditResponse LogAndReply" />
   <property name="UNKNOWN" value="DeclinedCreditResponse LogAndReply" />
</participant>
</group>

There's some masking there of the actual endpoints in play but rest assured, these are major players in the payment systems industry that we're talking to. Take note of two things:

There's no attempt to match to an original. [You can read about 'FindDuplicate' - an entirely different concept - here.]
Only in one of four cases does the issuer actually want to authorize the Return online. It's the smallest of the four authorizing bodies. In each of the three other cases (Endpoints 1 through 3), the first the issuer learns of the return is in the extract file sent later that evening.

Now, let's compare that to the implementation of a Terminal Reversal group of participants:

<group name="FSASaleReversal">
<participant class="org.jpos.ev.PopulateFSACreditTranLog"
     logger="Q2" realm="populate-fsa-credit-tranlog">
   <property name="itc" value="05301" />
   <property name="cardType" value="FS" />
   <property name="checkpoint" value="populate-fsa-credit-tranlog" />
</participant>

&validate_terminal;


<participant class="org.jpos.ev.SelectEndPointForReversal">
   <property name="EP1" value="FSASaleReversal_Continue LogAndReply" />
   <property name="EP2" value="FSASaleReversalLocal LogAndReply" />
   <property name="EP3" value="FSASaleReversalLocal LogAndReply" />
   <property name="EP4" value="FSASaleReversalLocal LogAndReply" />
   <property name="UNKNOWN" value="DummyFSAResponse LogAndReply" />
</participant>
</group>

<group name="FSASaleReversal_Continue">
&find_original_purchase;


<participant class="org.jpos.ev.WasRemoteAuthorized">
   <property name="yes" value="FSASaleReversalSAF" />
   <property name="no" value="FSASaleReversalLocal" />
</participant>
</group>


<group name="FSASaleReversalSAF">
<participant class="org.jpos.ev.CreateEP1Request"
     logger="Q2" realm="create-ep1-request">
   <property name="mti"        value="0400" />
   <property name="pcode"      value="000000" />
   <property name="template"   value="cfg/ep1-credit-template.xml" />
   <property name="space"      value="jdbm:ep1-stan" />
   <property name="checkpoint" value="create-ep1-request" />
</participant>

&store_and_forward;

<participant class="org.jpos.ev.FlagReversal">
   <property name="reversal-class" value="T" />
</participant>
&force_approval;
&fsa_response;
</group>

Lots of concepts are thrown together in there. I don't want to touch them all here, but - in general - you can see that the reversal treatment is more complex than the return. I chose to show the FSA group here instead of straight credit because the bar is higher in terms of real-time notification of the endpoint. The highlights:

For Endpoint 1 ('EP1'), we go through a group that is going to notify the issuer via Store and Forward (near real-time) that a reversal got done.
Note that we try to find the original purchase!
We always use SAF to send reversals.
We use FlagReversal to tag the original as being reversed (revInd = 'T').
Protect the Originator (!) by ensuring Reversals and Voids always get an Approval ('force_approval').

Note that Voids get routed through the same group. At the point-of-sale, voids differ from reversals in that Voids require an overt clerk or administrator action - e.g., canceling the transaction - whereas reversals are system-generated (e.g., when no response is received to certain transaction class - Debit, EBT, for example - within a proscribed transaction timeout period).

Also note that Host-based Timeouts are intrinsically different than the Terminal-based Timeouts described above. The 'H' varieties are handled via the 'query host' participant of the original. Here's a real-time example:

There, we're giving the endpoint 25 seconds to respond to the Mux. If we don't see a response in that time, we flag the original as being host-reversed (revInd = 'H') and place a corresponding 0420 (reversal) in the endpoint's SAF queue.

One important conceptual difference between terminal-based and host-based reversal scenarios is this: in terminal-based scenario, you end up with two cross-linked records on the tranlog (the original and the reversal); in the host-based scenario, there's only one transaction (the original).

750,000,000

2009-07-13T08:25:42-05:00

We passed the three-quarters -of-a-billion mark this past week at our flagship OLS.Switch acquirer location. The winning ticket was an AMEX credit card transaction for $12.00. You can see in my little screenshot image that we got it authorized remotely by AMEX in a total 750 milliseconds.

That’s apropos for a spotlight transaction – it’s in line with traditional payment processing and the duration is reflective of what we see for remotely authorized credit and offline debit. This transaction category is at the bedrock of what we do. These are fundamental things that any OLTP payment host has to do almost straight out of the box. We’ve got well-honed, repeatable processes in place to “on-board” these types of interfaces in as routine a fashion as possible. [In fact, I admire AMEX’s well-documented, standards-based approach to such a degree that I made it the focus of my on-boarding guide.]

What’s very interesting in examining the transaction make-up is how much our newer, non-traditional initiatives are starting to lift the overall daily transaction counts. The most prominent and notable is the impact of MethCheck…something we’ve on-boarded quite readily into our payments processing platform as a way to help our clients address the pseudoephedrine compliance challenge. We’re now fielding over 10,000 of these transactions (on average) per day.

Put request, response tranlog columns in new table

2009-07-03T10:32:41-05:00

Here’s a Release Note I wrote about a new feature we implemented this week into our production environments (some redacting performed – see items in brackets):

-------------------------------------------------

Put ‘request’, ‘response’ tranlog columns in new table:

In the test system implementation, the defined terminals mostly reference terminal profiles in which the ‘Audit’ flag is turned on (i.e., set to ‘Y’). Doing so provides [the OLS.Switch customer] with visibility to the raw, hexadecimal Store System request and response images in the transaction UI display. This level of detail is often vital while reviewing testing efforts. Behind the scenes, the OLS application stores this raw transactional data in the request and response columns of the tranlog as ‘varbinary’ objects.

In production, the Audit flags for the “in use” terminal profiles are by default set to ‘N’. However, there may be situations in which [an OLS.Switch implementer] is asked (for troubleshooting, fraud detection, etc. reasons) to temporarily track the raw request and response sequences from one or more targeted, production transaction origination points. This raises some ‘handling’ concerns:

For financial transactions, the raw store system request may reveal the card number in the clear (if it is sent unencrypted). Taken alone, this isn’t a concern if undertaken inside a well-fortified PCI Zone with corresponding good practices. However…
Once the exercise is concluded, to subsequently remove evidence of the raw request and response requires direct SQL-level action on the tranlog – either to delete specific rows (not recommended as we want the tranlog to be a faithful record of all attempted transactional activity) or to set the request and response columns to NULL. In general, manipulation of the production tranlog along these lines is a practice that should be avoided.
Should the Audit exercise continue for more than one day, items are copied to the [historical repository] of the tranlog. Raw message information from the production system should not be perpetuated in the repository.

For these reasons, in this release we are moving the ‘request’ and ‘response’ columns out of the tranlog and into a new raw_message table. The structure of the table is as follows (this is the MS SQL Server version of the schema commands):

create table raw_message (
id numeric(19,0) identity not null,
txnid numeric(19,0) null,
type varchar(3) null,
messageData varbinary(MAX),
PRIMARY KEY (id)
);

Where:

id is the auto-incrementing identity column
txnid is a reference to the corresponding identity column in the tranlog
type indicates whether the item is a raw request message (‘REQ’) or raw response message (‘RSP’)
messageData is the binary representation of the raw message as received from (in the case of ‘REQ’) or delivered to (in the case of ‘RSP’) the transaction origination point

Additionally, OLS has introduced a new property (‘auditTrace’). This value must be set to ‘true’ in the TransactionManager in order for raw message logging to take place. [The value is actually engaged in ‘log_and_reply.inc’, a file that is referenced by all TransactionManager instances.]

After the implementation of this change, a production post-audit clean-up becomes a simpler, unencumbered exercise: raw data is not moved to [the historical repository]; and information logged to ‘raw_message’ table can be cleaned up via SQL without engaging the tranlog.

-------------------------------------------------

One follow-up to this Release Note: I asked Dave how we should set ‘auditTrace’ in production – my thought was to set it to ‘true,’ thinking we’d be at the ready to turn on tracing without a service re-cycle. Dave strongly disagreed and stated: OLS.Switch ought to be “Secure by Default” in production. I really liked that. Here’s how I memorialized it in our SVN logs:

[See Dave’s follow-up to my piece here.]

Add node and channelName columns to tranlog

2009-07-03T10:08:02-05:00

Related Readings:

The Q2 Log: Nice Package!

jPOS and SQL Server: A Compilation of Best Practices

Love Your DBA

Here’s a Release Note I wrote about a new feature we implemented this week into our production environments (some redacting performed – see items in brackets):

-------------------------------------------------

Add ‘node’ and ‘channelName’ columns to tranlog:

When troubleshooting issues in the OLS.Switch operating environment, it would be of great assistance to know the node (i.e., APP01 or APP02) and – if external authorization is attempted – channel (e.g., for an FDR transaction, Hagerstown or Chandler) affiliated with a specific transaction. This information allows the investigator to choose the right node for q2 log selection, make well-formed questions to remote partners, formulate SQL queries tallying node- or channel-specific activity, etc.

The node information is populated from the OS-level server name. For example, the server name for APP01 is app01.nn.yyyyyyy.us. [This is not set inside the OLS application; it is OS-level information configured by the System Administrator.] The first portion of the name – ‘app01’ in this example – is placed into the ‘node’ column. By default, the ‘node’ column is populated for every transaction that traversed the OLS application.

The channel information is populated from the ‘channel-adapter’ name in the channel-related XML deploy files. For example, in the FDR set-up, the 10_fdr_channel.xml file has a ‘channel-adapter’ name of ‘fdr’:

<channel-adaptor name='fdr' class="org.jpos.q2.iso.ChannelAdaptor" logger="Q2">

…and the 10_fdr_channel_1.xml file has a ‘channel-adapter’ name of ‘fdr1’:

<channel-adaptor name='fdr1' class="org.jpos.q2.iso.ChannelAdaptor" logger="Q2">

As a general rule, for dual-channel muxes the 10_nnn_channel.xml file has the ‘channel-adapter’ name of ‘nnn’ and the 10_nnn_channel_1.xml file has the ‘channel-adapter’ name of ‘nnn1.’ These names are the same across nodes, i.e., in the FDR example described above ‘fdr’ and ‘fdr1’ are the channel-adapter names on the APP01 and APP02 nodes. When taken together, the node and channelName columns provide sufficient information to determine the external authorization path for a transaction.

The ‘channelName’ column is only populated when:

The transaction class is externally authorized – so, for example, the column is not populated for local application authorization transactions like employee validation and discount coupons.
The transaction attempted an external authorization – so, for example, the column is not populated for a Credit transaction in which the card-id-source = ‘M’ (i.e., a Manager Override – a locally authorized variation of Credit).
The transaction model governing the external interface is ISO8583 (the ‘Channel Info Filter’ – see next bullet – currently does not extend to non-ISO8583 interfaces) – so, for example, the column is not populated on [National Department Store], [Electronic Check Acceptance Provider] and Verizon transactions.
The ‘Channel Info Filter’ is added to the channel’s XML definition file:

<filter class="org.jpos.iso.filter.ChannelInfoFilter" direction="outgoing">
<property name="channel-name" value="1000" />
</filter>

The filter is added to the following ISO8583 interfaces: AMEX; [Private Label Provider]; FDR; Green Dot; Incomm; and SVS

The filter is not added to the following non-ISO8583 interfaces: [National Department Store Interface]; [Electronic Check Acceptance provider]; and Verizon

The filter is not added to the following ‘one-shot channel’, non-ISO8583 interfaces: Authorized Returns; Drivers License Decode; and MethCheck