I know it may sound cleche to state that a picture is worth a thousand words.  But some things are just really difficult to communicate in text.  You may have noticed a few diagrams showing up in my posts here.  Well, I have found that there are times in a conversation with other programmers, and more often in conversations with non-technical collegues that I am saying, “I need a whiteboard.”

Because of this propensity to illustrate my explanations, people have jokingly said that I couldn’t talk if they took my dry erase markers away.  At the same time, being able to communicate with others that are not versed in programming, databases, chemestry, or any other disapline that you find yourself will be a useful skill that will help you with your immediate project, and can advance your career.

When the definition for a data field or element is undefined or otherwise open to the whim of the user, we are asking for trouble.  Not in the sense that this causes trouble.  But how it is then used that causes trouble.  And this is not to pick on the CDATA tag in XML or ZZ qualified elements in EDI.  It seems that every data format of any size of adoption commits this sin of categorisation.

My general recommendation is to avoid using undefined or user defined data.  Relying on a user defined field that we use to solve a problem by working around some limitation (or perceived limitation) of the file data definition, may give us a feeling of power.  But over the long haul turns the implementation into a proprietary implementation that is not reusable with more than one trading partner.

And having each integration using a slightly unique implementation leads to chaos.  Both in suppport, and in the applicaiton of updates.

Undefined data exists.  People will use it.  You will have to deal with it as some time, but whenever possible. avaid making it a critical part of the implementation, and always provide it with a healty dose of error handling.

Just like synergy, a projects time-line is generally greater than the sum of its tasks.  Taking the time to complete tasks in a project together, there will need to be time between tasks.  Some of this time will just be spent updating people, and coordination of project participants.  There are also other needs, like water and bio-breaks.

Just like productivity during the day generally gives people a 15 minute break every 3 hrs or so.  When we plan our projects, we need to include not just the time that it takes to perform the collective tasks, but other activities that will be happening during the course of the project.

Odd errors can occur when you get data that doesn’t fit the format, but matches close enough not to fail outright.  This type of error can be really hard to unravel. Preventing these messy errors, it is a good practice to implement a validation process before processing the data.  Some integration technologies come with validation, if yours doesn’t you should pre-process inbound data for validation.

In addition to this, pre validation can be used as a way to catch and handle other errors early.  Leveraging pre-processing with validation and error handling can give you a one stop place to catch, and report or correct systemic problems.

So it goes like this:

850 is a Purchase order, and is sent to the Vendor

855 is an Order Response, and is sent from the Vendor.  It confirms, updates or rejects each line, or the PO as a whole.

856 is the Ship Notice. and is sent from the Vendor.  It may contain one or more lines from one or more orders and can indicate shipment of partial lines from a PO.

810 is the Invoice, and is sent from the Vendor.  It contains one or more lines from one or more POs, but gennerally people keep these to only one PO.

820 is Remittance, and is sent to the Vendor.  It contains information relating to payment.

The accuracy of the timeline will only be as acurate as the accuracy of the least acurate task estimate.

The time estimate part of project planning is a tricky thing.  Generally there is some guessing involved.  To keep us from falling short a good practice is to pad estimates to account for the unforeseeable.  Project planners can’t just pad and pad.  There is pressure from the other side to keep your estimates accurate and keep your timeline short.  Time is money after all.

So how do we put enough time in the timeline to not fall short, and keep the timeline from getting too long and expensive?  Well that is a dance that takes some practice to learn.

Just remember, when you pad your estimate to your project leader or get and estimate from someone on your project, that padding is necessary, and you should discuss it openly.

Learn the basic tools.  Learn them first, and learn them well.  Sure, you can find some fancy XSLT tools or text editors.  And you can carry them around on a thumb drive and install them on all of you computers, workstations, laptops, and servers.  But sometime, some day, you will need to make a change, or fix a problem and you won’t have the fancy tool.  My experience says that this will also happen in the middle of the night when you are under pressure.  Knowing the basic tools like VI and Notepad, and any other tools that are a standard part of your systems will allow you to get your integration back up and running quickly and get you back to bed so you can go in and answer the questions about what happened with just that much more sleep.

This is like the Covey proverb, “Begin with the end in mind.”  In an integration project, the first step is always to define the target of where your data is going to end up.  This is true for a document conversion like XSLT or a EDI or Flat File transformation to XML or another version.  It is also true for ETL tasks where we are getting data in and out of a Database or repository.

Starting the work of transforming data or files without a well defined target for the data will cause you to do extra and unnecessary work, and to repeat some steps when the true destination is known.  It may seem obvious, but many many people fall into this trap, and then don’t understand why their projects keep exceeding their time budgets.

Keeping a copy of all of the documentation you create is a pretty general benifit.  It helps you in three major ways;

1. Having a personal copy means that if the systems that have the public copies become unavilable, you will still have access to them.
2. Some times projects that get shelved, lose their documentation.  If you have a personal copy, when the project comes back to life, you will not be starting over.
3. And you never know what future project you will be working on that will spark the memory, “Hey we solved a problem like this on this other project…”  And having the documentation for it will help you.

I have never regretted keeping a personal copy of documentation.  But I have always regretted knowing that I didn’t keep one when I could have used it.

Not to be masochistic, but among friends and neighbours a common topic of conversation between men is our occupations. “What do you do?” is a common question and conversation starter.  As we all know, sometimes it is difficult to explain what we do to those outside of our business.  There are people that I have known for years that still don’t really understand what it is that I do at work.

What do you do?

Partially this is caused by the relative youth of the Information Technology fields.  Other fields with a longer history have a high degree of context.  The Butcher, Baker and Candle Stick Maker are all occupations that are so well understood that they are almost self explanatory.  A Systems Administrator or Database Administrator are relatively new and unknown, but they share something with the butcher and maker in that they sound like what they are.

The challenge of history

My first job was doing technical support, and my mother still tells people that I ‘fix computers’ for a living.  Consequently her acquaintances continue to try to get free support from me.  (Don’t even get me started on what my Father-in-law thinks I do.)  I may still have some abilities here, but this is not my job.  It is like saying that a surgeon gives people shots.  Certainly the surgeon has this skill, but this is not what he does.  (And you will have a much less painful experience if you just let the nurse give you the shot.  Trust me.)

So how do I explain what I do?  The first rule is to realize that the part of this job that are exciting to you and me, won’t really translate to conversation with your neighbour hood storekeeper, farmer, or even teacher.  So I recommend not trying.  This does not mean that I don’t talk about it.  This means that I start with what is easily understood and then work towards Integration Engineering.

Lie

Okay, that may be harsh.  But really saying, “I am a computer programmer.”  Only requires explanation if there is a follow up question.  This might come as more and more people dabble in programming.  (And really, “computer programmer” is a rather nebulous term anyway.)  When you say this in at a parent-teacher conference or other semi-social occasion it lets you answer the question and move on to the topic at hand.

Dodge

Again, this is not meant to be harsh.  Say something like, “I work in procurement/supply chain management/on computers is still easy to understand, and is more accurate.  It keeps the relationship with your priest honest, but it doesn’t put him to sleep.  If you try this one on, you will come up with a very effective dodge that is entirely accurate but does not put people into comas or confuse them.  (Always gauge your audience and chose the level of detail carefully.)

Be honest

Normally honesty is the best policy.  Sometimes that works here too.  “I do supply chain automation and integration.  But generally if I go into detail I put people to sleep.  It is very exciting to do, but deathly boring to discuss.”  There, I have been honest and tried to explain why I am shutting the door on further conversation.  I have found that this one works well after your Mom or your spouse has just tried to explain what you do, and mangled it to the point that the “lie” and the “dodge” will no longer work.

So what do you do?

I mean, what do you do when asked what you do.  Do you have a pat answer or do you fumble each time trying to maintain conciousness in your listeners?  Or do you have no trouble with this because you don’t have any social engagements that don’t involve your co-workers?  Post your answers to this simple question, “What do you do?”

I remember a day when we had a problem with our order processing.  When we finally figured out what was wrong, there were a few embarrassed people.  We had introduced some new monitoring to the system, but had done so in such a way that we added a point of failure to the ordering process.

This was and is a bad thing.  And it can be avoided easily by making sure that you use a passive monitoring system.  By its very nature the state of a passive system has no impact on the system being monitored.

File Based System

In a file based system, the pieces that are being monitored create files to report status to the monitor.  The files sit on the file system where the monitor picks them up and processes them.  In this way, if the monitor develops problems, goes down, or gets behind in processing, the production process stays unaffected. (Unless you fill up the disk with files.  But this too can be managed and monitored, passively.)

By designing the status files so that they contain the relevant timestamps and data, it doesn’t even matter in which order they are picked up and processed by the monitor.  And the health of the monitor can be watched by watching the depth and age of files waiting for processing.

Queue Based System

In a queue based system, the process being monitored writes its status to a queue.  This can be a queue that is part of the system, or residing on the database that the product relies on.  It should not be a queue that is on a non-essential system.  Doing so places the queue as a point of failure for monitoring, or worse, for the product.

Similar to the file system, the monitor goes out and reads from the queue.  This allows the same flexibility for the monitor to get behind without impairing production processes.  And the queue depth and age can be monitored in the same way as the file system to check on the health and activity of the monitor.

Missing points of data

This can be frustrating when trying to plug all of the time stamps for the life cycle of your process onto a map, graph, or other representation.  And when there is only a small amount of traffic on your system, losing even one data point can seem important or even critical.  But including some deductive logic into the monitor, these points can be filled in with the data that exists.

This is the same technique I talked about on my post about Sanjay Shaw and RFID in Supply Chain Managment.  When we gather data points from more locations than needed, and one goes missing, we just gloss it over with an estimated point.  And with a passive system we can insert the real point of data later if it shows up.

We had a problem that sometimes we would not get the final status.  Orders were going out, but the monitor was not getting the last status of the system that delivered them.  We were getting the Order responses.  So we tied in the order response status to the order, and when an order that had no transmission time received a status, we filled in the transmitted time with the Order Response transmission time.  This was not entirely accurate, but stopped hundreds of tickets from being created for investigation where the only problem was the status file.

Insurance

To ensure that your passive system is truly passive, the best test is to turn it off, shut it down, and block access to it and any peices of it that can be turned off and still have the production system running.  This test will show you if you have missed or added something that introduces a new point of failure to your environment.  Anything less will not ensure that the monitor is passive.

“So, how long will that take?”  Is a question that some of us have grown to hate.  And it seems that it is a question, that in various forms, we are asked daily.  (If not more frequently)  And if you have been doing this for any time, you have probably come up with a way to answer these questions.  There was a time when I started answering this question with a standard answer of, “2 weeks.”  And let me explain why.

My Answer:

First, let me just say that my answer was really, “40 hours or 2 calendar weeks.”  When asked if I could do it in 40 hours but in one week, I would say, “No.” and explain that I always had interruptions and other priorities to juggle.  There was never a time that I could only focus on just one issue for a whole week.  Thus it would not make sense to give them the expectation that I could be done in a week, even if the actual hours worked on project was 40 hours.

Second, let me say that these where all mapping change requests, or new document mapping requests.  I am extremely good at this, and have a system for keeping things simple and effective.  You can read more about that by reading some of my mapping posts here.  And following that system, I could pull of a changed or new map in 2 weeks with confidence.

When asked if I was sure that it wouldn’t take longer, I would shrug and say, “It never has before.”  Well not a detailed justification for my time spent, this was an expression of confidence to get my time estimate accepted and put on the plan.

Other Answers:

There are many ways to come up with a time estimate.  I took a class once, that discussed project planning and explained Gantt charts and time lines.  I had been using versions of these for years without knowing they had a name.  I called mine a “white board” but I did learn the lingo of projects that was useful when communicating with other how your time line is looking.

Project Planning is a great skill to have, but don’t get carried away.  Lots of time can be spent on making a project plan instead of actually working.  When you are working as a lone ranger, or small team, extensive planning can actually get in the way.

Team Work:

One of the things that let me get away with the “2 Week” time estimate was that I was the only one doing what I was doing.  Its not that I am anti social, it just that I was the only one left from what was once a team of 6.  I was helping support the product, building new features, and writing my own requirements based on tickets and user/manager responses.  And not having to coordinate with other people made me much much faster.  (but of course you have to know what you are doing to pull this off)

I remember sitting in a meeting with a group of developers.  They had been working on an integration process for 6 months and were still months away from completion.  I remember feeling really sorry for them when I realised that my boss had asked me to produce an integration solution for the same thing they had been working on.  Without knowing their situation, I responded with my 2 week estimate.

Well, they were floored, and didn’t believe I could do it.  I did, and they realised why.  I didn’t have team meetings, (it was just me).  I didn’t follow their coding standards and review processes, (not having to teach a team about what I had just done).  I had a test system that looked just like production that I built on, (they had a dev system, that all of their projects were being built on.  And this meant they were always in each others way.)

In the end, (after I had done it my way) they redeveloped it in their way and the project got completed.

This is not a knock on teams.  Teams are important to have.  But sometimes it is smarter and faster, just to give a guy a direction, and then get out of his way.  Teams help with stability and code longevity, but they can slow the process of creation down.  And the bigger the team, the slower it can go.

Its time for another mapping exercise.  This time we will receive an EDI 832 in 4010 format, and map the data to a DB or flat file.  Getting catalog data into your procurement system is an important task.  And creating  variety of mapping exercises provides us with a better understanding of how mapping projects work than just having one that tries to be everything.  And mapping is one of the fun things that we get to do.  Following this exercise we will have worked through the the common issues and demonstrated how this process comes together.

Mapping Steps

There are five steps to this mapping process.  Here is a list with a brief description of each one;

• Defining the target:  This is the first step to mapping.  We must know where we are going if we plan on getting there.  And using a Target based mapping process we decrease the time and efort of the other steps.
• Defining the Data Source:  This is the second step to mapping.  Once we know what the target looks like, we know what data we need to complete it.  So this naturally leads us to defining the source(s) of the data.  This can be a single input data file or record, or it can be multiple types of data from multiple sources.
• Defining the Processes:  Some of the data that we will need in our target will not be in the proper form or format in the source.  We will have to have a process defined in our map to convert the source data into the correct form for the target format.
• Handling customisation Points:  When we are mapping data, we will see points where we will want to make a decision on what to do.  Not all data is created or received in an equal form.  Thus we may have points in our mapping process where we will need to do a different process depending on the source or content of the data.
• Monitoring the Process:  After we have created a mapping process we will need to monitor it so that we can know when something unexpected happens.  To do this we must identify the right places in the mapping process for it to report its status to a monitoring process.  This is the last step in the mapping process.

Catalog Target

If you have a catalog, then you will have your own schema.  We are getting our schema from a standard Open Office DB template.  This way you can create your own version for this exercise.

The Product table in the Open Office Business table has these fields:

• ID, INT
• CategoryID, INT
• Discontinued, BOOLEAN
• LeadTime, VARCHAR
• ProductID, INT
• ProductDescription, VARCHAR
• ProductName, VARCHAR
• ReorderLevel, INT
• Serialnumber, VARCHAR
• SupplierID, INT
• UnitPrice, DECIMAL
• UnitsInStock, INT
• UnitsOnOrder, INT

These are the basic fields that we will be inserting data into.  We will probably have a process that will validate the supplier ID before inserting new records, but that will come in during the process step.

Next Steps

Following defining our target, we will define our source.  But we will do that in the next instalment of this exercise.   Download and create start your paper map now, we will show them in the next post as well.

Starting off work on my On-line Status Repository, one of the things that I will be starting with is uploading and downloading SEF file from a data repository.  SEF stand for Standards Exchange Format.  SEF files are repositories of standards information that can then be exchanged between people and applications to define the format of EDI documents.

If you have used a standards editor, you probably know what an SEF file is, or have used it.  Some applications and EAI even use SEF files as part of their document creation and validation processes.  It becomes useful to describe briefly what SEF files look like, and what type of information they contain.

What is in there?

If you are familiar with EDI, and have cracked open an SEF file with your standard text editor of choice, you will already know what I am about to say.  The contents of the SEF file don’t look that complex to someone who is used to and familiar with how various EDI files look.  For those that are not so brave, I will explain this painlessly.

First, the SEF file is really just a text file.  It can be edited by hand, but I don’t really recommend it.  (not because you can’t, but because it is tedious.)

Second, it contains its data in sections.  There is a section for doc types, segments, elements, and encoded data.  There is also a few housekeeping sections like version and name etc.

What is an SEF file for?

SEF stands for Standards Exchange Format.  This is literally a file format that was designed to contain information about EDI standards.  It was created so that a computer application could understand an EDI file.  It is used by some application like Standard Repositories and Standards Editors to allow us humans to deal with a standard like it was a text document.  While at the same time keeping a repository that can be used by an integration application to form and validate an EDI file.

If you are familiar with the concept of metadata then an SEF file is the EDI standard’s metadata.  If you are not familiar with metadata, then skip this part.

Do I need to know about SEF files and EDI?

Nope.  SEF is only useful if you are exchanging standards using the format.  If your standard and usage is in the form of a PDF or a spreadsheet, that is fine.  Many people do just that.

For me, I need to know this because I am trying to build some Standard Repository tools, and want to use the SEF format to manage them.

And for those that want a deeper look, I will be following this article with some deeper and more detailed articles dealing with the internals of the SEF file.

One of the basic tasks Integration Engineers do is to compare files that we use or receive.  There are some interesting and useful tools that people can get out there to DIFF files.  But on Linux and Unix machines around the world there is a native tool that is almost always present.  Amazingly it is called DIFF.

Like some other command-line tools, its interface is not really intuitive.  Lets walk through the basics of how to get use out of this handy file comparing tool.  (If you are working with and comparing EDI files, you might want to look at the post on how to unwrap your EDI file so that our line by line comparison is more meaningful.)

How to used “diff”

You can get the real basics by executing “diff –help” to get the basic help and options for this application.  But in short, here is the thumb nail. “diff” is followed by some options.  Options are designated by a “-” and then a letter indicating the option.  Any options are then followed by the two file names that are being compared.  Lets look at an example.

Example:

We have two files, file1.txt and file2.txt

When we issue this command:  “diff file1.txt file2.txt” we get this result.

2c2
< And this is the first line of the first file.
—
> And this is the first line of the second file.
3a4
> Again.

• The first thing we see is “2c2″  This is line 2 of the first file, compared or changed to line 2 of second file.
• Next we have a < indicating the first file, and the line echoed.
• Following this we have a “—” as a separator between the line compared.
• Next we have > indicating the second file, and then that line is echoed.
• This is a comparison between to lines that were found to be different.
• For the next line that is shown, we have “3a4″ that indicates that there is a line added to the file.
• Finally, > indicates the second file followed by the line being echoed.

If we were to compare them in the other order, we end with these two lines:

4d3
< Again.

• Here, “4d3″ that the 4th line on the first file is deleted from the second file.
• Following this is < indicating the first file, and echoing the line.

Regular Options

Here are the list of options that “–help” gives you, with maybe some more explanation.

diff [-b] [-i] [-t] [-w] [-c] [-C] [-e] [-f] [-h] [-n] [-D string] [-l] [-r] [-s] [-S name] [fileone filetwo ] [directoryone directorytwo]

For comparing file, the first four options (-b, -i, -t, -w) are the most useful.  I don’t start with any options and add them as I need them to reduce the amount of change noise reported in the result set.

diff is your friend

Like many basic tools, “diff” is almost always there. And if you know how to use it effectively, it can really save time and frustration. Sure there are other cool file comparison tools. Some are even embedded into other products. But knowing how to use the basic tools that are always there will be a life saver in a crisis situation. And the only way to know how to use them is to actually use them sometimes.

Do you use “diff” with a set of options that does a specific task for you? If so, what are they, please share. And what other basic tools do you use?

If you have just a few Trading Partners, having a unique and separate map for each of them might be a good option.  However, if you have plans to scale your integration to 10s, 20s, or 100s and 1000s of trading partners, having a one map to one trading partner strategy is a recipe for a difficult to maintain and support integration solution.  And it doesn’t have to be this way.  There are a few strategies that will help you create an integration that will scale and be easy to support.

Industries have common ways to handle data.

This is something that we should understand.  Mandating that others use your standard usage implementation may work if you are Wal-Mart, but is not really necessary.  Most supply chain integrations are industry specific.  Probably most EDI integrations even the non-supply chain ones are also industry specific.  And you will find that most of the Trading Partners in a industry need to use the same data sets.  And you will find that most of them are using them in about the same way.

What this means is that when you set up a map for one Trading Partner, most of that mapping will also work for the next one.  This allows us to write maps that are flexible and can handle the variations in the data without having to have a map for each partner, or a crazy set of conditions that say, For Trading Partner 1 do xyz.”

This is of course for an inbound process of in other words, EDI data we receive from others.  And it is simple to accept a par number of other piece of data in more than one location and map that to our canonical.

What is more tricky to visualize is how this works on the outbound.  Well it is possible to map a part number or other piece of data to more than one location, this can be confusing to the trading partners that are receiving it.  Instead, we do a pre map manipulation or pre mapping for trading partners that step out of the normal map.  We could also have a post map process if your technology makes that the better choice.

Here is how that will look. First, we will have a map that is our standard preferred mapping.  This map takes our canonical data and maps it into our outbound EDI format.  Next, we create a process that will either modify the structure or content of our canonical to reflect the custom needs.  We also adjust the map to recognize and handle the new fields. Alternatively we could have a post map that maps our standard outbound EDI and alters data or structure.

Strategies:

1.  Broad mapping:

This is a mapping strategy that I have found works well on mapping data received well from Trading Partners.  One illustration can be found in the discussion of line item data.   Where a specific piece of data may be found in more than one location, you create your map to accept this data in all of the valid locations and out put that into your canonical.

It is also possible to do this on the outbound or map that formats data for consumption by your Trading Partners. This is not as straight forward, but there are times when you can see conditions in the data that cause the format or output to change. When these conditions exist, include them in the mapping and you will have a more agile and flexible map.

2.  Pre mapping:

This is a map before the map.  Well it doesn’t really have to be a whole map.  But the concept is this;  Data received or sent to a Trading Partner needs to be changed on a more Trading Partner specific basis.  It doesn’t actually need to be a TP condition, these pre maps can detect a data format, and alter it before sending it to the map.

For instance, I had a instance where we had a TP that was consistently sending us a DUNS number with the ID of “1″ and we needed “01″ for our system.   So we put together a pre-process that would look for these qualifiers of “1″ and altered it to “01″.  In this way our map didn’t have to have funky code, and we were able to integrate with these trading partners without having to ask them to change their vendor data in their DB that was trimming leading zeroes.  In any case, I have found that pre mapping is useful when there are consistent standards violations or data manipulation that can be corrected systematically.

3.  Post mapping:

This is similar to the pre mapping, but your process happens after the map.  This is a good use for mapping data out to your Trading Partner.  And again, they don’t have to be trading partner specific.  Data going out is altered slightly in form or format before it is delivered.

Again, I had a trading partner that transmitted catalog and part lookups in a format with a dash in the part number.  But they didn’t like getting the dash back on their orders.  So we instituted a simple mapping change that would strip dashes for them.  But later we realized that having a process to fix part number formats was something that more than one Vendor Trading Partner was requesting.  So we moved the mapping condition to a post mapping process that would look at the TP and find any part number rule and apply it.

Flexible and Agile is just better.

Okay, so that is a broad generalization.  But I can back it up with years of experience.  I never regret mapping broadly and implementing pre and post processes.  But I do regret when I have mapped narrowly, tried to have a single trading partner specific map, or include all types of TP specific logic into a map.

When two trading partners agree to send each other electronic documents.  And they begin to describe what EDI documents they will exchange and how the documents will flow, they should also exchange EDI specification documents.  EDI usage or specification documents describe what fields and what segments a trading partner will send or expect to convey the information necessary to complete a transaction.  It doesn’t matter if we are ordering widgets, or invoicing, or transmitting catalog data, or checking insurance claims eligibility, the EDI needs to contain the data that the two parties need to communicate.  To explain this, and document it to that both trading partners know what is expected, we create an EDI usage specification.

EDI Specifications

In the larger sense, the EDI specification is the set of rules that define each document type, the segments they contain, and the size and type of data in the elements.  When we talk about the EDI Standard Specification, we are talking about the whole set of valid EDI document types.  If something is valid EDI, then it complies with the EDI Standard Specification.  However, this large, all encompassing specification is not useful in coordinating the exchange of documents between two trading partners.

What goes in an EDI Usage Specification?

So we take EDI Standard Specification and we reduce it.  We remove the unused document types and the unused segments within the documents we will use.  And we even remove the unused elements and encoded values within the segments we will use.  At this point, we have created an EDI Usage Specification.  We can also add to the specification by including values that we need, like designating the an optional value or ID as required and specifying what type is should be.

How is an EDI specification Used?

Now that we have a usage specification we can use it to do to basic things.  First, it is the source of information for us to set-up our integration for our EDI document exchange.  And Second it is the format that we validate our EDI documents with to determine a valid transaction.

EDI Usage Specifications can be a source of data and integration documentation.  This can become extremely valuable when the choices and information about how your EDI interface works is recorded into the notes of the Usage Specification.

Beyond a visual validation of looking at the EDI file and comparing it with the specification, many times a specification can be found in the form of an SEF file.  The SEF file can be used in an EDI validation application.  This allows a potentially large and unwieldy EDI file to be scanned for compliance and accuracy.

There may be other uses, but these are the main two, integration data repository, tool for EDI validation.

EDI Specification Examples:

To walk through how a usage specification works, and is used, check out this review of BookNet Canada’s specification.

Then you can take a look at the EDI specifications published by Macy’sNet that they keep online to assist their trading partners in making and keeping effective integrations.

Sometimes we only use the EDI specification, they way we have always used them.  It can be instructive to browse the usage specifications of other companies.  We see what they remove, what they include, what comments they make, and what rules they document.  And seeing how others do things can help us to be more effective in how we use EDI specification documents.

I started thinking that I would create a comprehensive integration guide and framework. A document that would guide a person from the beginning to end of an integration, covering all the bases, and addressing all the issues. Then I ditched the idea. No one would ever want to read it. (not even me.) Instead here are 7 simple steps that should get your going, and that you should follow in each integration project. The rest of the plan is largely, and subjectively up to you.

1.  Identify the Trading Partners

This may seem really simple.  And it is.  But it is not too simple for us.

We need to know who we will integrate with.  Are these new contacts that have never worked with our company before?  Or are they existing customers that will be moving to this new integrated connections?  Both groups have their own sets of challenges and benefits.

Is this integration a one time event, or will we be integrating multiple trading partners using this same process?  This is also important and will make a difference on how trading partner specific we want to make it.

What is the technical sophistication of the proposed trading partners?  Are they going to be able to comply with our needs or will we have to do everything?

2.  Identify the Data

Again, this is basic but not too basic.

We need to know if we are dealing with just orders, or if there will be invoicing and catalogs.  We need to know if what type of performance we are looking for.  Is this a once a month catalog load that can take hours and gets a couple or retry windows, or is this real time price comparison that need to respond in seconds.

The data and how it needs to flow will be a big piece of what we need to know.

3.  Identify the Target

What is the target?

This can be taking some of your data and sending it to your Trading Partner.  In this case your target is the format and communication protocol that your partner can receive.

Or it can be an internal Canonical Data Format that your systems use that you are creating from transactions received from your partner sends you.

Or it could be something else.  Whatever it is, you will need to know what it is, and know all about it before you can really get started.

4.  Identify the Tools and Technology

Are you using EDI?  Or XML?  What EAI tools are you and your partner using?  SAP?  WebMethods?  GXS?  Other?

How do complete the task will rely to a great degree on what tools and technologies you are working with.  If you are an employee, you may already know this, but if you are a consultant you probably don’t.  You need to find out.

5.  Identify Support needs.

What happens when things go wrong?  Not necessarily “horribly wrong” but just wrong.  Is an error produced when the translation fails?  How about creating a 850 with no line items?  (no one wants to receive this.)  Or if the https post or SFTP send fails, can it retry, or does it die quietly?  These are questions that you need to ask and keep asking.  (Please include all of the answers in your documentation.)

6.  Identify the Time Frame

How long do you have to complete the integration?  Is there a plan to find and fix bugs after implementation?  How long is the integration planning to be used?  (short term during the merger of two companies, or for as long as Walmart will buy your product?)

All of the aspects of time, time to complete, time to live and time to revisit and fix problems are good things to know, or at least have an idea about.

7.  Get started, and Document your work.

Getting started, and starting the Documentation of a project should happen together.  Almost at the same moment.  Okay, really, at the same moment.

Documenting the answers to the questions above along with others is really the start of the integration project.  And the documentation process will help you to focus your work on getting the answers that you need to finish the project.

When you know what you should be doing, and how you should do it, many tasks become easy.  One of the illustrations of this is shown in my Mapping Excersizes.

ACK is an Acknowledgement segment. It is commonly found on response transactions to Purchase Orders or 855. It is one of the lines found in the PO1 loop or group.  As this line modifies the PO1 line data from the Purchase Order, what it can contain and how that data is handled is important for the stability of the supply chain integration.  For clarity, we are going to use the 4010 x12 standard.

Parts of ACK

Here is a diagram of a sample ACK segment:

Like any other segment in EDI, the ACK segment is composed of elements.  Here is a list, element by element and what it contains.

Best Practices

With the ACK the best practices are more in how to implement and what to expect.  So here are some things to remember when implementing this segment.

1. An ACK is part of a PO1 line for a response to another PO1 line transaction.  This segment supplies a status, and can modify the line item contents.
2. There can be more than one ACK segment for a line item.  (4010 855s allow 104 repetitions of the ACK)
3. When an ACK explicitly changes the line item it can only change that one type.
4. There is a status called Accepted With Changes.  “AC”  For this, you will need to parse the ACK and see if any of the values have changed.