As we customers do more and more of our initial shopping on the internet, businesses that don’t show up on a search are never visited.

My example:

A couple of years ago, the company I worked for was discarding some equipment.  Among the items being disposed of was a photography tripod.  It was of good quality, but was missing the piece that attaches to the camera and then clips into the mount.  (I don’t know what this is called, but will update this post with the right name if someone will put it in the comments.)

So I grabbed it and immediately searched for it on-line to see if I could find a store that sold the parts.

And I did, in Toronto.  (Not trying to pick on Canada)

And I bought it and had it shipped to me.  (had to wait like a week or something.)

Now my boss was a photography nut.  And soon after I had this new toy, he stopped in a photography shop to get some filters or something.  (I don’t know I just point the camera and pray.)  We had gone to lunch together and so I went in too, just to look around.

Well, of course I saw that they had the same tripod, and sold the mount thing for the same price.

But they didn’t have a website, or a list of products or anything on-line to let me know they were there!

I see this happening all of the time.  Perfectly good businesses and they are failing to do the basics.

• Every business, (yes, even the shoe store), should request and gather email addresses from web site and physical site visitors.
• Every business should have a website, and list some products on it.
• Every business should create some useful and pertinent content and send it to their customers to build a relationship and keep reminding the customer about the great experience they had.

I can’t think of any reason not to do three these basic things.

Will doing these get you to the front page of Google?  Probably not.  But I guarantee that not doing these is costing you money.  (and it costs the customer money to when they can’t find what they want and need.)

If you want to know more, lets talk. roy@theIntegrationEngineer.com

Or check out MarketMilestone.com to find out more.

Now let me explain this statement just a bit.  A business is a business when it starts to provide its goods and services to others.  In the past (distant past) people could and would “put out their shingle” to start serving their customers.  Times were simpler then.  And almost all markets were local.  At that time, the act of “putting out your shingle” was all you had to do to let your market know and you were in business.

In today’s market, a website is the modern day shingle.   Businesses aren’t really businesses if they don’t have a website.

But that is not all they need to do.

Some businesses are doing the equivalent to hanging their shingle out in the back yard or in back alleys where only people who know them will ever see it.  These are businesses that put up a website, but have used the wrong strategy, used the wrong keywords, and have no relationships with any other websites that can let search engines and people know where they are.

If you have a business, but aren’t getting visitors and customers from web traffic, you are probably doing something wrong.

Lets talk.  roy@theIntegrationEngineer.com

Or check out MarketMilestone.com to find out more.

First you will need to have a Google Analytics account.  If you don’t have one, you might want to check it out.  Its free and you can sign up at http://www.google.com/analytics

As soon as you are ready with you Google Analytics account.  And you have told it that you want to monitor your site.  It will generate some code for you to place on every page that you want to monitor.  But don’t worry, Magento doesn’t need you to do much to accomplish this.  If you look at your accounts page, you will see something like this:

I have blurred out my domain and account, but what you will see in the red box, is the number that you will need to put into Magento.

So copy it to a text file or just leave the page up and go back to it when you need this number.

Now, go to your Admin page on your Magento installation.  You will need to go to System>configuration as shown below.

Once in configuration, you should find a set of navigation links on the left hand of the page.  In the “Sales” section you will find a “GoogleAPI” selection.

This will take you to the section with a Google Analytics control.

Expand the Google Analytics control and set Enabled to Yes.

Now here is where you type or past in your account number from the Google Analytics report page we saw at the top of this article.

Once you have typed it in, click on the “Save Config” button.

When you see this message, you are ready to go.

You might have to wait a few hours before the green check mark appears on the Google Analytics site.

Have a plan of what you want the reader to understand.

When you start taking notes, you won’t have this plan, and that is okay.  Notes are mostly for yourself.  But by the time you start to create a document that you want someone else to read and understand, you will need to have a plan on what the reader will take away from the document.  Ask yourself, “Is this a ‘how to’, or a ‘how works’ document?”

There is some overlap and subcategories for these two general types, but in basic if you are documenting how to start the system, or documenting how the parts of the system work together, you will use a different approach.  There will be some examples in the ‘how works’ document that will be like the ‘how to’  and any good ‘how to’ explains some of ‘how works’ that is inherent in the doing.  But these two approaches produce documents that look very differently.

Where is this document going?

As a guide for the focus, (and if you are like me that you have to write documentation in several sittings), place a bold “How To” or “How Works” at the top of the document or where you will see it when you come back to continue with your documentation task.  This is just a direction guide.  So you will remove this before sharing this document.  And it is just a guide, you can still include how to and how works information when you need it to make your point and get the information across that you are trying to convey.

The purpose of your document is …

Another way to keep your focus and direction consistent is to write the summary first.  And I do mean first.  Lead the document with a paragraph that says, “This document will explain XXXX.  When you have read this document you should be able to X, Y and Z and understand Q.”  And this is a paragraph that you may want to keep when you are done.  It is nice to find a document that gives you exactly what you need.  It is even nicer when you don’t have to read multiple documents to find it but just the headings.

Break it down.

After deciding on the direction, and what your target take-away is, you will probably have more than one part.  Even if this is just a single page “How To.”  So you can now break this out by creating your topic headings.  Sections like, “Tools Needed”, “Access Needed” and “Knowledge Needed” are good starters.  Pick what applies to you.

If you don’t know what sections you should have, take a look at documentation that already exists and see how it is broken up.  This is not cheating.  Not only does this solve your problem of coming up with sections, but it can help make your documentation more consistent.  And documentation that is in a coherent format with other documentation in the same repository is easier to read and understand.

Starting the work of transforming data or files without a well defined target for the data will cause you to do extra and unnecessary work, and to repeat some steps when the true destination is known.  It may seem obvious, but many many people fall into this trap, and then don’t understand why their projects keep exceeding their time budgets.

This introduction aims at addressing the basic questions of convention and implementation of the EDI technology.  I have a list of topics, and hope that over time I can cover all of them.  I also hope that people reading here may have questions that I haven’t thought to write about.  Please let  me know what your questions are and I will see what I can do about providing answers.

• What is EDI?
• Parts of EDI
• Segments
• Delimiters
• Conventions of EDI
• Why are there more than one version of EDI?
• Standards and Usage
• Encoded Values
• EDI in Good Form
• Element Notation
• Wrapped and Unwrapped EDI
• Enveloping Part One
• Enveloping Part Two – ISA
  
• Enveloping Part Three – GS
• Enveloping Part Four – ST
• Loops in EDI
• Hierarchical Levels

Of course understanding how EDI presents and represents data is the first step.  From here we may want to go to our Data Mapping Guide, or if we are familiar with mapping there are some examples in our mapping case studies of EDI to and from other formats.  Again, if you have an interesting problem or solution, I would love to here about it.  Please send it to
roy@TheIntegrationEngineer.com and I will see what I can do.

1. Having a personal copy means that if the systems that have the public copies become unavailable, you will still have access to them.
2. Some times projects that get shelved, lose their documentation.  If you have a personal copy, when the project comes back to life, you will not be starting over.
3. And you never know what future project you will be working on that will spark the memory, “Hey we solved a problem like this on this other project…”  And having the documentation for it will help you.

I have never regretted keeping a personal copy of documentation.  But I have always regretted knowing that I didn’t keep one when I could have used it.

With this as the abstract version of our goal. We break the task into three sections. Data types that we will exchange both outbound and inbound. What transport or connectivity solutions we will used to ensure a robust and reliable transmission and reception of the data. And what data translation, storage and processing solutions we will employ.

Data Types

Many people don’t think about this part very much because they have only dealt with EDI systems that are already in service or in the process of being launched.  By this point the types of data being passed between trading partners have already been decided.  So to begin this process we ask what type of business we are conducting.

If we are doing e-commerce, we are probably going to exchange orders, confirmations, invoices and ship notices.  If we are a retailer, we may be receiving catalog data, where if we are a vendor we will be transmitting it.  And so forth.

We can find out what document types to use in two easy ways.  First, any of our trading partners currently exchange EDI data with other parties, (like integrating with a vendor that has other customers doing EDI), we can ask for if we haven’t already received the EDI specifications that they use.  If we are setting up a new integration with a new trading partner with no EDI history, we will then need to look at the list of EDI document types and find the ones that will fill our needs of data exchange.  (All of the EDI document specifications have a name as well as a number.  For this exercise we will need to look at their name.)

Transformation, Processing and Storage

For data coming into the process for our trading partners is what I call “inbound” data.  And so of course data going out to a trading partner is called “outbound” data.  In either case we start the Transformation, processing and storage by identifying what our destination format, and location will be for the data.  Inbound data has the destination of whatever process or storage we are going to place it, and in the format that we need it in to process or store.  And outbound data has the destination of our connectivity solution to our trading partner and the format of the EDI document type that we have agreed to use.

This is called Target Based planning and it goes hand in hand with Target Based Mapping.  We plan by starting with where we want to end up, in the transformation, processing and storage of the data.  Then we work backword from that goal and create a way to get the data from where it is, to where we need it to be.

For instance if we have the destination of an EDI 850 on our AS2 connection, we start with a standards format of the 850 the trading partner can accept and the location or API that our AS2 connection needs us to use.  After we have that target defined, we map and plan out how to get our order data to that location, coming out of our database, ERP, e-commerce system or some file that we are processing.  (Possibly from a combination of these.)

And we do the opposite if we are processing an inbound 850, we start with the format and interface that we need to input an order into our system, and then build the process to get the information we need out of the 850 and trading partner database etc.

Connectivity

This is a critically important issue, that we will not go into detail here.  But you will want a connectivity solution that is reliable and provides you with a positive confirmation that a file has been received by the other partner and establishes accountability for the transaction, and a process for remediation if there is any dispute.  I will have to talk more about connectivity another time when I can go into greater detail.

Summary

Breaking big tasks like rolling out an EDI strategy can become very complex.  There are two secrets to being successful.  First, break your big tasks into three pieces; Inputs, Outputs and Processes.  The second secret is to begin with your end, or goal, or target.  And then work back from there.  Putting these two pieces of advice into use will make your EDI strategy more manageable and come of more smoothly.

While working for a certain company, I requested some integration information from a manager, and got the information in an attached ppt file.  I soon noticed that all of the managers used power point presentations to communicate everything.  Yes, we had interminable meetings where managers would show a slide show that they would read to us, but also received benefits, project updates and other data in ppt file attachments.

Well, I immediately realized that this was a key to communicating with my manager.  And after some observation determined that is was a part of my companies management culture.  (And yes, I started creating power point presentation to communicate with them more effectively.)

But this power point preference is not pervasive.  I have worked at many companies since where almost no one used power point even when giving presentations.  So I have observed that with every company culture, there are communication norms that have developed.  And if people work there long enough they don’t even realize that they are creating a PDF file when it is not needed, because all important data comes in a PDF attachment.

For the outsider, or consultant, observing what methods of communication are used, and then responding with the same type or form of communication may open doors and establish you as part of the team faster than just having good information and advice.  Adopting the ‘language of the tribe’ so to speak, is a rite of passage that you should not try to avoid.

Early in my integration career I was working on a medical supply chain.  Accuracy was important as you can imagine.  And there was a good deal of stress that we shared as we brought our products on-line and let them encounter real data.

We tried to ensure that the data that we sent to suppliers was received, and when things when wrong we were on the phone quickly asking for confirmation that orders were received.  As time when on we became better both at reliably getting data to the correct place, and in determining when problems where really problems.

“There was one Vendor that sold capital equipments.  So they received few orders, but when they did, their orders were for a high dollar items.  They also kept shutting down the system that we sent orders to.  I think years went by where every order was accompanied by a phone call to their systems administrator asking that they re-boot their server.”

The EDI way

The easiest and most EDI like way to establish accountability of EDI transactions is the use of a 997 transaction.  Basically a 997 works as an acknowledgment.  The name of this document actually is Functional Acknowledgment.  So we are acknowledging that a specific functional EDI component was received, and possibly how it was received.

If you are versed in the EDI enveloping levels you have already realized the  link of between the 997 and the GS level of the EDI envelope.  Also you may remember that you can request a 997 be returned for your transaction in the ISA segment in ISA_14.  But merely setting up your ISA to request a 997 does not actually force your trading partner to comply.   Your trading partner may not be able to, or may not wish to return a 997 for either business or technology reasons.

Assuming that you are able to get a 997 back.  Just receiving the 997 does not mean that all is well with your transaction.  The 997 is an EDI transaction.  It can contain data indicating just the success of receiving the data, or a few validation levels indicating data corruption in transit or rejection for business process reasons.

The Transport Technology way

Another way is to place the accountability on the transport of the message.  Using AS2 or other transport technologies that attempt to guarantee delivery of messages, it is possible to extend our reliable message solution to transactions of multiple types.  AS2 and other transport solutions return an MDN or message disposition notification letting the transporting party know that its message has been received.

The BPM way

In spite of the facility of EDI to process and utilize 997s and of AS2 to utilize MDNs, the most exciting method of assuring message delivery and accountability is utilizing business intelligence.  For me, I remember a system that we developed to help us with the health care supply chain.  We simple began tracking the response time that was typical for each of our supplier trading partners.  After gathering trending data, we applied alerts when responses were overdue by a specified degree of variance from the norm.

Since this process relied on the past 6 months of performance as the baseline, it was adaptive to changes in the vendor.  We could also manually change the amount and degree if we started getting false alerts.  In this way we had a way to react to failed transaction without requiring agreement and arrangements with the trading partner to confirm receipt or processing of transactions.

The Business Agreement

Even in cases where you have implemented a BPM process to alert to failed transactions, you have to have an agreement and arrangement with your trading partners that allow you to deal with failures in transmission and processing of your e-commerce transactions.  At a minimum this needs to include contact information for support teams on each side of the transaction.  It does no good to know there is a problem with a transaction but to be unable to solve the problem because the other party is unavailable.

In the end, it is what you have agreed with our trading partner that make the sender or receiver responsible for processing the transaction.

Best Practices should be the basis for your project plans.

Hopefully your project plans are started by using a project plan template.  If not, then you should.  On this template, your “Best Practices” should be represented.  Too many times I have been on projects where the project manager starts by gathing the information about what must be done.  Instead, start with how it should be done. Otherwise you will encounter the painful process of trying to apply best practices to a project that is already in motion.

Be sure to sign up to be notified when the video of the complete presentation becomes available.

Sometimes resource planning and project management pressure encourages us to allocate additional engineering resources to development projects.  And there are definitely times when having a team is the best way to work quickly.

However, on integration projects more than a couple people slows the progress of the project.

I was once involved in a merger of companies.  The acquiring company had a large team of developers working on replacing the functionality of my company, the acquired one.  I became a team of one and the plan was that I would maintain the systems until the migration.

Management soon became frustrated as new feature development stagnated.  So they called me and asked how long it would take to produce a specific new integration feature.  My estimate, 40hr over two weeks.  Then they called a meeting and asked the leader of the development team how long.  His estimate, 6 months or more.

This wasn’t solely a matter of competence.  Sure, I was still there because I was the best on my technology.  But the other team was not populated with slouches.  There were just too darn many of them.  They got in each others way.

I, on the other hand, had no one to coordinate with.  No one to slow me down.

Teams of one are not always the best, but teams of 50 are never the best.

Or as a friend of mine likes to say, “It takes one woman nine months to have one baby, but putting nine women on the job doesn’t get you a baby in only one months.”

For me, Pre Hypertext Processing is the meaning of PHP.

PHP is a server side scripting language like ASP.

PHP allows dynamic activities to be embedded into web pages.  On the web-server php pages can access databases, directories, etc.  But to the web browser only a web page is presented. With care, a PHP page can create tools to do many clever things.  But more on that later.

Data Integration is the process of transforming heterogeneous data into a useful homogeneous data set.

All of the techniques and disciplines that we use as integration engineer help us to transform (not must map) heterogeneous data (data in more than one form, source, type, etc)  into a useful (not common, and not necessarily the final) homogeneous (single form, schema, location, etc) data set.

There are two parts to this task.  First is the Administration Cache Refresh.  Second is the Cache File Removal.

Administration Cache Refresh

Login to the Administration Panel

Go to the System menu

You will find the Cache Management here.

Once here, select all of the options unless you want a more narrow refresh.  Then choose “Refresh” from the drop down menu.

Then Click on the “Save Cache Settings” button.

Cache File Removal

Sometimes you will have made dramatic changes to the theme or CMS of your Magento store.  In these cases, it is safer to also remove the cached files.  The cache files are located in this directory:

[Magento Root]/var/cache/

At the command line, browse to this directory, and issue “rm -rf *” to remove all of the cache sub-directories.  You only want to do this after refreshing all of the cache from the Administration Menu.

There are two versions of the funnel.  First is where we pick a format and map that data to our internal system.  Then other formats we modify to be in this preferred format, and then map then through the original process.

Second, we create a intermediary format of all of the required data, and them translate all external data to this format of convenience.  Using a convenient dataset as our intermediate format, we can apply mapping logic in the external translation, and business logic on the importation of the convenient data.

I prefer using the convenient dataset, but either way we are able to reuse the mapping logic and using that as leverage are able to do more rapid integrations.

I hate it when that happens.  So as a form of perl tutorial, I am posting my working script here as an example of sending email from a limited server using perl and the MIME::Lite library.

• Server:  CentOS 5.2
• Internet connection: Present
• Task:  Automation Scripts written in Perl

So I had perl, but how to send an email.  Well the first thing, is get the mail libraries.  On a Red Hat based server, this is yum install perl-MIME-tools.  You may have a different way.  But this is required for the sending messages in MIME.  (if you can’t install perl and perl modules, you may want to use CPAN)

Then I wrote this perl script:

—-begin code  —-

#!/usr/bin/perl
#declare that you are using the MIME library at the top.
use MIME::Lite;

#list files in /Email into array
opendir MYDIR, “/share/Email/” or die “couldn’t open $! \n”;

#This will only grab files with “PDF” in their name.

#this will eliminate other files, and more importantly, “.” and “..” that are directories.
@contents = grep(/pdf/,readdir (MYDIR));
closedir MYDIR;

#we want to know how many files are in our list.
$fileCount = @contents;

#loop array to sendmail function
for ($i = 0; $i < $fileCount; ++$i){
$fileName = “/share/Email” . $contents[$i];
$Subject = “Alert Request for: ” . $contents[$i];
#print $fileName . “\n”;

my $msg = MIME::Lite->new(
From    => ‘mail@yourserver.com’,
To    => ‘roy@TheIntegrationEngineer.com’,
Cc    => ”,
Subject    => $Subject,
Type    => ‘multipart/mixed’,
);

$msg->attach(
Type    => ‘Text’,
Data    => ‘Please read this file’,
);

$msg->attach(
Type    => ‘Data’,
Path    => $fileName,
Filename    => $contents[$i],
);

$msg->send(’smtp’, ‘mail.yourserver.com’);

#Remove File
$cmd = “mv ” . $fileName . ” /alert/backup/” . $filename . “.” . epoc . ” \n”;
print $cmd;
system ($cmd);
}

—- end code —-

Summary

Basically this script polls the specified directory for PDF files.  If it finds one or more it grabs each of them and emails them to me.  Then, as a last step, it moves the file to a backup location to be archived.

There are probably other ways to do this, but this is a script that I threw together and have working on one of my severs.  If anyone want to add some corrections to this I will try them out and post my results.

However, my father was a mechanical engineer, and he had a different philosophy that he wanted me to learn.  Routinely after I had “fixed” something. He would show me how getting the right tool,  and part worked so much better and longer.

The same is true in software development and integration work.  However the bailing wire and duct-tape have been replaced with buzz-word products.

Many projects get bogged down as a new, cutting edge technology that sounded cool is added to the project when an existing tool or application will do the job.

The right tool for the right job is better than a cool tool that almost words for the job.