Flux and Floify

We're Hiring!

Thu, 09 Aug 2012 12:01:28 -0700

Sales at Flux — Boulder, Colorado http://lnkd.in/Q56kzf

Floify is a hybrid cloud-based workflow service for business.

Mon, 13 Feb 2012 11:35:03 -0800

Floify is a hybrid cloud-based workflow service for business.

5 Best Practices in Managed File Transfer

Fri, 21 Oct 2011 14:33:46 -0700

Must-Haves for Trouble-Free File Transfers

When you embark on a new managed file transfer project or you just want to review your existing file transfer processes, here are 5 things for you to review in order to make your system more trustworthy to everyone involved and to create smoothly running processes that require minimal human intervention.

1. Confirm Your File Transmissions

Everyone wants to know that their files arrived successfully. Your company, your vendors, your partners — everybody. An explicit confirmation is great. A simple way to do that is to have the receiver generate a feedback file to confirm that everything was received correctly. This feedback file can be sent back to the sender by any way that makes sense:

Placed on the sender’s FTP server
Sent by email
Delivered via a Web Service invocation
Placed onto a message queue

2. Transmission Failures and Notices

What if a file transfer fails? If a file literally cannot be sent from one FTP server to another, the failure could be for common reasons such as:

Network problems
Server down
Unable to login
Path to file doesn’t exist
Insufficient permissions to access file

A common best practice is to soft fail for a problem above, wait one hour, and then retry automatically, without any human intervention. After five soft failures, a hard failure causes a failure notification to be sent to both the sending FTP server’s staff as well as the receiving FTP server’s staff.

3. Files Fail Verification

Files can transfer successfully, but they don’t arrive in the expected format. For example, a zip file might be corrupted or there might be some missing files from an uncorrupted zip file.

In this failure scenario, soft and hard failures don’t apply.

Common best practices include:

On the receiving side, verify files after reception.
On the sending side, verify large files before transmission.
Immediately notify both sending and receiving staff in case of verification failure.
Sometimes, when verification fails from the sending side, notification to the receiving side can be delayed if it’s possible that the problem can be fixed before the transmission deadline.

4. Best Protocol

Oftentimes you are constrained as to what file transfer protocol to use. One side or the other may only support one protocol and be unable or unwilling to change.

However, if you do have a choice, using SFTP (SSH File Transfer Protocol) is a best practice. It’s widely used, well supported and understood, and very secure.

5. Using Dates and Timestamps in File Naming Conventions

New data files are usually generated daily, even hourly. To avoid confusion and to prevent older files from being overwritten with current data, you need to keep these data sets cleanly separated from each other. Use dates and timestamps embedded in your file names to achieve this goal.

2011-10-18-sales.zip (sales data for 18 Oct 2011)
18-Oct-2011-sales.zip (a different date format)
18-Oct-2011-09-00.zip (include hourly sales data for 09:00)
monday-morning-sales.zip (Monday morning’s sales data)
2011-10-18-09-45.zip (sales data for 09:45 on 18 Oct 2011)

5 Best Practices in Managed File Transfer

Use these 5 best practices to create a robust managed file transfer environment for your organization and your file trading partners. These 5 things offer a simpler environment, allowing your managed file transfer operations to be efficiently integrated with other systems, applications, and databases. These 5 best practices have proven successful in saving valuable engineering bandwidth and time for your operations staff.

Original blog post: http://flux.cm/opdtsh

Micro Case Study of a Data Workflow

Thu, 14 Apr 2011 14:21:00 -0700

Micro case study of a data workflow for a typical Flux user:

Watch an FTP server for archive files to appear.
Download those archive files and extract the data files from the archives.
For each data file, perform the following in parallel…
…Invoke a Web Service using that file
…Load the file into the database using a stored procedure

Orchestrate ETL Processes in SAP in a Few Easy Steps

Mon, 11 Apr 2011 11:07:00 -0700

This blog post shows how to implement typical ETL workflows in SAP in a few easy steps.

ETL processes are not an island. They extract data from some source, transform the data in the way the business requires, and load the transformed data to a destination.

A simple scenario for an ETL process, like SAP’s BusinessObjects Data Services, is to:

Download files from an FTP or file server
Initiate an ETL process for those files
Archive the downloaded files after they have been processed

Here’s a workflow that illustrates these steps.

How can we implement typical ETL scenarios like this in SAP in a few easy steps?

Implement Flux. Flux lets you automate ETL scenarios using file, process, and workflow orchestration. Flux is pretty simple to install, use, and configure. Flux is cost-effective.
Configure Flux to initiate ETL processes in BusinessObjects Data Services. Because Data Services supports a Web Services interface, Flux can connect to Data Services over HTTP or HTTPS, initiate an ETL process, and wait for the results.

Conclusion — This Is Pretty Simple

ETL workflow solutions in SAP can be hard to implement and expensive to buy. While Data Services has some basic file capabilities built-in, they are basic and simple; they don’t meet the kinds of typical use cases described in this blog post.

Alternatives like SAP NetWeaver Process Integration, due to their inherent complexity, can take a long time to implement, require highly experienced staff to configure, and can be expensive to purchase.

Flux provides a pretty simple way to design, configure, and run ETL workflow processes within SAP environments using pretty simple tools and in a few pretty easy steps.